diff options
| -rw-r--r-- | TODO.md | 4 | ||||
| -rw-r--r-- | src/core/core.c | 196 |
2 files changed, 184 insertions, 16 deletions
@@ -536,7 +536,7 @@ brief someone without re-reading the code. - [ ] keep improving the borrow checker: fewer annotations in ordinary code, clearer messages naming both the owner and the conflicting use, and new fixtures for each rule that changes. - [ ] x86-64 jit, phase 2 follow-ups (phase 2 is in the done list). we are not chasing llvm parity on the jit (owner's decision). - region 474..692 in `tests/programs/o3.zc` falls back to the region interpreter; find out why and compile it. - - fib is slightly slower under o3 than plain (0.86 against 0.78 ms), likely region entry and exit on every recursive call; related to direct o3-to-o3 calls. + - [x] direct o3-to-o3 calls on x86-64 (`rcall_x64` / `rfinish_x64`, like arm64's `rcall`): callee stub via `fn_nat`, linkstk depth, slow path through `o3_invoke`; generation guard clears `fn_nat` on `rforget` and covers the function prologue so `icon.replace_function` drops the region (`tests/icon/regression/o3_caller_replace.zc`). - a compiled o3 region that stores into its own code finishes the current activation on the old code; the change takes effect on the next entry. either document it or bail to the region interpreter when the hook drops the running region. - [ ] (in progress on work/exactsize) exact-size bounds checks: n is stored in p[-4] bits 32..62 and the check is 2 loads. - [ ] (on work/checks) bounds checks out of loops and `--explain-checks`. done: a for loop's body is read ahead (`body_scan` in analyzer.zc) for the names it assigns, increments or takes the address of, closures, and anything that might free a block (calls, push/pop/resize/defer/asm/quote/new, stores to locals that can own); loop proofs now need a body that leaves the index, the bound and the array alone (before, a body that moved the index after a proved access was unchecked on the next iteration). `i < n` with an int n hoists the check to a flag computed at the loop entry (`hoist_guard`, `hoist_stub`): the check still runs, in place, when the flag says it can fail, so faults and side effects keep their order. a pointer set from `rt.new(n)` with a constant n is proved by `i < k` (k <= n) or hoisted, using the requested size (never the rounded block size), and only when the body cannot free or resize anything. `zy --explain-checks [file] < src` prints the kept checks as json lines from the same decisions (format v1 in docs/SPEC.md 4.15.4; consumers ignore unknown kinds, reasons and fields; `o3-candidate` is a possible future kind). later: hoisting for growable arrays with a variable bound, `while` loops, pointer sizes from `len()` once exact sizes land, mechanical `edit`s, tracking ownership precisely enough to hoist past calls that cannot free. cleanup: let the `bs_*` body scanner reuse capture_scan's token skipping instead of its own (about 0.5 to 1k genome words). the `--explain-checks` writer is `src/lib/explain.zc`, embedded and compiled only in that mode (a unit that defines its own `explain` namespace would clash with it). @@ -551,7 +551,7 @@ brief someone without re-reading the code. ## done in the sorted list - [x] a cross-reference report: for every global of the compiler, the functions that write it and the ones that only read it. build it from the language server's references (`textDocument/references` on `build/bin/zclsp`, so scoping is exact) and classify each reference as a write when an assignment operator, `++` or `--` follows it. output under `build/xref/`; a committed baseline and a `--check` mode that fails when a new writer of a global appears, so it can run in ci. done: `--check` runs in `tests/test.sh` through `xref_selftest.sh` against `tests/xref.baseline`. - [x] x86-64 jit phase 2: compile o3 regions to x86-64 (port `rcompile` and the region register allocation; regions now run on the region interpreter and are 2 to 4 times slower than their plain versions on this jit) and inline enhancements (lower the optimized micro-ops to x86 registers as the arm64 path does, instead of calling `xrun`). validate with `tests/drivers/jit_x64_test.sh` and `tests/bench/bench.sh`, which now has an x86-64 jit column. add the driver to `tests/test.sh` on x86-64 linux. done: inline enhancements merged as 1f4aceb, compiled o3 regions merged as ec49c6d. - done so far: enhancements are inlined (`x64_spec_enh`, `x64_enh`, `x64_uop` in core.c; seven value registers rsi, rdi, r8..r11, rbp; rax, rcx, rdx scratch; floats, byte loads and stores, native calls, popcnt and f2s/s2f still go through `xrun`; `ZYNOENH=1` calls everything as before). bench.sh plain kernels on the x86-64 jit, ms before/after: collatz 49.1/32.1, sieve 4.4/4.1, digits 41.9/14.7, fib 0.83/0.77; o3 unchanged (regions still interpreted). a 256-word pointer sweep with bounds checks 291 to 72 ms. `tests/programs/enh_ops.zc` covers every lowered op on edge values; jit_x64_test runs in test.sh's jit section (not in the quick path). o3 regions now compile (`rjit_x64`, `x64_ralloc`, `remit_x64` in core.c): vars stay in their frame slots, values in six registers, calls go through x_disp; `ZYO3INTERP=1` keeps the region interpreter. bench.sh o3 kernels on the x86-64 jit, ms interpreted/compiled: collatz 289/10.4, sieve 8.7/0.40, digits 89/3.0, fib 3.0/0.92 (fib is still slower than its plain version: calls and returns go through the dispatch stub). still interpreted: regions with more than six live values (one in tests/programs/o3.zc) or native calls. later: vars in registers across a loop, direct o3 calls, floats in xmm registers, spilling instead of giving up. caveat: a store from a region's native code into its own words drops the region, but the block that stored finishes on the old code (arm64 does the same). `-DZYRDUMP` dumps each region's blocks (off by default). the loop-entry flag is done on work/checks (see below). next, after `--explain-checks` v1: direct o3-to-o3 calls on x86-64 (like arm64's `rcall`), so fib stops paying for the dispatch stub. a direct o3-to-o3 call must keep a callee generation guard (like llvm's `inline_fn` checking `C->gen`) so a function rewritten by `icon.find_replace` is not bypassed; `tests/icon/regression/o3_caller_replace.zc` calls an o3 callee from an o3 caller before and after a replacement. region 474..692 of tests/programs/o3.zc stays on the region interpreter because it has more than six live values (`x64_ralloc` gives up; needs spilling, dropped from this round, so not a small fix). fib under o3 is slower than plain because each o3 call and return goes through the dispatch stub (`x_disp`) while the plain jit calls directly; the fix is the direct o3 calls above, not small. + done so far: enhancements are inlined (`x64_spec_enh`, `x64_enh`, `x64_uop` in core.c; seven value registers rsi, rdi, r8..r11, rbp; rax, rcx, rdx scratch; floats, byte loads and stores, native calls, popcnt and f2s/s2f still go through `xrun`; `ZYNOENH=1` calls everything as before). bench.sh plain kernels on the x86-64 jit, ms before/after: collatz 49.1/32.1, sieve 4.4/4.1, digits 41.9/14.7, fib 0.83/0.77; o3 unchanged (regions still interpreted). a 256-word pointer sweep with bounds checks 291 to 72 ms. `tests/programs/enh_ops.zc` covers every lowered op on edge values; jit_x64_test runs in test.sh's jit section (not in the quick path). o3 regions now compile (`rjit_x64`, `x64_ralloc`, `remit_x64` in core.c): vars stay in their frame slots, values in six registers, o3-to-o3 calls use `rcall_x64`; `ZYO3INTERP=1` keeps the region interpreter. bench.sh o3 kernels on the x86-64 jit, ms interpreted/compiled: collatz 289/10.4, sieve 8.7/0.40, digits 89/3.0, fib 3.0/0.92 (before direct o3 calls). still interpreted: regions with more than six live values (one in tests/programs/o3.zc) or native calls. later: vars in registers across a loop, floats in xmm registers, spilling instead of giving up. caveat: a store from a region's native code into its own words drops the region, but the block that stored finishes on the old code (arm64 does the same). `-DZYRDUMP` dumps each region's blocks (off by default). the loop-entry flag is done on work/checks (see below). direct o3-to-o3 calls on x86-64 landed (`rcall_x64` / `rfinish_x64`, `fn_nat` generation guard, prologue coverage for `icon.replace_function`; `tests/icon/regression/o3_caller_replace.zc`). region 474..692 of tests/programs/o3.zc stays on the region interpreter because it has more than six live values (`x64_ralloc` gives up; needs spilling, dropped from this round, so not a small fix). - [x] llvm by default (decided): llvm is turned on automatically on non-x86-64 hosts; on x86-64 the jit stays the default and llvm is opt-in with `LLVM=1`. - [x] stale known-line check: conformance and parity fail when a known entry passes again (merged in c1072ab). - [x] `defer expr;` and `defer { ... }`: tokens grabbed at the `defer`, recompiled inline at every exit (block end through `block_close`, `return` after its value, `break`/`continue` for entries deeper than the loop), newest first; only defers reached so far register. names are lexical (a local declared after the `defer` is hidden in the replay). a deferred `rt.free(p)` is `p`'s transfer to the analyzer. each entry replays in the analyzer mode it was written in; closures get their own base through `ctx_push`. `return`, `break` and `continue` that would leave a deferred statement are rejected. a failed entry reports once. limit 256 pending per function (`DFR_MAX`). spec 4.14, learn section 34, `tests/programs/defer.zc`, seven reject fixtures. diff --git a/src/core/core.c b/src/core/core.c index e7b1ef6..748c508 100644 --- a/src/core/core.c +++ b/src/core/core.c @@ -3309,6 +3309,13 @@ static Rg *rat(int p) #ifdef ZYNATIVE static VM_TLS unsigned char *cov; /* how many live translations cover the word (255 sticks) */ static VM_TLS void **tab; /* vm address -> translation */ +#if defined(ZYJIT) || defined(ZYJIT_X64) +typedef struct { void *stub; int fslots; int argc; } FnNat; +static VM_TLS FnNat *fn_nat; +#define LINKN 1024 +static VM_TLS void *linkstk[LINKN]; +static VM_TLS Cell jit_depth; +#endif static void be_drop(int j) { tab[ix_blk[j].a] = 0; cdec(cov, ix_blk[j].a, ix_blk[j].b); if (trace) fprintf(stderr, "jit: dropped %d..%d\n", ix_blk[j].a, ix_blk[j].b - 1); } #ifdef ZYJIT typedef unsigned JW; /* code is a string of 32-bit instruction words */ @@ -3630,7 +3637,12 @@ static int hook(Cell a, Cell v) m[a] = v; hit = ix_drop_at((int)a); for (j = 0; j < nrg; j++) - if (rgs[j]->lifted && rgs[j]->lo <= a && a < rgs[j]->hi) { + if (rgs[j]->lifted && ((rgs[j]->lo <= a && a < rgs[j]->hi) || + ((unsigned)rgs[j]->fn < (unsigned)code_words && a >= rgs[j]->fn && a < rgs[j]->fn + 2))) { + /* a store into the function prologue replaces the function: the region is done. + a store into the body only (lo..hi) clears it so it can lift again. */ + if ((unsigned)rgs[j]->fn < (unsigned)code_words && a >= rgs[j]->fn && a < rgs[j]->fn + 2) + rgs[j]->dead = 1; if (trace) fprintf(stderr, "o3: dropped %d..%d (a store to %d)\n", rgs[j]->lo, rgs[j]->hi - 1, (int)a); rclear(rgs[j]); hit = 1; @@ -3981,11 +3993,6 @@ static VM_TLS unsigned char rk[MAXUOP * 4 + 16], rhome[MAXUOP * 4 + 16], rfp[MAX static VM_TLS Cell rkv[MAXUOP * 4 + 16]; static VM_TLS int rlast[MAXUOP * 4 + 16]; static VM_TLS unsigned **rfa; static VM_TLS int *rfb, rnf; /* branches to blocks, fixed up at the end */ -#define LINKN 1024 -static VM_TLS void *linkstk[LINKN]; -static VM_TLS Cell jit_depth; -typedef struct { void *stub; int fslots; int argc; } FnNat; -static VM_TLS FnNat *fn_nat; static void rforget(Rg *R) { int e; @@ -5323,9 +5330,20 @@ static void rmark(Rg *R) { int e; R->marked = 1; cinc(rcov, R->lo, R->hi); cinc(cov, R->lo, R->hi); + /* cover the function prologue (R->fn): icon.replace_function patches it with + lit+jmp, and a direct o3 caller must not keep the old stub when that happens. */ + if ((unsigned)R->fn < (unsigned)code_words) { + cinc(rcov, R->fn, R->fn + 2); cinc(cov, R->fn, R->fn + 2); + tab[R->fn] = 0; + if ((unsigned)(R->fn + 1) < (unsigned)code_words) tab[R->fn + 1] = 0; + } for (e = 0; e < R->ne; e++) if ((unsigned)R->ent[e] < (unsigned)code_words) tab[R->ent[e]] = 0; } -static void runmark(Rg *R) { cdec(rcov, R->lo, R->hi); cdec(cov, R->lo, R->hi); } +static void runmark(Rg *R) +{ + cdec(rcov, R->lo, R->hi); cdec(cov, R->lo, R->hi); + if ((unsigned)R->fn < (unsigned)code_words) { cdec(rcov, R->fn, R->fn + 2); cdec(cov, R->fn, R->fn + 2); } +} #else static void rforget(Rg *R) { (void)R; } static void rmark(Rg *R) { R->marked = 1; cinc(rcov, R->lo, R->hi); } @@ -6298,6 +6316,123 @@ static void rvset(Rg *R, int j, int r) } static VM_TLS JW **rfa; static VM_TLS int *rfb, nrf; static void rjmp(JW *at, int b) { rfa[nrf] = at; rfb[nrf++] = b; } +/* leave a region to a native caller: restore the frame, drop depth, ret. depth 0 + leaves through x_exit (bytecode entered the region). */ +static void rfinish_x64(Rg *R) +{ + JW *z, *ne; + mov_ri(r_cx, (Cell)(uintptr_t)&jit_depth); + mov_rm(r_ax, r_cx, 0); + test_rr(r_ax, r_ax); + z = j32(c_e); + mov_ri(r_cx, R->hi); + alu_rr(a_cmp, r_dx, r_cx); + ne = j32(c_ne); + alu_ri(a_sub, rSP, 8); + mov_mi(rSP, 0, 0); + fix32(ne, cp); + if (R->fslots) { + mov_rm(r_ax, rMB, 8 * ZFP); + alu_ri(a_add, r_ax, R->fslots); + mov_mr(rMB, 8 * ZFP, r_ax); + } + mov_ri(r_cx, (Cell)(uintptr_t)&jit_depth); + mov_rm(r_ax, r_cx, 0); + alu_ri(a_sub, r_ax, 1); + mov_mr(r_cx, 0, r_ax); + mov_ri(r_cx, (Cell)(uintptr_t)linkstk); + mov_rx(r_ax, r_cx, r_ax, 8, 0); + jmp_r(r_ax); + fix32(z, cp); + mov_rr(r_ax, r_dx); + jmp_to(x_exit); +} +/* o3-to-o3: call the callee stub with a built frame. else a vm call via x_disp. + generation guard: fn_nat stub is cleared in rforget when the callee drops. */ +static void rcall_x64(Rg *R, RB *B) +{ + JW *to_slow, *is_null, *too_deep, *too_small, *argc0, *loop; + int e, hopped; + rspill(R); + /* rdx = callee target */ + mov_ri(r_cx, (Cell)(uintptr_t)&code_words); + alu_rm(a_cmp, r_dx, r_cx, 0); + to_slow = j32(c_ae); + mov_ri(r_ax, (Cell)(uintptr_t)&fn_nat); + mov_rm(r_ax, r_ax, 0); + mov_rr(r_cx, r_dx); + shift_ri(4, r_cx, 4); + alu_rr(a_add, r_ax, r_cx); + mov_rm(r_9, r_ax, 0); + test_rr(r_9, r_9); + is_null = j32(c_e); + mov_ri(r_cx, (Cell)(uintptr_t)&jit_depth); + mov_rm(r_8, r_cx, 0); + alu_ri(a_cmp, r_8, LINKN); + too_deep = j32(c_ae); + mov_r32m(r_si, r_ax, 8); + mov_r32m(r_di, r_ax, 12); + mov_rm(r_ax, rMB, 8 * ZFP); + alu_rr(a_sub, r_ax, r_si); + mov_ri(r_cx, (Cell)(uintptr_t)&frame_floor_word); + mov_rm(r_cx, r_cx, 0); + mov_rx(r_8, rMB, r_cx, 8, 0); + mov_ri(r_cx, (Cell)(uintptr_t)&frame_guard_word); + mov_rm(r_cx, r_cx, 0); + mov_rx(r_cx, rMB, r_cx, 8, 0); + alu_rr(a_add, r_8, r_cx); + alu_rr(a_cmp, r_ax, r_8); + too_small = j32(c_b); + push_r(r_9); + mov_mr(rMB, 8 * ZFP, r_ax); + mov_rr(r_8, r_ax); + test32(r_di, r_di); + argc0 = j32(c_e); + loop = cp; + alu_ri(a_sub, r_di, 1); + mov_rm(r_cx, rSP, 0); + alu_ri(a_add, rSP, 8); + lea(r_ax, r_8, r_di, 1, 0); + mov_xr(rMB, r_ax, 8, 0, r_cx); + test32(r_di, r_di); + jcc_to(c_ne, loop); + fix32(argc0, cp); + pop_r(r_9); + mov_ri(r_cx, (Cell)(uintptr_t)&jit_depth); + mov_rm(r_ax, r_cx, 0); + alu_ri(a_add, r_ax, 1); + mov_mr(r_cx, 0, r_ax); + alu_ri(a_sub, r_sp, 8); + call_r(r_9); + alu_ri(a_add, r_sp, 8); + hopped = 0; + for (e = 0; e < R->ne; e++) if (R->ent[e] == B->x2) { jmp_to((JW *)R->stub[e]); hopped = 1; break; } + if (!hopped) { mov_r32i(r_ax, (unsigned)B->x2); jmp_to(x_disp); } + fix32(to_slow, cp); + fix32(is_null, cp); + fix32(too_deep, cp); + fix32(too_small, cp); + /* slow: publish stacks (region code: no vd), o3_invoke, resume */ + mov_rr(r_cx, rSP); alu_rr(a_sub, r_cx, rMB); shift_ri(5, r_cx, 3); mov_mr(rST, 8, r_cx); + mov_rr(r_cx, rRP); alu_rr(a_sub, r_cx, rMB); shift_ri(5, r_cx, 3); mov_mr(rST, 16, r_cx); + lea(r_di, rST, -1, 1, 8); + lea(r_si, rST, -1, 1, 16); + /* rdx still holds target from the early path */ + mov_ri(r_cx, B->x2); + call_abs((void *)o3_invoke); + reload_stacks(); + mov_ri(r_cx, (Cell)(uintptr_t)&halt_at); + mov_rm(r_ax, r_cx, 0); + alu_ri(a_cmp, r_ax, 0); + { + JW *ok = j32(c_l); + jmp_to(x_exit); + fix32(ok, cp); + } + hopped = 0; + for (e = 0; e < R->ne; e++) if (R->ent[e] == B->x2) { jmp_to((JW *)R->stub[e]); hopped = 1; break; } + if (!hopped) { mov_r32i(r_ax, (unsigned)B->x2); jmp_to(x_disp); } +} static void remit_x64(Rg *R, RB *B) { int i, j, d; @@ -6344,16 +6479,22 @@ static void remit_x64(Rg *R, RB *B) if (B->s2 >= 0) rjmp(jm32(), B->s2); else rexit(R, B->x2); break; case T_CALL: - rspill(R); - mov_mi(rRP, 0, B->x2); alu_ri(a_add, rRP, 8); - mov_rr(r_ax, r_dx); jmp_to(x_disp); + rcall_x64(R, B); break; case T_RET: rspill(R); alu_ri(a_sub, rRP, 8); mov_rm(r_ax, rRP, 0); jmp_to(x_disp); break; - default: - rspill(R); mov_rr(r_ax, r_dx); jmp_to(x_disp); + default: { + JW *hs; + rspill(R); + mov_ri(r_cx, R->hi); + alu_rr(a_cmp, r_dx, r_cx); + hs = j32(c_ae); + mov_rr(r_ax, r_dx); jmp_to(x_disp); + fix32(hs, cp); + rfinish_x64(R); + break; } } } /* 0: compiled; -1: can't be; -2: no room now */ @@ -6379,7 +6520,19 @@ static int rjit_x64(Rg *R) for (b = 0; b < R->np; b++) fprintf(stderr, "ps %d -> var %d\n", R->ps[b], R->pv[b]); #endif for (e = 0; e < R->ne && !rc; e++) { + JW *skip; R->stub[e] = cp; + if (e == 0) { + mov_ri(r_cx, (Cell)(uintptr_t)&jit_depth); + mov_rm(r_ax, r_cx, 0); + test_rr(r_ax, r_ax); + skip = j32(c_e); + alu_ri(a_sub, r_ax, 1); + pop_r(r_cx); + mov_ri(r_8, (Cell)(uintptr_t)linkstk); + mov_xr(r_8, r_ax, 8, 0, r_cx); + fix32(skip, cp); + } mov_rm(r_ax, rMB, 8 * ZFP); lea(r_bp, rMB, r_ax, 8, 0); rjmp(jm32(), R->entb[e]); } @@ -6394,6 +6547,11 @@ static int rjit_x64(Rg *R) for (k = 0; k < nrf; k++) fix32(rfa[k], lab[rfb[k]]); R->nat = start; for (e = 0; e < R->ne; e++) if ((unsigned)R->ent[e] < (unsigned)code_words) tab[R->ent[e]] = R->stub[e]; + if ((unsigned)R->fn < (unsigned)code_words) { + fn_nat[R->fn].stub = R->stub[0]; + fn_nat[R->fn].fslots = R->fslots; + fn_nat[R->fn].argc = R->argc; + } if (trace) fprintf(stderr, "o3: %d..%d -> %d bytes of x86-64\n", R->lo, R->hi - 1, (int)(cp - start)); } else cp = start; free(lab); free(rfa); free(rfb); @@ -6403,6 +6561,7 @@ static void rforget(Rg *R) { int e; if (R->nat) for (e = 0; e < R->ne; e++) if ((unsigned)R->ent[e] < (unsigned)code_words && tab[R->ent[e]] == R->stub[e]) tab[R->ent[e]] = 0; + if ((unsigned)R->fn < (unsigned)code_words) fn_nat[R->fn].stub = 0; R->nat = 0; } static void rcompile(Rg *R, int eager) @@ -6603,6 +6762,13 @@ static int vm_until(int pc, int stop_eq, int stop_ge, Rg *re, Cell *ps, Cell *pr void *saved_eq = 0, *saved_ent[RMAXE]; int i, nent = 0, flushed = 0, ep0 = drop_epoch, term = 0; Cell return_depth = *pr - (stack_control ? stack_get(SC_RETURN_LOW) : RS0) - 1; + Cell saved_depth = jit_depth, *saved_links = 0; + if (saved_depth) { + saved_links = malloc((size_t)saved_depth * sizeof *saved_links); + if (!saved_links) return PC_FRAME_OVERFLOW; + memcpy(saved_links, linkstk, (size_t)saved_depth * sizeof *saved_links); + } + jit_depth = 0; enter.u = x_enter; if (stop_eq >= 0 && stop_eq < code_words) { saved_eq = tab[stop_eq]; tab[stop_eq] = 0; } if (re) { @@ -6654,6 +6820,8 @@ static int vm_until(int pc, int stop_eq, int stop_ge, Rg *re, Cell *ps, Cell *pr if (need_flush) { need_flush = 0; jitwrite(1); flush(); jitwrite(0); } } *ps = S[1]; *pr = S[2]; + jit_depth = saved_depth; + if (saved_links) { memcpy(linkstk, saved_links, (size_t)saved_depth * sizeof *saved_links); free(saved_links); } flushed = drop_epoch != ep0; if (!flushed) { if (stop_eq >= 0 && stop_eq < code_words) tab[stop_eq] = saved_eq; @@ -6912,7 +7080,7 @@ static int code_reserve(Cell end) CODE_COLUMN(o3ent, code_words, cap); CODE_COLUMN(o3ei, code_words, cap); #ifdef ZYNATIVE CODE_COLUMN(tab, code_words, cap); -#ifdef ZYJIT +#if defined(ZYJIT) || defined(ZYJIT_X64) CODE_COLUMN(fn_nat, code_words, cap); #endif #elif defined(ZYLLVM) @@ -7004,7 +7172,7 @@ static void context_engine_free(void) { for (i = 0; i < NENH; i++) { free(enh[i].u); free(enh[i].reg); } free(pg_head); free(o3ent); free(o3ei); free(gn_); free(gh_); #ifdef ZYNATIVE -#ifdef ZYJIT +#if defined(ZYJIT) || defined(ZYJIT_X64) free(fn_nat); #endif free(tab); free(cov); if (code && code != MAP_FAILED && munmap(code, CODEW * 4) == 0) atomic_fetch_sub(&context_code_live, 1); |