merge work/jit2: x86-64 jit inline enhancements

8 files changed, 400 insertions, 9 deletions

diff --git a/TODO.md b/TODO.md
index 6d33896..2520491 100644
--- a/TODO.md
+++ b/TODO.md
@@ -520,7 +520,8 @@ brief someone without re-reading the code.
 - [x] (done on work/line-tables, see "agent work at the end of the session") runtime source positions. architecture: at each statement start (`stmt()`) record (code address `here`, line); per function a permanent data block [entry, end, name, file, count, pairs...] linked into a runtime list (`rt.line_blocks`), or appended to run-only images by `build_image` like the symbol blob. `lines on;`/`lines off;` like `docs on;`; on for user programs, off for the genome unless the growth is small. a new syscall op returns the return-stack pointer (the return stack is in vm memory, 0x1A4100..0x1B40FF, growing up); implement it in `src/core/core.c`, `src/micro/micro.c`, `src/js/tinyvm.ts` and the wasm port. `rt.trace()` walks the return stack, printing `  at file:line in function` for words inside a known function range. the runtime's fatal reports (`oob`, bad heap pointer, double free, out of memory, frame overflow) print the trace after their unchanged first line. check mode skips all of it.
 - [ ] borrow returns tied to an argument. inference already works for a callee defined before the call: `char* skip(char* s) { ...; return s + i; }` makes `let r = skip(t); rt.free(t); r[0]` an error. the gap: when the call comes before the callee's definition, nothing is reported and the program reads freed memory. extend the deferred effect requirements (`effect_require`, `effect_finish` at unit end) so a forward-called function whose return turns out to alias a parameter produces the same error. for bodyless declarations, externs and function types, add an unobtrusive spelling, for example a `borrows s` clause after the parameter list (`char* skip(char* s) borrows s;`), checked against the body when one exists.
 - [ ] variadic arguments and `std.print`. design: a pack is passed as one hidden argument, a caller-built read-only growable-array handle (`fixed T[?]` view: [len, cap, items, element size] in the caller's frame, so `len(a)` and `a[i]` work unchanged and nothing is allocated). `...int a` is a typed pack; several packs per function when their element types differ (`void f(...int a, ...char b)`): each extra argument goes to the pack whose element type matches its static type, an ambiguous match is an error, packs come after the fixed parameters. `...any args` packs each value with a kind code (int, float, char, string, bstr, pointer, ...) and its type record: `any` is a small built-in record { kind, value, type }. `...*any args` makes the call comma-free: after the fixed arguments, the parser reads unary expressions until `)`, commas optional, so `std.print("x = " name ", age " age "\n");` works and binary expressions must be parenthesized. `std.print(...*any args)` prints each argument by its kind (`put_n`, the float printer, `put_s`, `put_bs`, a character for `char`, hex for pointers).
-- [ ] x86-64 jit phase 2: compile o3 regions to x86-64 (port `rcompile` and the region register allocation; regions now run on the region interpreter and are 2 to 4 times slower than their plain versions on this jit) and inline enhancements (lower the optimized micro-ops to x86 registers as the arm64 path does, instead of calling `xrun`). validate with `tests/drivers/jit_x64_test.sh` and `tests/bench/bench.sh`, which now has an x86-64 jit column. add the driver to `tests/test.sh` on x86-64 linux.
+- [ ] (inlining done on work/jit2) x86-64 jit phase 2: compile o3 regions to x86-64 (port `rcompile` and the region register allocation; regions now run on the region interpreter and are 2 to 4 times slower than their plain versions on this jit) and inline enhancements (lower the optimized micro-ops to x86 registers as the arm64 path does, instead of calling `xrun`). validate with `tests/drivers/jit_x64_test.sh` and `tests/bench/bench.sh`, which now has an x86-64 jit column. add the driver to `tests/test.sh` on x86-64 linux.
+  done so far: enhancements are inlined (`x64_spec_enh`, `x64_enh`, `x64_uop` in core.c; seven value registers rsi, rdi, r8..r11, rbp; rax, rcx, rdx scratch; floats, byte loads and stores, native calls, popcnt and f2s/s2f still go through `xrun`; `ZYNOENH=1` calls everything as before). bench.sh plain kernels on the x86-64 jit, ms before/after: collatz 49.1/32.1, sieve 4.4/4.1, digits 41.9/14.7, fib 0.83/0.77; o3 unchanged (regions still interpreted). a 256-word pointer sweep with bounds checks 291 to 72 ms. `tests/programs/enh_ops.zc` covers every lowered op on edge values; jit_x64_test runs in test.sh's jit section (not in the quick path). next: o3 regions (`rcompile`, region register allocation), then the loop-entry flag.
 - [x] `memdebug`'s `obj_base` used to search only 64 words back for a block header, so `buf[255]` on a 256-word block reported "bad heap pointer". it now rounds the pointer down to each size class's alignment (largest first) and takes the first header whose check word, size and class agree and whose payload holds the pointer, so interiors at any depth work, a deep pointer into a freed block reports use after free, and header words and free space still report "bad heap pointer". regression: `tests/programs/memdebug_interior.zc` (in `tests/test.sh` beside the other memdebug checks). note that a stray pointer that lands inside another live block is accepted, since memdebug cannot know which block it came from.
 - [ ] (partly done, see "agent work at the end of the session") `defer` follow-ups (small): accept `defer if (...) ...;` and other single statements besides an expression; decide whether deferred code may name an owner declared after the `defer` (now rejected by the lexical rule; one line in `defer_one`).
 - [ ] build the llvm core by default when `llvm-config` is available (the makefile falls back to the interpreter otherwise), so x86 hosts get a native tier and the language server stops being interpreted; keep `ZYNOLLVM=1` and `-DZYINTERP` as the opt-outs.
diff --git a/src/core/core.c b/src/core/core.c
index 2a0c467..c4306ae 100644
--- a/src/core/core.c
+++ b/src/core/core.c
@@ -4,8 +4,8 @@
    memory. stores that land on translated words throw those translations away,
    so code that rewrites itself, including the resident compiler, just gets
    retranslated on the next visit. the arm64 jit also compiles o3 regions; on
-   x86-64 they run on the region interpreter, and enhancements are called, not
-   inlined. on other hosts (or with -DZYINTERP) a plain interpreter with the same
+   x86-64 they run on the region interpreter; enhancements are inlined on both
+   (on x86-64 those with float, byte or native-call ops are still called). on other hosts (or with -DZYINTERP) a plain interpreter with the same
    semantics runs instead. ZYTRACE=1 logs translations on stderr. */
 #include <stdio.h>
 #include <stdlib.h>
@@ -633,7 +633,7 @@ static int unew(int op, int a, int b, int c, Cell imm)
 	u = &ub[un++]; u->op = op; u->d = unv++; u->a = a; u->b = b; u->c = c; u->imm = imm;
 	return u->d;
 }
-static Cell mread(int a) { return (unsigned)a < MW ? m[a] : 0; }
+static Cell mread(Cell a) { return (UCell)a < MW ? m[a] : 0; }	/* the whole word: an address past 2^32 reads 0, as on the jits and the microcore */
 
 /* recipe text: one statement per line or per ';'
      r0 = sdiv r0 r1      dst = op operands, operands are rN, parameter names or numbers
@@ -1014,11 +1014,12 @@ static void xdump(int slot, Enh *e)
 	}
 	for (i = 0; i < e->nout; i++) fprintf(stderr, "  out%d = v%d\n", i, e->out[i]);
 }
-/* registers for the jit: values live in x0..x15, straight-line linear scan.
-   if they don't fit, reg stays 0 and the jit calls the interpreter instead. */
-static void regalloc(Enh *e)
+/* registers for the jit: values live in x0..x15 (registers 0..nr-1 of the
+   x86-64 jit's own set there), straight-line linear scan. if they don't fit,
+   reg stays 0 and the jit calls the interpreter instead. */
+static void regalloc_n(Enh *e, int nr)
 {
-	int *last = malloc(e->nv * sizeof(int)), i, j, k, fr = 0xffff;
+	int *last = malloc(e->nv * sizeof(int)), i, j, k, fr = (1 << nr) - 1;
 	signed char *r = malloc(e->nv);
 	for (i = 0; i < e->nv; i++) { last[i] = -2; r[i] = -1; }
 	for (i = 0; i < e->n; i++) {
@@ -1028,7 +1029,7 @@ static void regalloc(Enh *e)
 		if (uarity[u->op] > 2) last[u->c] = i;
 	}
 	for (i = 0; i < e->nout; i++) last[e->out[i]] = e->n;
-#define TAKE(v) do { for (k = 0; k < 16 && !(fr >> k & 1); k++); if (k == 16) goto fail; fr &= ~(1 << k); r[v] = (signed char)k; } while (0)
+#define TAKE(v) do { for (k = 0; k < nr && !(fr >> k & 1); k++); if (k == nr) goto fail; fr &= ~(1 << k); r[v] = (signed char)k; } while (0)
 	for (j = 0; j < e->nin; j++) if (last[j] >= -1 && last[j] != -2) TAKE(j);
 	for (i = 0; i < e->n; i++) {
 		Uop *u = &e->u[i]; int ops[3], no = 0;
@@ -1044,6 +1045,7 @@ static void regalloc(Enh *e)
 fail:
 	free(last); free(r); e->reg = 0;
 }
+static void regalloc(Enh *e) { regalloc_n(e, 16); }
 /* the install opcode. descriptor: slot, kind (0 recipe, 1 vm code), inputs,
    outputs, payload address, payload length, parameter names (or 0). kind 2
    is an o3 region (below).
@@ -5901,6 +5903,7 @@ static void rcompile(Rg *R, int eager)
    translated (the c loop sees it first), and the region interpreter runs it. */
 static VM_TLS int vd;
 static VM_TLS int x64_noo3;	/* ZYNOO3=1: leave o3 regions alone and run their vm code through the jit (what the microcore does) */
+static VM_TLS int x64_noenh;	/* ZYNOENH=1: call every enhancement through xrun, as phase 1 did */
 #define SL(k) (8 * (vd + (k)))
 static void ldv(int r, int k) { mov_rm(r, rSP, SL(k)); }
 static void stv(int k, int r) { mov_mr(rSP, SL(k), r); }
@@ -6016,6 +6019,150 @@ static void float_op(int i, int k)
 	(void)i;
 }
 
+/* ---- enhancements, inlined ----
+   an enhancement's optimized micro-ops are lowered to x86-64 instead of being
+   run by xrun: values live in rsi, rdi, r8..r11 and rbp (regalloc_n with seven
+   registers), rax, rcx and rdx are the scratch that division, shifts and setcc
+   need, and constants are made where they are used. like the arm64 jit, a run
+   of literals in front of the opcode specializes the enhancement on them
+   (x64_spec_enh), so the optimizer sees them as constants. an enhancement with
+   an op this lowering lacks (floats, byte loads and stores, native calls), one
+   that does not fit the registers, or one too long still goes through xrun. */
+static const int xreg[] = { r_si, r_di, r_8, r_9, r_10, r_11, r_bp };
+#define XNREG 7
+#define XMAXOP 400
+static VM_TLS int xkc[MAXUOP * 4 + 16]; static VM_TLS Cell xkv[MAXUOP * 4 + 16];
+static VM_TLS signed char *xr;
+static int x64_op_ok(int op)
+{
+	switch (op) {
+	case U_LI: case U_MOV: case U_ADD: case U_SUB: case U_MUL: case U_SDIV: case U_UDIV: case U_SREM: case U_UREM:
+	case U_AND: case U_OR: case U_XOR: case U_SHL: case U_SHR: case U_SAR: case U_NEG: case U_NOT:
+	case U_LT: case U_LTU: case U_EQ: case U_NE: case U_MIN: case U_MAX: case U_MINU: case U_MAXU:
+	case U_MULHS: case U_MULHU: case U_CLZ: case U_CTZ: case U_BSWAP: case U_ROTL: case U_ROTR:
+	case U_ABS: case U_SEL: case U_LD: case U_ST:
+		return 1;
+	}
+	return 0;
+}
+/* the enhancement in slot s, specialized on nk literals that feed its last inputs
+   (nk may be 0) and given x86-64 registers. 0 if it can be inlined */
+static int x64_spec_enh(int s, Cell *ks, int nk, Enh *te)
+{
+	Enh *e = &enh[s];
+	int nin = e->nin - nk, i, j, out[8], na;
+	static VM_TLS int mp[MAXUOP * 4 + 16];
+	if (e->nv > MAXUOP * 4 || e->n > XMAXOP) return -1;
+	for (i = 0; i < e->n; i++) if (!x64_op_ok(e->u[i].op)) return -1;
+	un = 0; unv = nin; uerr = 0;
+	for (j = 0; j < nin; j++) mp[j] = j;
+	for (j = 0; j < nk; j++) mp[nin + j] = unew(U_LI, 0, 0, 0, ks[j]);
+	for (i = 0; i < e->n; i++) {
+		Uop *u = &e->u[i];
+		na = nargs(u->op);
+		j = unew(u->op, na > 0 ? mp[u->a] : 0, na > 1 ? mp[u->b] : 0, na > 2 ? mp[u->c] : 0, u->imm);
+		if (u->op != U_ST) mp[u->d] = j;
+	}
+	for (j = 0; j < e->nout; j++) out[j] = mp[e->out[j]];
+	memset(te, 0, sizeof *te);
+	if (uerr || optimize(te, nin, out, e->nout)) return -1;
+	for (i = 0; i < te->n; i++) if (!x64_op_ok(te->u[i].op)) { free(te->u); return -1; }
+	regalloc_n(te, XNREG);
+	if (!te->reg) { free(te->u); return -1; }
+	return 0;
+}
+/* the register value v lives in, or scratch with the constant v loaded into it */
+static int xv(int v, int scratch) { if (xkc[v]) { mov_ri(scratch, xkv[v]); return scratch; } return xreg[(int)xr[v]]; }
+static int xsmall(int v) { return xkc[v] && xkv[v] >= -2147483647 - 1 && xkv[v] <= 2147483647; }
+static void cmov(int cc, int d, int s) { reg_op(0, 1, 1, 0x40 + cc, d, s); }
+static void x64_uop(Uop *u)
+{
+	int op = u->op, d = is_store(op) ? -1 : xreg[(int)xr[u->d]], k;
+	JW *z, *m1, *e1, *e2;
+	switch (op) {
+	case U_MOV: mov_rr(d, xv(u->a, r_ax)); return;
+	case U_ADD: case U_SUB: case U_AND: case U_OR: case U_XOR:
+		k = op == U_ADD ? a_add : op == U_SUB ? a_sub : op == U_AND ? a_and : op == U_OR ? a_or : a_xor;
+		mov_rr(r_ax, xv(u->a, r_ax));
+		if (xsmall(u->b)) alu_ri(k, r_ax, (int)xkv[u->b]); else alu_rr(k, r_ax, xv(u->b, r_cx));
+		mov_rr(d, r_ax); return;
+	case U_MUL:
+		mov_rr(r_ax, xv(u->a, r_ax)); reg_op(0, 1, 1, 0xaf, r_ax, xv(u->b, r_cx)); mov_rr(d, r_ax); return;
+	case U_SHL: case U_SHR: case U_SAR: case U_ROTL: case U_ROTR:
+		k = op == U_SHL ? 4 : op == U_SHR ? 5 : op == U_SAR ? 7 : op == U_ROTL ? 0 : 1;
+		if (xkc[u->b]) { mov_rr(r_ax, xv(u->a, r_ax)); if (xkv[u->b] & 63) shift_ri(k, r_ax, (int)(xkv[u->b] & 63)); }
+		else { mov_rr(r_cx, xv(u->b, r_cx)); mov_rr(r_ax, xv(u->a, r_ax)); shift_cl(k, r_ax); }
+		mov_rr(d, r_ax); return;
+	case U_NEG: case U_NOT:
+		mov_rr(r_ax, xv(u->a, r_ax)); reg_op(0, 1, 0, 0xf7, op == U_NEG ? 3 : 2, r_ax); mov_rr(d, r_ax); return;
+	case U_LT: case U_LTU: case U_EQ: case U_NE:
+		mov_rr(r_ax, xv(u->a, r_ax)); xor32(r_dx);
+		if (xsmall(u->b)) alu_ri(a_cmp, r_ax, (int)xkv[u->b]); else alu_rr(a_cmp, r_ax, xv(u->b, r_cx));
+		setcc(op == U_LT ? c_l : op == U_LTU ? c_b : op == U_EQ ? c_e : c_ne, r_dx); mov_rr(d, r_dx); return;
+	case U_MIN: case U_MAX: case U_MINU: case U_MAXU:	/* keep a unless the other one wins */
+		mov_rr(r_ax, xv(u->a, r_ax)); mov_rr(r_dx, xv(u->b, r_dx)); alu_rr(a_cmp, r_ax, r_dx);
+		cmov(op == U_MIN ? c_ge : op == U_MAX ? c_le : op == U_MINU ? c_ae : c_be, r_ax, r_dx); mov_rr(d, r_ax); return;
+	case U_SEL:
+		mov_rr(r_ax, xv(u->c, r_ax)); mov_rr(r_dx, xv(u->b, r_dx)); mov_rr(r_cx, xv(u->a, r_cx));
+		test_rr(r_cx, r_cx); cmov(c_ne, r_ax, r_dx); mov_rr(d, r_ax); return;
+	case U_ABS:
+		mov_rr(r_ax, xv(u->a, r_ax)); mov_rr(r_dx, r_ax); reg_op(0, 1, 0, 0xf7, 3, r_dx);
+		test_rr(r_ax, r_ax); cmov(c_l, r_ax, r_dx); mov_rr(d, r_ax); return;
+	case U_MULHS: case U_MULHU:
+		mov_rr(r_cx, xv(u->b, r_cx)); mov_rr(r_ax, xv(u->a, r_ax)); reg_op(0, 1, 0, 0xf7, op == U_MULHS ? 5 : 4, r_cx);
+		mov_rr(d, r_dx); return;
+	case U_SDIV: case U_SREM:	/* by 0: 0 or a; by -1: -a or 0, which also covers the minimum; else idiv */
+		mov_rr(r_cx, xv(u->b, r_cx)); mov_rr(r_ax, xv(u->a, r_ax));
+		test_rr(r_cx, r_cx); z = j32(c_e);
+		alu_ri(a_cmp, r_cx, -1); m1 = j32(c_e);
+		xb(0x48); xb(0x99); reg_op(0, 1, 0, 0xf7, 7, r_cx);	/* cqo; idiv rcx */
+		if (op == U_SREM) mov_rr(r_ax, r_dx);
+		e1 = jm32();
+		fix32(m1, cp);
+		if (op == U_SDIV) reg_op(0, 1, 0, 0xf7, 3, r_ax); else xor32(r_ax);
+		e2 = jm32();
+		fix32(z, cp);
+		if (op == U_SDIV) xor32(r_ax);
+		fix32(e1, cp); fix32(e2, cp);
+		mov_rr(d, r_ax); return;
+	case U_UDIV: case U_UREM:	/* by 0: 0 or a */
+		mov_rr(r_cx, xv(u->b, r_cx)); mov_rr(r_ax, xv(u->a, r_ax));
+		test_rr(r_cx, r_cx); z = j32(c_e);
+		xor32(r_dx); reg_op(0, 1, 0, 0xf7, 6, r_cx);		/* div rcx */
+		if (op == U_UREM) mov_rr(r_ax, r_dx);
+		e1 = jm32();
+		fix32(z, cp);
+		if (op == U_UDIV) xor32(r_ax);
+		fix32(e1, cp);
+		mov_rr(d, r_ax); return;
+	case U_CLZ:	/* bsr leaves zf set for 0: 127 ^ 63 is 64 */
+		mov_rr(r_ax, xv(u->a, r_ax)); mov_r32i(r_cx, 127); reg_op(0, 1, 1, 0xbd, r_dx, r_ax); cmov(c_e, r_dx, r_cx);
+		alu_ri(a_xor, r_dx, 63); mov_rr(d, r_dx); return;
+	case U_CTZ:
+		mov_rr(r_ax, xv(u->a, r_ax)); mov_r32i(r_cx, 64); reg_op(0, 1, 1, 0xbc, r_dx, r_ax); cmov(c_e, r_dx, r_cx);
+		mov_rr(d, r_dx); return;
+	case U_BSWAP:
+		mov_rr(r_ax, xv(u->a, r_ax)); xb(0x48); xb(0x0f); xb(0xc8); mov_rr(d, r_ax); return;
+	case U_LD:	/* out of range reads 0, like the interpreter */
+		mov_rr(r_ax, xv(u->a, r_ax)); emit_load(); mov_rr(d, r_cx); return;
+	case U_ST:	/* outside memory stores nothing; like xexec, no store hook */
+		mov_rr(r_ax, xv(u->a, r_ax)); alu_ri(a_cmp, r_ax, MW); z = j32(c_ae);
+		mov_xr(rMB, r_ax, 8, 0, xv(u->b, r_dx)); fix32(z, cp); return;
+	}
+}
+/* inputs from the vm stack into their registers, the ops, the outputs back */
+static void x64_enh(Enh *e)
+{
+	int i, j;
+	xr = e->reg;
+	for (i = 0; i < e->nv; i++) xkc[i] = 0;
+	for (i = 0; i < e->n; i++) if (e->u[i].op == U_LI) { xkc[e->u[i].d] = 1; xkv[e->u[i].d] = e->u[i].imm; }
+	for (j = 0; j < e->nin; j++) if (xr[j] >= 0) ldv(xreg[(int)xr[j]], e->nin - 1 - j);
+	for (i = 0; i < e->n; i++) if (e->u[i].op != U_LI) x64_uop(&e->u[i]);
+	vd += e->nin - e->nout;
+	for (j = 0; j < e->nout; j++) stv(e->nout - 1 - j, xv(e->out[j], r_ax));
+}
+
 static void *translate(int p)
 {
 	JW *start;
@@ -6045,6 +6192,25 @@ static void *translate(int p)
 			i += 2;
 			break;
 		}
+		if (w & 1 && cp - start < (1 << 18) && !x64_noenh) {	/* literals feeding an enhancement: specialize it on them */
+			int L = 0, s, nk, j;
+			Cell ks[8], x;
+			Enh te;
+			while (L < 8 && i + L < code_words && (m[i + L] & 1)) L++;
+			x = i + L < code_words ? m[i + L] : 1;
+			if (!(x & 1) && (s = (int)(x / 2) - XOP0) >= 0 && s < NENH && enh[s].live && enh[s].nin) {
+				nk = L < enh[s].nin ? L : enh[s].nin;
+				for (j = 0; j < nk; j++) ks[j] = m[i + L - nk + j] >> 1;
+				if (!x64_spec_enh(s, ks, nk, &te)) {
+					for (j = 0; j < L - nk; j++) { Cell v = m[i + j] >> 1; vd--; if (v >= -2147483647 - 1 && v <= 2147483647) mov_mi(rSP, SL(0), (int)v); else { mov_ri(r_ax, v); stv(0, r_ax); } }
+					if (nused < 64) used[nused++] = s; else used[0] = -1;
+					x64_enh(&te);
+					free(te.u); free(te.reg);
+					i += L + 1; n += L;
+					continue;
+				}
+			}
+		}
 		if (w & 1) {
 			Cell v = (w - 1) / 2;
 			int small = v >= -2147483647 - 1 && v <= 2147483647, ko = -1;
@@ -6128,6 +6294,10 @@ static void *translate(int p)
 		}
 		if (k >= XOP0 && k < XOP0 + NENH) {
 			if (nused < 64) used[nused++] = k - XOP0; else used[0] = -1;	/* wired or not, an install makes this a different block */
+			if (enh[k - XOP0].live && cp - start < (1 << 18) && !x64_noenh) {	/* inline */
+				Enh te;
+				if (!x64_spec_enh(k - XOP0, 0, 0, &te)) { x64_enh(&te); free(te.u); free(te.reg); i++; continue; }
+			}
 			if (enh[k - XOP0].live) {	/* the interpreter runs it: x0 = data stack, slot */
 				lea(r_di, rSP, -1, 1, 8 * vd); mov_r32i(r_si, (unsigned)(k - XOP0));
 				call_m(rST, 80);
@@ -6245,6 +6415,7 @@ static int x64_boot(Cell sp, Cell rp, void *stop)
 	code = mmap(0, CODEW * 4, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (code == MAP_FAILED) { perror("mmap"); return 1; }
 	x64_noo3 = getenv("ZYNOO3") != 0;
+	x64_noenh = getenv("ZYNOENH") != 0;
 	jitwrite(1); build_stubs(); jitwrite(0); syncicache(code, cp);
 	S[0] = (long)m; S[1] = sp; S[2] = rp; S[3] = (long)tab; S[4] = (long)cov;
 	S[5] = (long)Q; S[6] = (long)G; S[7] = (long)&J; S[8] = (long)hook;
diff --git a/src/runtime/rt.zc b/src/runtime/rt.zc
index 7089e26..f1414e4 100644
--- a/src/runtime/rt.zc
+++ b/src/runtime/rt.zc
@@ -87,6 +87,10 @@ enhance ishl(a, b) micro "a = shl a b";
 enhance lfa(k) micro "r1 = ld 589824; k = add r1 k";
 // 1 when i is outside 0..n-1 (the unsigned compare also catches negative i), else 0.
 enhance outside(i, n) micro "i = ltu i n; i = xor i 1";
+//: the block length has exactly two definitions, which must agree: the r7 lines of this
+//: recipe (the inline check on every backend; the x86-64 and arm64 jits lower it to
+//: registers, with nothing pointer-specific of their own) and paylen below (bchk's report).
+//: an exact-size check or a new header layout changes those two places and nothing else.
 // the index check of a pointer, as one instruction: 1 when p is the start of a heap
 // block (its check word p[-3] is p ^ MAGIC) and the byte or word offset i * m is not
 // inside the block's payload (q units per word: 1 for word elements, 8 for packed
diff --git a/tests/parity.known b/tests/parity.known
index 6892499..5ddbdf2 100644
--- a/tests/parity.known
+++ b/tests/parity.known
@@ -16,3 +16,4 @@ tests/programs/bench_strings.zc # reads the repeat count from stdin; with none,
 # zc0 reads unbounded and ignores it, and emits no index checks at all
 tests/programs/bounds_fixed.zc # bounds checks: only the self-hosted compiler checks a fixed array index
 tests/programs/bounds_heap.zc # bounds checks: only the self-hosted compiler checks a pointer into a heap block
+tests/programs/bounds_proof.zc # bounds checks: zc0 emits none, so the variable-bound loop is as short as the unbounded one
diff --git a/tests/programs/bounds_proof.zc b/tests/programs/bounds_proof.zc
new file mode 100644
index 0000000..a678fb3
--- /dev/null
+++ b/tests/programs/bounds_proof.zc
@@ -0,0 +1,21 @@
+namespace;
+include rt;
+
+// a loop test i < K with a constant K proves the index of a fixed array of at
+// least K elements, so the bounds check is dropped: the checked loop compiles
+// to exactly as many words as the unbounded one. a variable bound proves
+// nothing and keeps the check. sizes come from the functions' line blocks.
+int[8] t;
+int konst() { int i, s = 0; for (i = 0; i < 8; i++) s = s + t[i]; return s; }
+unbounded int bare() { int i, s = 0; for (i = 0; i < 8; i++) s = s + t[i]; return s; }
+int vary(int n) { int i, s = 0; for (i = 0; i < n; i++) s = s + t[i]; return s; }
+unbounded int vbare(int n) { int i, s = 0; for (i = 0; i < n; i++) s = s + t[i]; return s; }
+
+int size(word f) { word b = rt.line_block_of(f); return b[2] - b[1]; }
+
+int main() {
+	rt.put_n(konst() + bare() + vary(8) + vbare(8)); rt.put_c(10);
+	rt.put_n(size(konst) == size(bare)); rt.put_c(10);
+	rt.put_n(size(vary) > size(vbare)); rt.put_c(10);
+	return 0;
+}
diff --git a/tests/programs/enh_ops.expected b/tests/programs/enh_ops.expected
new file mode 100644
index 0000000..6549235
--- /dev/null
+++ b/tests/programs/enh_ops.expected
@@ -0,0 +1,34 @@
+add 882028350189717406
+sub 6655513972688108814
+mul 8202745059402640650
+sdiv -7272167455729189346
+udiv 2339759234614203099
+srem -1116402929106219647
+urem 9047360979782569854
+and 6728770653513201046
+or -5846742303323483640
+xor 5871231116872866930
+shl -343427929681503244
+shr -5759255359902579034
+sar -6075979569754238740
+lt -8494581876920277443
+ltu 5806102239654568498
+eq -7243535849964624607
+ne 2622739368221699999
+min 2867643923396239862
+max -1985615573206522456
+minu -7534928991011776213
+maxu 8416957341201493619
+mulhs -4206753455779030529
+mulhu -2870129158516768804
+rotl -6540633553243608262
+rotr -7155384646341566258
+neg -5182646291052210387
+not 5851949150302311436
+clz -7956734789639098207
+ctz 4862650619482391916
+abs 6757983799103834145
+bswap 5143948399074944670
+sel -578455337847416602
+ld st 71455
+mix -2049273682776724457
diff --git a/tests/programs/enh_ops.zc b/tests/programs/enh_ops.zc
new file mode 100644
index 0000000..da3cc09
--- /dev/null
+++ b/tests/programs/enh_ops.zc
@@ -0,0 +1,148 @@
+namespace;
+include rt;
+
+// every micro-op the x86-64 jit lowers inline, on edge values (0, -1, the
+// minimum, shift counts past 63, out-of-range loads), called with variables
+// and with literals (which specialize the enhancement on them). the output must
+// be the same on every backend; tests/drivers/jit_x64_test.sh compares it
+// against the interpreter.
+enhance e_add(a, b) micro "r0 = add a b";
+enhance e_sub(a, b) micro "r0 = sub a b";
+enhance e_mul(a, b) micro "r0 = mul a b";
+enhance e_sdiv(a, b) micro "r0 = sdiv a b";
+enhance e_udiv(a, b) micro "r0 = udiv a b";
+enhance e_srem(a, b) micro "r0 = srem a b";
+enhance e_urem(a, b) micro "r0 = urem a b";
+enhance e_and(a, b) micro "r0 = and a b";
+enhance e_or(a, b) micro "r0 = or a b";
+enhance e_xor(a, b) micro "r0 = xor a b";
+enhance e_shl(a, b) micro "r0 = shl a b";
+enhance e_shr(a, b) micro "r0 = shr a b";
+enhance e_sar(a, b) micro "r0 = sar a b";
+enhance e_lt(a, b) micro "r0 = lt a b";
+enhance e_ltu(a, b) micro "r0 = ltu a b";
+enhance e_eq(a, b) micro "r0 = eq a b";
+enhance e_ne(a, b) micro "r0 = ne a b";
+enhance e_min(a, b) micro "r0 = min a b";
+enhance e_max(a, b) micro "r0 = max a b";
+enhance e_minu(a, b) micro "r0 = minu a b";
+enhance e_maxu(a, b) micro "r0 = maxu a b";
+enhance e_mulhs(a, b) micro "r0 = mulhs a b";
+enhance e_mulhu(a, b) micro "r0 = mulhu a b";
+enhance e_rotl(a, b) micro "r0 = rotl a b";
+enhance e_rotr(a, b) micro "r0 = rotr a b";
+enhance u_neg(a) micro "r0 = neg a";
+enhance u_not(a) micro "r0 = not a";
+enhance u_clz(a) micro "r0 = clz a";
+enhance u_ctz(a) micro "r0 = ctz a";
+enhance u_abs(a) micro "r0 = abs a";
+enhance u_bswap(a) micro "r0 = bswap a";
+enhance e_sel(a, b, c) micro "r0 = sel a b c";
+enhance e_ld(a) micro "r0 = ld a";
+enhance e_st(a, b) micro "st a b; r0 = ld a";
+enhance e_mix(a, b) micro "r2 = mul a 3; r3 = add r2 b; r4 = sdiv r3 7; r5 = minu r4 b; r0 = xor r5 r2";
+
+int[12] v;
+int h;
+int mix(int r) { h = h * 31 + r; return 0; }
+int show(char* name) { rt.put_s(name); rt.put_c(32); rt.put_n(h); rt.put_c(10); h = 0; return 0; }
+
+unchecked int main() {
+	int i, j, a, b; word g = rt.new(4);
+	v[0] = 0; v[1] = 1; v[2] = 0 - 1; v[3] = 1 << 63; v[4] = (1 << 63) - 1; v[5] = 7; v[6] = 0 - 7; v[7] = 63; v[8] = 64; v[9] = 65; v[10] = 100; v[11] = 0x123456789;
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_add(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_add(a, 0)); mix(e_add(a, 0 - 1)); mix(e_add(a, 3)); mix(e_add(a, 64)); mix(e_add(a, 0x7fffffff)); }
+	show("add");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_sub(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_sub(a, 0)); mix(e_sub(a, 0 - 1)); mix(e_sub(a, 3)); mix(e_sub(a, 64)); mix(e_sub(a, 0x7fffffff)); }
+	show("sub");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_mul(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_mul(a, 0)); mix(e_mul(a, 0 - 1)); mix(e_mul(a, 3)); mix(e_mul(a, 64)); mix(e_mul(a, 0x7fffffff)); }
+	show("mul");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_sdiv(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_sdiv(a, 0)); mix(e_sdiv(a, 0 - 1)); mix(e_sdiv(a, 3)); mix(e_sdiv(a, 64)); mix(e_sdiv(a, 0x7fffffff)); }
+	show("sdiv");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_udiv(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_udiv(a, 0)); mix(e_udiv(a, 0 - 1)); mix(e_udiv(a, 3)); mix(e_udiv(a, 64)); mix(e_udiv(a, 0x7fffffff)); }
+	show("udiv");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_srem(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_srem(a, 0)); mix(e_srem(a, 0 - 1)); mix(e_srem(a, 3)); mix(e_srem(a, 64)); mix(e_srem(a, 0x7fffffff)); }
+	show("srem");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_urem(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_urem(a, 0)); mix(e_urem(a, 0 - 1)); mix(e_urem(a, 3)); mix(e_urem(a, 64)); mix(e_urem(a, 0x7fffffff)); }
+	show("urem");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_and(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_and(a, 0)); mix(e_and(a, 0 - 1)); mix(e_and(a, 3)); mix(e_and(a, 64)); mix(e_and(a, 0x7fffffff)); }
+	show("and");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_or(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_or(a, 0)); mix(e_or(a, 0 - 1)); mix(e_or(a, 3)); mix(e_or(a, 64)); mix(e_or(a, 0x7fffffff)); }
+	show("or");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_xor(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_xor(a, 0)); mix(e_xor(a, 0 - 1)); mix(e_xor(a, 3)); mix(e_xor(a, 64)); mix(e_xor(a, 0x7fffffff)); }
+	show("xor");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_shl(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_shl(a, 0)); mix(e_shl(a, 0 - 1)); mix(e_shl(a, 3)); mix(e_shl(a, 64)); mix(e_shl(a, 0x7fffffff)); }
+	show("shl");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_shr(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_shr(a, 0)); mix(e_shr(a, 0 - 1)); mix(e_shr(a, 3)); mix(e_shr(a, 64)); mix(e_shr(a, 0x7fffffff)); }
+	show("shr");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_sar(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_sar(a, 0)); mix(e_sar(a, 0 - 1)); mix(e_sar(a, 3)); mix(e_sar(a, 64)); mix(e_sar(a, 0x7fffffff)); }
+	show("sar");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_lt(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_lt(a, 0)); mix(e_lt(a, 0 - 1)); mix(e_lt(a, 3)); mix(e_lt(a, 64)); mix(e_lt(a, 0x7fffffff)); }
+	show("lt");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_ltu(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_ltu(a, 0)); mix(e_ltu(a, 0 - 1)); mix(e_ltu(a, 3)); mix(e_ltu(a, 64)); mix(e_ltu(a, 0x7fffffff)); }
+	show("ltu");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_eq(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_eq(a, 0)); mix(e_eq(a, 0 - 1)); mix(e_eq(a, 3)); mix(e_eq(a, 64)); mix(e_eq(a, 0x7fffffff)); }
+	show("eq");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_ne(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_ne(a, 0)); mix(e_ne(a, 0 - 1)); mix(e_ne(a, 3)); mix(e_ne(a, 64)); mix(e_ne(a, 0x7fffffff)); }
+	show("ne");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_min(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_min(a, 0)); mix(e_min(a, 0 - 1)); mix(e_min(a, 3)); mix(e_min(a, 64)); mix(e_min(a, 0x7fffffff)); }
+	show("min");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_max(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_max(a, 0)); mix(e_max(a, 0 - 1)); mix(e_max(a, 3)); mix(e_max(a, 64)); mix(e_max(a, 0x7fffffff)); }
+	show("max");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_minu(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_minu(a, 0)); mix(e_minu(a, 0 - 1)); mix(e_minu(a, 3)); mix(e_minu(a, 64)); mix(e_minu(a, 0x7fffffff)); }
+	show("minu");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_maxu(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_maxu(a, 0)); mix(e_maxu(a, 0 - 1)); mix(e_maxu(a, 3)); mix(e_maxu(a, 64)); mix(e_maxu(a, 0x7fffffff)); }
+	show("maxu");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_mulhs(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_mulhs(a, 0)); mix(e_mulhs(a, 0 - 1)); mix(e_mulhs(a, 3)); mix(e_mulhs(a, 64)); mix(e_mulhs(a, 0x7fffffff)); }
+	show("mulhs");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_mulhu(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_mulhu(a, 0)); mix(e_mulhu(a, 0 - 1)); mix(e_mulhu(a, 3)); mix(e_mulhu(a, 64)); mix(e_mulhu(a, 0x7fffffff)); }
+	show("mulhu");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_rotl(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_rotl(a, 0)); mix(e_rotl(a, 0 - 1)); mix(e_rotl(a, 3)); mix(e_rotl(a, 64)); mix(e_rotl(a, 0x7fffffff)); }
+	show("rotl");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_rotr(a, b)); }
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_rotr(a, 0)); mix(e_rotr(a, 0 - 1)); mix(e_rotr(a, 3)); mix(e_rotr(a, 64)); mix(e_rotr(a, 0x7fffffff)); }
+	show("rotr");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_neg(a)); } mix(u_neg(0)); mix(u_neg(0 - 1)); mix(u_neg(8));
+	show("neg");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_not(a)); } mix(u_not(0)); mix(u_not(0 - 1)); mix(u_not(8));
+	show("not");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_clz(a)); } mix(u_clz(0)); mix(u_clz(0 - 1)); mix(u_clz(8));
+	show("clz");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_ctz(a)); } mix(u_ctz(0)); mix(u_ctz(0 - 1)); mix(u_ctz(8));
+	show("ctz");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_abs(a)); } mix(u_abs(0)); mix(u_abs(0 - 1)); mix(u_abs(8));
+	show("abs");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(u_bswap(a)); } mix(u_bswap(0)); mix(u_bswap(0 - 1)); mix(u_bswap(8));
+	show("bswap");
+	for (i = 0; i < 12; i++) { a = v[i]; mix(e_sel(a, 5, 9)); mix(e_sel(a, a, 0 - a)); mix(e_sel(0, a, 2)); }
+	show("sel");
+	mix(e_ld(0 - 1)); mix(e_ld(1 << 62)); a = g; mix(e_ld(a) == g[0]); mix(e_st(a, 42)); mix(g[0]); mix(e_st(0 - 5, 1));
+	show("ld st");
+	for (i = 0; i < 12; i++) for (j = 0; j < 12; j++) { a = v[i]; b = v[j]; mix(e_mix(a, b)); mix(e_mix(a, 5)); }
+	show("mix");
+	rt.free(g);
+	return 0;
+}
diff --git a/tests/test.sh b/tests/test.sh
index 7a80c3c..cdce083 100755
--- a/tests/test.sh
+++ b/tests/test.sh
@@ -227,6 +227,8 @@ ABC" "$l o3 lifts byte loads and stores"
 	chk "$($vm < tests/programs/boundsrules.zc 2>&1 | head -3)" $'5\n7\nzy: index 5 out of range (length 4)' "$l unbounded as prefix, block and statement, o3 implies it; checked code reports"
 	chk "$($vm < tests/programs/bounds_fixed.zc 2>&1 | head -3)" $'55\n29\nzy: index 6 out of range (length 6)' "$l a fixed array with a variable index is checked against its length"
 	chk "$($vm < tests/programs/bounds_heap.zc 2>&1 | head -4)" $'3\n0\nx\nzy: index 11 out of range (length 11)' "$l a pointer to a heap block is checked against its payload"
+	chk "$($vm < tests/programs/enh_ops.zc | md5sum)" "$(md5sum < tests/programs/enh_ops.expected)" "$l enhancement micro-ops on edge values, with variables and with literals"
+	chk "$($vm < tests/programs/bounds_proof.zc)" $'0\n1\n1' "$l a loop test against a constant drops the fixed-array check; a variable bound keeps it"
 	chk "$($vm < tests/programs/zc0unbounded.zc)" "42" "$l unbounded in the unit, a prefix and a statement"
 	local r; r=$(bash tests/runerr.sh "$vm" tests/errors | tail -1)
 	chk "$r" "rejected: $(ls tests/errors/*.zc | wc -l | tr -d ' ') passed, 0 failed" "$l analyzer rejects every bad program"
@@ -500,6 +502,15 @@ s_jit() {
 			chk "$(cd $T && qemu-aarch64 -L /usr/aarch64-linux-gnu $T/zyjd < $R/tests/programs/structret.zc)" "3 4 3.500000 4.000000 24 8 1.000000 3.500000" "jit structs returned from a c function (aapcs64 results)"
 		fi
 	else echo "skip jit (no aarch64 toolchain or qemu)"; fi
+	# the x86-64 jit against the interpreter, every program (about a minute; not in the quick path)
+	if [ "$(uname -s)-$(uname -m)" = Linux-x86_64 ] && [ $QUICK = 0 ]; then
+		$CC -pthread -O2 -w -fwrapv -DJTYPE=int -DZYINTERP src/core/core.c src/core/io_linux.c $GI -o $T/zyi || exit 1
+		$CC -pthread -O2 -w -fwrapv -DJTYPE=int src/core/core.c src/core/io_linux.c $GI -o $T/zyx || exit 1
+		bash tests/drivers/jit_x64_test.sh $T/zyi $T/zyx > $T/jit_x64.out 2>&1; r=$?
+		grep '^DIFF' $T/jit_x64.out
+		chk $r 0 "x86-64 jit agrees with the interpreter: $(tail -n 1 $T/jit_x64.out)"
+		chk "$(ZYNOENH=1 $T/zyx < tests/programs/enh_ops.zc | md5sum)" "$(md5sum < tests/programs/enh_ops.expected)" "x86-64 jit with enhancements called through xrun (ZYNOENH=1)"
+	fi
 }
 
 s_llvm() {