From f13f84044be1bd18160f4845a739592219638862 Mon Sep 17 00:00:00 2001 From: zinc Date: Wed, 14 Oct 2026 09:28:59 +0000 Subject: docs: remove temporary notes and fix onboarding drift - delete TASKLIST.md and docs/design/handoff.md; the handoff content lives in README.md - move docs/BOOTSTRAP-OWNERSHIP.md to docs/history/ and index it in docs/history/README.md - point ONBOARD.md, docs/BORROW.md and the ticked TODO.md item at the new path - ONBOARD.md section 9 no longer mentions TASKLIST.md - ONBOARD.md section 3 and 4 markers are now `// ...` (line comments lost their colon) verified from a clean build: make clean, make (fixpoint 376903 words), the section 0-7 commands (section 4 after the marker fix), and the rejection suites. make check was started but did not complete (stopped before it finished). --- docs/BOOTSTRAP-OWNERSHIP.md | 183 ------------------------------------ docs/BORROW.md | 2 +- docs/design/handoff.md | 20 ---- docs/history/BOOTSTRAP-OWNERSHIP.md | 183 ++++++++++++++++++++++++++++++++++++ docs/history/README.md | 1 + 5 files changed, 185 insertions(+), 204 deletions(-) delete mode 100644 docs/BOOTSTRAP-OWNERSHIP.md delete mode 100644 docs/design/handoff.md create mode 100644 docs/history/BOOTSTRAP-OWNERSHIP.md (limited to 'docs') diff --git a/docs/BOOTSTRAP-OWNERSHIP.md b/docs/BOOTSTRAP-OWNERSHIP.md deleted file mode 100644 index 6cc61ce..0000000 --- a/docs/BOOTSTRAP-OWNERSHIP.md +++ /dev/null @@ -1,183 +0,0 @@ -# the bootstrap did not compile: owned values could not be moved between locals - -status: resolved. the analyzer gained the two rules proposed below and a clearer -recast diagnostic, and `make genome` reaches its fixpoint from a clean checkout. -see "resolution" at the end; the rest of this note is the record of the problem. -affects: `make genome` and the `fixpoint` check -see also: `docs/BORROW.md` - -## summary - -`make` bootstraps the compiler with `zc0` compiling `src/runtime/rt.zc`, -`src/core/core.zc` and `src/compiler/compiler.zc` into a stage-0 image, stage-0 -becoming `build/zy0`, and `zy0` compiling the same three files again into the -genome. the second compilation runs with lifetime checking enabled and rejects -the compiler's own source. `mkgenome` then read the diagnostics as if they -were an image: - -``` -mkgenome: input is not a genome for this map -``` - -the real output of `zy0` is a diagnostic list, not an image. nothing after the -fixpoint can run, so a checkout with no prebuilt genome cannot be built. - -this is a completeness gap in the ownership rules, not a soundness failure. the -checker is refusing correct programs; it is not accepting unsafe ones. the -practical cost is that the compiler can no longer compile itself. - -## the affected code - -`methdef` in `src/compiler/compiler.zc` reassembles a typed generic method's -token list. it saves the output stream state, redirects the stream at a scratch -buffer, renders the return type into it, splits the result into tokens, builds a -replacement buffer that interleaves the new tokens with the old ones, and hands -the replacement back in place of the original. - -```zc -word output = rt.outbuf, grow = rt.outgrow, buffer = rt.new(4096), return_tokens, joined; -rt.outmode = 2; rt.outbuf = m*buffer; rt.outn = 0; rt.outcap = 4095; rt.outgrow = 0; -tprint(rty); if (rtag) rt.put_s(".u"); buffer[rt.outn] = 0; -rt.outmode = mode; rt.outbuf = output; rt.outn = length; rt.outcap = capacity; rt.outgrow = grow; -return_tokens = tokens(buffer); -... -joined = rt.new((r[0] + return_tokens[0]) * 3 + 1); -... -word previous = m*r; r = d*joined; rt.free(return_tokens); rt.free(buffer); rt.free(previous); -``` - -three separate moves are needed here, and each one hits a different limit: - -1. lend the local scratch buffer to the output stream, then keep using it -2. replace `r`, an owned local, with `joined`, another owned local -3. store the local block into the global code-block list - -## what the checker reports - -``` -zc: line 5571: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) -zc: line 5573: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) -zc: line 5589: r: is an owned object and can't be copied into another variable -zc: line 5589: r: still owns an object; overwriting it would leak it -zc: line 5589: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) -zc: line 5590: joined: d object was never given away -zc: line 10572: only a named owned object can be recast -zc: line 10575: only a named owned object can be recast -zc: 8 errors -``` - -the line numbers are into the concatenation, not into `compiler.zc`. subtract -1401 lines: 5570 is `rt.outbuf = m*buffer;`, 5589 is the `previous` line, and -10572 and 10575 are `rt.code_blocks = m*block;` in `syntax_resident`. - -## what was tried - -| change | result | -|---|---| -| `rt.outbuf = buffer;` — drop the `m*` | still reported as a transfer; any owned value assigned into a global transfers, and a second diagnostic appeared at the following line | -| move the scratch buffer to a module global `rbuf`, assign it plainly, borrow it into `rt.outbuf` | accepted; removes five of the eight errors. this is the existing idiom, see `sgb` at `sig_attach` and `ltxt` in the literal pool | -| `word previous = m*r; r = d*joined; rt.free(previous);` unchanged | `initial declarations still cannot copy an owned local`, per BORROW.md line 30 | -| declare `previous` in the block's declaration list, then `previous = m*r;` | `r: is an owned object and can't be copied into another variable` | -| `rt.free(r); r = joined;` — the documented free-then-replace idiom | `joined: is an owned object and can't be copied into another variable`, and `r` is then reported as given away at its next use | - -## is this a borrow checker issue - -yes, and it is worth separating two kinds of gap. - -**a move between two locals has no syntax.** the accepted moves in the -compiler's own source all originate at a call: - -```zc -word p = m*rt.new(capacity); // src/compiler/compiler.zc:644 -buf = m*inc_open(full); // src/compiler/compiler.zc:4632 -dgbuf = chk_resize(dgbuf, dgpos, cap) // src/compiler/compiler.zc:401 -rt.free(d*buf); // src/compiler/compiler.zc:4447 -``` - -an owned value already held in a local can be disposed with `d*`, but it cannot -be handed to another local, and it cannot replace one. the free-then-replace rule -in BORROW.md line 30 covers `rt.free(buffer); buffer = replacement;` — but only -when `replacement` arrives from a call. with a local on the right-hand side the -same rule rejects the assignment and then reports the destination as given away. - -the replacement built here is a genuine local: it is written field by field and -then has to become `r`. no current spelling expresses that. the shortest -workaround is to route the swap through a call that takes both pointers and -returns the survivor, so the move originates at a call result. - -**a local cannot be lent to a global.** the output stream is global state. -redirecting it at a local scratch buffer is a borrow, not a transfer, but every -spelling is a transfer: - -- `rt.outbuf = m*buffer;` transfers, so every later use of `buffer` is a use - after transfer -- `rt.outbuf = buffer;` also transfers -- the three working redirects in the compiler all assign a global or a pointer - derived from one — `rt.outbuf = dgbuf + dgpos + 5;`, `rt.outbuf = sgb;`, - `rt.outbuf = ift + at;`, `rt.outbuf = ltxt;` - -so the scratch buffer has to live at module scope. that is workable here, and it -is what the module-global `rbuf` change adopts, but it costs something the -local version did not: the buffer is shared state for the duration of the call, -so any reentrancy through `methdef` would collide, and the checker cannot see -that the borrow ends at the restore. - -## proposal - -in rough order of value: - -1. allow a move into an existing local when the source is an owned local and the - destination's previous owner is released in the same statement list, generalizing - the free-then-replace rule beyond call results. this alone resolves the - `methdef` reassembly. -2. allow a global to hold a borrow of a local for a bounded region, so an output - stream can be pointed at a scratch buffer without a module-level variable. a - scoped-borrow rule on assignment into a global field would do it, and it removes - the reentrancy hazard that the module-global workaround introduces. -3. give `rt.code_blocks = m*block;` a supported spelling. the error, `only a named - owned object can be recast`, does not describe what is wrong: `block` is named and - owned. it is used again at `here = block + 3;` two lines later, which is the real - constraint, and the diagnostic should say so and point at the later use. - -none of these accept a program that is currently accepted unsafely. each widens -the set of programs the checker can describe. item 3 is also a message bug -independent of the rule change: a wrong diagnostic on a correct-looking transfer. - -## reproducing - -`src/tools/bootstrap-check.sh` rebuilds the two bootstrap stages and prints the -compiler's diagnostics on its own source instead of the word dump. it exits -through `mkgenome` when the image is malformed, which is the same failure `make` -reports. - -``` -bash src/tools/bootstrap-check.sh -``` - -a passing run prints `image ok: bytes`. - -## resolution - -the analyzer (`src/compiler/compiler.zc`) now accepts the three spellings: - -1. **moves into a released owner.** `rt.free(r); r = joined;` moves `joined` into `r` - when `r` was released first and `joined` is an owner of the same block or a deeper - one. `joined` is retired by the move. a destination that still owns something is - still rejected (`still owns an object; overwriting it would leak it`), and so is a - copy of an owner into a fresh declaration. -2. **bounded lends to globals.** `g = m*x;` with `x` an owned local lends `x` to `g`: - `x` keeps ownership and stays usable, and the lend ends when the same block assigns - `g` again. giving `x` away, returning, breaking or leaving the block while the lend - is open is an error (`is lent to global g; assign the global again before ...`). the - lend list is per function and per block depth (`lend_add`, `lend_clear`, - `lend_exit`), so a restore inside a nested branch does not end a lend made outside it. -3. **recasts of plain locals.** `rt.code_blocks = m*block;` failed because `block` came - from `chk_resize`, which returns collector-managed storage, so `block` never owned - anything. the compiler now writes `rt.code_blocks = block;`, and a recast of a plain - local reports `block: isn't an owned object, so there is no ownership to recast` - instead of the old message. - -`methdef` uses the lend for its scratch buffer and the move for the joined token list. -the regressions are `tests/programs/ownmove.zc` and `tests/errors/lend_*.zc`, -`recast_plain.zc` and `move_live_owner.zc`. - diff --git a/docs/BORROW.md b/docs/BORROW.md index f270574..64a3183 100644 --- a/docs/BORROW.md +++ b/docs/BORROW.md @@ -45,7 +45,7 @@ the compiler functions, standard library, and language server currently run with the analysis runs with the existing parser, ownership checker, and lifetime records; it does not have a separate source parser or a complete path-sensitive proof. snapshots conservatively join branches and loops. low-level and unknown boundaries can hide behavior, so a quiet diagnostic set is not a whole-program safety proof. zc is single-threaded; shared closure cells do not define concurrent mutation rules. -two completeness limits once stopped the compiler from compiling itself: an owner held in a local could not move into another local, and a local could not be lent to a global. the replacement and lend rules above close them; `docs/BOOTSTRAP-OWNERSHIP.md` keeps the record. +two completeness limits once stopped the compiler from compiling itself: an owner held in a local could not move into another local, and a local could not be lent to a global. the replacement and lend rules above close them; `docs/history/BOOTSTRAP-OWNERSHIP.md` keeps the record. the maintained regressions cover these cases: diff --git a/docs/design/handoff.md b/docs/design/handoff.md deleted file mode 100644 index 0b08048..0000000 --- a/docs/design/handoff.md +++ /dev/null @@ -1,20 +0,0 @@ -# implementation notes - -this note summarizes implemented language and tooling features that were previously tracked across design documents. feature details live in the project readme and examples. - -## language and library - -- struct methods and static functions support typed parameters and return types. the existing untyped form remains available. -- `src/lib/std.zc` groups helpers in structs of static functions, called with `Group.name(args)`. `Text`, `Bytes`, `Host`, `Arena` and `Buf` provide string, byte, host, arena and buffer helpers. linux `syscall` is host-only and bypasses vm checks. -- tagged enums support payload members, constructors that return owned enum pointers, and `case member(fields)` bindings in ordinary `switch` statements. omitted members produce a warning when there is no `default`. -- struct values can cross native calls on x86-64 sysv and linux arm64 aapcs64 for the interpreter, o3, llvm and jit paths described in the readme. apple arm64 remains unchecked. - -## language server - -the language server runs the compiler in check mode, so diagnostics and semantic highlighting use the same parser and symbol tables as compilation. hover supports declarations, parameters, locals, built-ins and documented keywords. it does not follow declarations reached through `include` yet. - -## optimization - -o3 remains the explicit way to request native regions; automatic tiering was dropped. llvm inlining and global copy propagation are implemented. see [tiering](tiering.md) for the rationale and [struct-by-value](struct-by-value.md) for native aggregate limits. - -the runtime memory map is generated from a preset. fixed code and compiler tables have finite capacity; the heap grows within the configured machine memory. see the project readme's memory map section for the current build controls. diff --git a/docs/history/BOOTSTRAP-OWNERSHIP.md b/docs/history/BOOTSTRAP-OWNERSHIP.md new file mode 100644 index 0000000..6cc61ce --- /dev/null +++ b/docs/history/BOOTSTRAP-OWNERSHIP.md @@ -0,0 +1,183 @@ +# the bootstrap did not compile: owned values could not be moved between locals + +status: resolved. the analyzer gained the two rules proposed below and a clearer +recast diagnostic, and `make genome` reaches its fixpoint from a clean checkout. +see "resolution" at the end; the rest of this note is the record of the problem. +affects: `make genome` and the `fixpoint` check +see also: `docs/BORROW.md` + +## summary + +`make` bootstraps the compiler with `zc0` compiling `src/runtime/rt.zc`, +`src/core/core.zc` and `src/compiler/compiler.zc` into a stage-0 image, stage-0 +becoming `build/zy0`, and `zy0` compiling the same three files again into the +genome. the second compilation runs with lifetime checking enabled and rejects +the compiler's own source. `mkgenome` then read the diagnostics as if they +were an image: + +``` +mkgenome: input is not a genome for this map +``` + +the real output of `zy0` is a diagnostic list, not an image. nothing after the +fixpoint can run, so a checkout with no prebuilt genome cannot be built. + +this is a completeness gap in the ownership rules, not a soundness failure. the +checker is refusing correct programs; it is not accepting unsafe ones. the +practical cost is that the compiler can no longer compile itself. + +## the affected code + +`methdef` in `src/compiler/compiler.zc` reassembles a typed generic method's +token list. it saves the output stream state, redirects the stream at a scratch +buffer, renders the return type into it, splits the result into tokens, builds a +replacement buffer that interleaves the new tokens with the old ones, and hands +the replacement back in place of the original. + +```zc +word output = rt.outbuf, grow = rt.outgrow, buffer = rt.new(4096), return_tokens, joined; +rt.outmode = 2; rt.outbuf = m*buffer; rt.outn = 0; rt.outcap = 4095; rt.outgrow = 0; +tprint(rty); if (rtag) rt.put_s(".u"); buffer[rt.outn] = 0; +rt.outmode = mode; rt.outbuf = output; rt.outn = length; rt.outcap = capacity; rt.outgrow = grow; +return_tokens = tokens(buffer); +... +joined = rt.new((r[0] + return_tokens[0]) * 3 + 1); +... +word previous = m*r; r = d*joined; rt.free(return_tokens); rt.free(buffer); rt.free(previous); +``` + +three separate moves are needed here, and each one hits a different limit: + +1. lend the local scratch buffer to the output stream, then keep using it +2. replace `r`, an owned local, with `joined`, another owned local +3. store the local block into the global code-block list + +## what the checker reports + +``` +zc: line 5571: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) +zc: line 5573: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) +zc: line 5589: r: is an owned object and can't be copied into another variable +zc: line 5589: r: still owns an object; overwriting it would leak it +zc: line 5589: buffer: used after it was given away; owner buffer was transferred on line 5570 (declared on line 5569) +zc: line 5590: joined: d object was never given away +zc: line 10572: only a named owned object can be recast +zc: line 10575: only a named owned object can be recast +zc: 8 errors +``` + +the line numbers are into the concatenation, not into `compiler.zc`. subtract +1401 lines: 5570 is `rt.outbuf = m*buffer;`, 5589 is the `previous` line, and +10572 and 10575 are `rt.code_blocks = m*block;` in `syntax_resident`. + +## what was tried + +| change | result | +|---|---| +| `rt.outbuf = buffer;` — drop the `m*` | still reported as a transfer; any owned value assigned into a global transfers, and a second diagnostic appeared at the following line | +| move the scratch buffer to a module global `rbuf`, assign it plainly, borrow it into `rt.outbuf` | accepted; removes five of the eight errors. this is the existing idiom, see `sgb` at `sig_attach` and `ltxt` in the literal pool | +| `word previous = m*r; r = d*joined; rt.free(previous);` unchanged | `initial declarations still cannot copy an owned local`, per BORROW.md line 30 | +| declare `previous` in the block's declaration list, then `previous = m*r;` | `r: is an owned object and can't be copied into another variable` | +| `rt.free(r); r = joined;` — the documented free-then-replace idiom | `joined: is an owned object and can't be copied into another variable`, and `r` is then reported as given away at its next use | + +## is this a borrow checker issue + +yes, and it is worth separating two kinds of gap. + +**a move between two locals has no syntax.** the accepted moves in the +compiler's own source all originate at a call: + +```zc +word p = m*rt.new(capacity); // src/compiler/compiler.zc:644 +buf = m*inc_open(full); // src/compiler/compiler.zc:4632 +dgbuf = chk_resize(dgbuf, dgpos, cap) // src/compiler/compiler.zc:401 +rt.free(d*buf); // src/compiler/compiler.zc:4447 +``` + +an owned value already held in a local can be disposed with `d*`, but it cannot +be handed to another local, and it cannot replace one. the free-then-replace rule +in BORROW.md line 30 covers `rt.free(buffer); buffer = replacement;` — but only +when `replacement` arrives from a call. with a local on the right-hand side the +same rule rejects the assignment and then reports the destination as given away. + +the replacement built here is a genuine local: it is written field by field and +then has to become `r`. no current spelling expresses that. the shortest +workaround is to route the swap through a call that takes both pointers and +returns the survivor, so the move originates at a call result. + +**a local cannot be lent to a global.** the output stream is global state. +redirecting it at a local scratch buffer is a borrow, not a transfer, but every +spelling is a transfer: + +- `rt.outbuf = m*buffer;` transfers, so every later use of `buffer` is a use + after transfer +- `rt.outbuf = buffer;` also transfers +- the three working redirects in the compiler all assign a global or a pointer + derived from one — `rt.outbuf = dgbuf + dgpos + 5;`, `rt.outbuf = sgb;`, + `rt.outbuf = ift + at;`, `rt.outbuf = ltxt;` + +so the scratch buffer has to live at module scope. that is workable here, and it +is what the module-global `rbuf` change adopts, but it costs something the +local version did not: the buffer is shared state for the duration of the call, +so any reentrancy through `methdef` would collide, and the checker cannot see +that the borrow ends at the restore. + +## proposal + +in rough order of value: + +1. allow a move into an existing local when the source is an owned local and the + destination's previous owner is released in the same statement list, generalizing + the free-then-replace rule beyond call results. this alone resolves the + `methdef` reassembly. +2. allow a global to hold a borrow of a local for a bounded region, so an output + stream can be pointed at a scratch buffer without a module-level variable. a + scoped-borrow rule on assignment into a global field would do it, and it removes + the reentrancy hazard that the module-global workaround introduces. +3. give `rt.code_blocks = m*block;` a supported spelling. the error, `only a named + owned object can be recast`, does not describe what is wrong: `block` is named and + owned. it is used again at `here = block + 3;` two lines later, which is the real + constraint, and the diagnostic should say so and point at the later use. + +none of these accept a program that is currently accepted unsafely. each widens +the set of programs the checker can describe. item 3 is also a message bug +independent of the rule change: a wrong diagnostic on a correct-looking transfer. + +## reproducing + +`src/tools/bootstrap-check.sh` rebuilds the two bootstrap stages and prints the +compiler's diagnostics on its own source instead of the word dump. it exits +through `mkgenome` when the image is malformed, which is the same failure `make` +reports. + +``` +bash src/tools/bootstrap-check.sh +``` + +a passing run prints `image ok: bytes`. + +## resolution + +the analyzer (`src/compiler/compiler.zc`) now accepts the three spellings: + +1. **moves into a released owner.** `rt.free(r); r = joined;` moves `joined` into `r` + when `r` was released first and `joined` is an owner of the same block or a deeper + one. `joined` is retired by the move. a destination that still owns something is + still rejected (`still owns an object; overwriting it would leak it`), and so is a + copy of an owner into a fresh declaration. +2. **bounded lends to globals.** `g = m*x;` with `x` an owned local lends `x` to `g`: + `x` keeps ownership and stays usable, and the lend ends when the same block assigns + `g` again. giving `x` away, returning, breaking or leaving the block while the lend + is open is an error (`is lent to global g; assign the global again before ...`). the + lend list is per function and per block depth (`lend_add`, `lend_clear`, + `lend_exit`), so a restore inside a nested branch does not end a lend made outside it. +3. **recasts of plain locals.** `rt.code_blocks = m*block;` failed because `block` came + from `chk_resize`, which returns collector-managed storage, so `block` never owned + anything. the compiler now writes `rt.code_blocks = block;`, and a recast of a plain + local reports `block: isn't an owned object, so there is no ownership to recast` + instead of the old message. + +`methdef` uses the lend for its scratch buffer and the move for the joined token list. +the regressions are `tests/programs/ownmove.zc` and `tests/errors/lend_*.zc`, +`recast_plain.zc` and `move_live_owner.zc`. + diff --git a/docs/history/README.md b/docs/history/README.md index ead4aa0..1a91d19 100644 --- a/docs/history/README.md +++ b/docs/history/README.md @@ -54,3 +54,4 @@ the code folders are snapshots. they're not built or tested from here, the top l - 51 llvm-inline-stdlib: llvm inlining of o3 calls, stdlib text and num helpers, host syscalls through SYS_RAW, and the fuzz campaign's findings - 52 closures: closure calls work (the block is never run; a call into a heap closure goes through its entry word), captures chain through enclosing closures, and the closure test joins the suite - 53 typed source: every function, local and global in the compiler, the runtime and the language server is typed (zc0's image does not change), and the extension caches diagnostics by a sha256 of the tree they cover +- BOOTSTRAP-OWNERSHIP.md: notes only (the two ownership rules that let the compiler compile itself, the partial fix before them, and the diagnostic that was reworded) -- cgit v1.2.3