1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
|
- [x] read the compiler, runtime, vm cores, language server, and build tools.
- [x] verify formatted markdown and code in vs code hovers, including bundled server builds.
- [x] add mandatory documentation for every built-in type.
- [x] use `/* ... */` comments as documentation, including comments for individual variables and arguments on the same line.
- [x] run focused tests between changes and avoid repeating the full suite.
- [x] check every core source file through the lsp document protocol with its expected source dependencies; reproduce and fix editor diagnostics, including those in the compiler (`src/compiler/parts`, checked as one document).
- [x] rename current i/o and string helpers consistently, including `puts` → `put_s`, `putn` → `put_n`, and `scat` → `cat_s`; update compiler references, examples, tests, and documentation.
- [x] review passing a temporary owned return to an output function; preserve the restriction if it is needed for lifetime safety, otherwise fix it with regression coverage.
- [x] draft `docs/BORROW.md` with a concrete plan for a less intrusive borrow checker and lifetime analyzer.
- [x] grow collector mark work without losing pending objects or counting compiler storage as user objects on the default map.
- [x] verify more than 64 generic declarations, more than 128 instantiations, more than 32 captures, and more than 8 nested closures on the default map.
- [x] move source files into appropriate subdirectories of `src/`, tests into subdirectories of `tests/`, and documentation and designs into `docs/`.
- [x] consolidate windows build, run, and test entry points into one interactive `tinyvm.bat`.
- [x] add a matching interactive `tinyvm.sh` for linux, with direct command mode.
- [x] remove stale planning references from current documentation and code.
- [x] consolidate useful memory-limit information, then remove `CEILINGS.md` if redundant.
- [x] reject stale lsp diagnostics after closing and reopening a document.
- [x] make raw native stack picks obey the same address bounds as the portable backends.
- [x] show array, pointer, struct, enum, alias, and function types in hover previews.
- [x] show inferred types for untyped variables, function returns, arguments, and other inferred declarations.
- [x] verify inference for 600 functions and 5000 locals without silently dropping later edits.
- [x] grow stable frame segments and preserve older local addresses through collection on native, microcore, and javascript.
- [x] verify stable frame growth with llvm 23 and fix the macos libc helper-name conflict.
- [x] inventory fixed addresses and capacities across native, microcore, javascript, and lsp backends in docs/design/memory.md.
- [x] grow execution data and return stacks on native, microcore, and javascript, with explicit runtime reservations and collection after relocation.
- [x] verify operand and return stack growth on llvm and check invalid requests and denied growth without corrupting the vm.
- [x] repeat growth checks on a small generated map.
- [x] verify growing frames on small generated maps.
- [x] preserve all generic parameter names and inference state across nested compilation; infer later function arguments.
- [x] grow assembly and remaining parser scratch storage.
- [x] preserve long lsp request identifiers, grow json and header scratch buffers, and reset tagged-enum records between document checks.
- [x] grow image-building output independently of the building host’s genome slot while preserving logical image addresses.
- [x] grow native code-address metadata and preserve warm calls and invalidation after bulk rewrites.
- [x] restore missing javascript float opcodes and binary32 conversions, then compare focused arithmetic and formatting tests across backends.
- [x] grow program emission and image-building capacity without relocating live code or globals.
- [x] correct the lexer’s float-buffer load and read long float spellings without clipping their exponents.
- [x] point documented image boot headers at the symbol table after packed documentation.
- [x] polish the learning guide and verify its examples against their documented output.
- [x] fix reproducible bugs and apply useful optimizations with regression coverage.
- [x] support build-time adjustment through a consistent generated layout.
- [x] finish growing all compiler tables and execution stacks during execution, preserving live references; heap growth alone does not satisfy this requirement.
- [x] test changed limits, memory growth, invalid configurations, and agreement between image and host layout.
- [x] document available controls and any remaining fixed constraints precisely.
- [x] update builds, runners, editor bundles, tests, and links for the new paths, keeping the repository root focused on entry points.
- [x] preserve useful commands, propagate failures, and handle paths containing spaces.
- [x] verify runner behavior on available hosts and document untested platform behavior.
- [x] polish the learning guide and reference documentation; keep examples executable.
- [x] replace obsolete build commands and distinguish current behavior from historical notes.
- [x] make comments consistent and keep section overviews separate from unrelated declarations.
- [x] keep documentation and comment prose lowercase, matching the older style; preserve case-sensitive code and identifiers.
- [x] preserve full 64-bit integer literals, including values outside the single-word literal format.
- [x] grow stage 0 source input and image output beyond their initial capacities.
- [x] execute javascript vm and micro enhancement recipes with the same slot and output rules as the native hosts.
- [x] add basic bounds and lifetime analysis that warns about provable out-of-bounds access and use after free.
- [x] test definite failures and safe cases so uncertain behavior does not produce a definite warning.
- [x] show known size and lifetime information when hovering over an array or pointer.
- [x] fix the macos first-call crash in native mutual o3 recursion and verify it with a focused regression.
- [x] preserve nonfatal bounds and lifetime warnings across native, microcore, javascript, and both extension hosts.
- [x] verify block documentation, inferred pointer types, and lifetime metadata in the real editor.
- [x] replace quadratic hint deduplication with a growing set and verify large javascript documents.
- [x] make compact compiler maps grow past startup pressure and update obsolete fixed-limit test expectations.
- [x] fix darwin stdio variadic argument placement exposed by native integration tests.
- [x] index local names and preserve shadow chains to avoid rescanning all earlier locals.
- [x] keep cast bounds conservative and infer aggregate aliases as pointers to their stored addresses.
- [x] allow run-only heaps to grow and separate freestanding physical backing from logical vm limits.
- [x] complete final integration checks and refresh generated editor artifacts after the final fixes.
- [x] stop the leftover background test terminal and ensure focused tests exit cleanly.
- [x] show position-aware known values in variable and constant hovers; discard facts after unknown writes and effects.
- [x] verify value previews with focused compiler and editor tests and refresh the editor package.
- [x] separate section comments from declarations, exclude section headings from hover docs, and attach trailing declaration comments correctly.
- [x] make owned-object assignments to globals consume m or u source references through an automatic u transfer, with regression coverage.
- [x] explain inferred lifetimes in hovers, including ownership and borrow sources and where variables are passed up or down.
- [x] support explicitly typed closure returns with fn:type(args) syntax and show the declared result in hovers.
- [x] default compiler hints to blue informational squiggles and support positional hints on, small, and off directives.
- [x] add definition and reference navigation across the current document and included files.
- [x] stream diagnostics and cancel stale check contexts immediately when source changes, keeping editor feedback responsive.
- [x] optional, lowest priority: add a configurable pretty-printer with formatting behavior similar to clang-format; implementation is not required to finish the other work.
- [x] verify embedded runtime, compiler, server and standard library source navigation in generated editor hosts.
- [x] verify relative and nested includes, diagnostics on include strings, and explicit `include std;` without automatic imports.
- [x] finish compiler-backed lsp completions with signatures, documentation, scope, fuzzy relevance and consumed variables last and crossed out.
- [x] repair embedded-source hovers for every declaration in comma-separated and same-line declarations, including constants and backing arrays.
- [x] put section overview comments next to the structures they describe; give initial closure capacities their own descriptions.
- [x] describe base and encoded opcodes inside asm and vm blocks without treating comments, operands or ordinary expressions as instructions.
- [x] describe std in include std;.
- [x] replace fixed-array type syntax [N]T with T[N] in the compiler, bootstrap parser, maintained sources, tests and documentation.
- [x] refresh generated compiler and editor hosts and verify hovers and fixed-array syntax across the available backends.
- [x] use T[?] for growable arrays and T[] for fixed arrays with an inferred declaration-time length; migrate maintained sources and docs.
- [x] support array initializer lists, zero filling and index designators such as { 1, [5] = 0 }, including nested arrays.
- [x] support first-class array compound literals such as (int[]){1,2,3}, with argument decay and lifetime-safe temporary storage.
- [x] add //: non-documentation comments to the compiler, bootstrap compiler and source-only hovers; mark organizational source comments explicitly.
- [x] extend cached analysis results for repeated hover, navigation, semantic-token and completion requests, with source and include invalidation.
- [x] format local hovers as (type) let.lifetime name; // local, retaining transfer details without redundant ownership prose.
- [x] document compiler built-ins, including symbol-table columns such as sval, and expose their source comments in built-in hovers.
- [x] add static to syntax highlighting and show its form and meaning in hover.
- [x] audit built-in runtime signatures for complete source documentation and add hover help for len_s and transfer helpers.
- [x] infer the owned return transfer so returning a local owner does not require an explicit u* recast.
- [x] expand the standard library last with word-string suffix checks and packed-byte length, equality, search, and suffix helpers.
- [x] remove 45 `unchecked` annotations from compiler, runtime, standard library, and language-server helpers where checked ownership contracts and bootstrap/runtime tests proved them safe; give include paths an explicit consumed `.d` lifetime, extract checked format-file parsing, and make navigation walk parent paths in place.
- [x] remove all 12 remaining compiler `unchecked` function annotations and generate checked native struct-return stubs; declare owned token/source returns and consumed generic buffers, make internal allocation lifetimes explicit, and accept same-scope replacement moves into released owners with expired-source diagnostics.
- [x] implement BORROW.md temporary slots and cleanup for proved non-escaping calls, including returned aliases, nested calls, branches and early returns.
- [x] infer parameter read/write/retain/consume/returned-alias effects, resolve recursive summaries to a fixed point, and keep unknown/unchecked/native effects conservative.
- [x] record compiler ownership, alias, bounds, and lifetime facts; infer last-use borrow endings and unambiguous consuming-call transfers.
- [x] track field and index projections, proven disjoint accesses, and invalidation after array mutation or unknown effects.
- [x] improve definite lifetime/bounds diagnostics with cause and later-use locations, retaining hard errors and conservative unknown results.
- [x] make mutable closure captures share storage so counters and stateful callbacks observe updates.
- [x] modernize void-like examples and document the already implemented generated layouts and growing compiler tables without reintroducing fixed limits.
- [x] verify every BORROW.md acceptance case, cleanup/allocation counts and safe neighbors across available backends; update BORROW.md to describe implemented guarantees.
- [x] add deterministic compiler/runtime/stdlib fuzzing for 128 generated expressions, 300 scoped allocations and GC, token-input immutability, and 32 randomized Text/Bytes cases on native and microcore; fuzz LSP open/change, semantic tokens, and `include std` across 32 documents.
- [x] extend deterministic fuzz coverage to 320 malformed and byte-mutated compiler inputs across native and microcore, the same generated runtime/stdlib checks across native, microcore, JavaScript and WebAssembly, and 96 open/change/close/reopen LSP cache cycles in one server session.
- [x] cache packed-byte haystack and prefix lengths once in `Bytes.starts`; verify its results in the stdlib fuzz run and measure a 7.6% median improvement over 7 paired runs of 800,000 prefix checks.
- [x] repair the byte-load benchmark to auto-size its run-only image and use a monotonic timer with repeated samples, so benchmark failures cannot masquerade as timings.
- [x] before further implementation, add acceptance, rejection and regression coverage for every unchecked TODO item, then keep working until all of those tests and the required integration checks pass.
- [x] suppress magic-number hints for literals immediately preceded by an inline block comment, explain this escape in the hint and documentation, and test nearby-comment and uncommented cases.
- [x] hint on unused named function arguments at the parameter, respect hints on/small/off, and cover writes, reads, shadowing, methods, closures, generics and repeated `_` discards.
- [x] infer untyped word strings as `char*` and packed strings as `bstr`; cover reassignment, grouped declarations, parameters, returns, completions and hovers.
- [x] support comma-separated includes, repeated imports, and diagnostics for unknown modules and empty entries.
- [x] repair F12 navigation for every built-in, including `s_to_n`, so definitions open the correct embedded source (`rt.zc` for runtime functions); verify native, JavaScript and WebAssembly LSP hosts and the extension's virtual-source provider.
- [x] normalize all variable hovers to show the type first, including globals, fixed arrays, parameters, fields, inferred and embedded-source declarations; verify `(char*) let hello // global;` and `(word[20000]) let prog // global;` at declarations and uses without a trailing `; type ...`.
- [x] audit maintained string arguments, variables and results so string data uses explicit string types instead of word; preserve word for actual raw cells and buffers.
- [x] implement the compiler API namespace before further standard-library expansion. Outside `compiler.zc`, require `include run;` and `run.name` (for example `run.evals`); bare compiler names are allowed only inside the compiler module. Reserve `core` for VM core APIs.
- [x] implement matching runtime and language-server namespaces before standard-library expansion. Outside their respective modules, require `include rt;` with `rt.name`, or `include lsp;` with `lsp.name`.
- [x] support array receiver calls for len, cap, push, pop and resize, with element-aware lsp completions; reject fixed-array mutation through receiver and free calls, and verify 15 focused accept/reject cases on native and microcore plus lsp completion regressions.
- [x] add consistent `--help` output to the compiler, lsp and other command-line toolchain programs; describe available options and input modes, return successfully without reading source or starting the server, and cover help output and invalid options with focused tests.
- [x] add array operations for push, pop, find, map, filter and related common transformations; support receiver calls such as `x.push(4)` and `x.map(fn (e) { ... })`, include array-method completions, and document and test fixed-array and growable-array behavior.
- [x] install enhancement descriptors before starting self-hosted run-only images; verify arithmetic, generic array methods and reference pop ownership in standalone microcore images.
- [x] indent `case` and `default` labels one level deeper than their `switch` when pretty-printing, and update formatter documentation and regressions.
- [x] propagate inferred types across calls and return flows, including inferring a caller's untyped parameter from a callee's typed parameter and a caller's return type from the returned callee result; cover chains such as `mklist(n)` and `return l` with compiler and hover regressions.
- [x] allow `fixed` and `const` to qualify the same declaration, with consistent parsing, immutable fixed-array semantics, documentation and accept/reject coverage.
- [x] suggest qualified namespace members after prefixes such as `rt.` and verify completion ordering, signatures and documentation for each imported module.
- [x] show library-specific hover documentation on each module name in `include name;`, including comma-separated includes.
- [x] add LSP quick fixes for missing module imports and related compiler diagnostics; insert the relevant explicit include without changing unrelated source text.
- [x] verify collector-growth accounting relative to existing allocations and investigate conservative roots retained after clearing large global arrays.
- [x] allow repeated `_` discard placeholders in parameters and patterns, including tagged-enum payloads and catch-all cases; warn when a bound discard value is read.
- [x] add generic native and hosted microcore thread contexts with isolated vm state, channels, cancellation and cleanup; verify 14 native and 12 microcore fixtures while keeping actor-specific code in its library.
- [x] give core built-ins and every other toolchain module their own namespace and require an explicit include to access qualified exports: `include core;` enables `core.name`, with the same rule for all other modules. remove implicit toolchain imports, migrate maintained sources and generated code, and verify missing-include diagnostics, completions, hovers and navigation before expanding the standard library.
- [x] add a generic include-time syntax-extension interface so imported libraries can extend compiler parsing and lowering without module-specific compiler code; keep all extension hooks generic and inactive unless the corresponding library is included.
- [x] implement `include actor;` as a library that injects actor syntax into the compiler through the generic include-time extension interface: `actor` declarations in place of `struct`, implicit `this`, and generic `Actor<I,O>` input/output types. Provide asynchronous parallel execution with independent state and multiple instances; keep actor-specific code out of `compiler.zc` and every file it includes or reads by default, and cover syntax activation, execution, messaging and type safety with tests.
- [x] implement and integrate `src/lib/icon.zc` with structured runtime syntax transformations, template matching, execution inspection and stepping, with explicit contracts and comprehensive tests.
- [x] change the producer of the genome header (`build/core/genome.h`) to emit a c translation unit that embeds the compiler, core and runtime zc sources with a `#embed` directive instead of concatenating them into a 1.5 mb generated header; keep the `.zc` sources as the single source of truth and extend the fixpoint check in `make genome` so it still verifies the emitted genome without materializing a header at all.
- [x] close the two ownership completeness limits that stop the compiler compiling itself, so `make genome` completes from a clean checkout: allow an owned value held in a local to move into another local when the destination's previous owner is released in the same statement list, and allow a global to hold a bounded borrow of a local so an output stream can be redirected at a scratch buffer without a module-level variable. also give `rt.code_blocks = m*block;` a supported spelling and fix the `only a named owned object can be recast` diagnostic, which does not describe the real constraint. written up in `docs/history/BOOTSTRAP-OWNERSHIP.md`, which also records a verified partial fix that removes five of the eight current errors.
- [x] bring documentation, examples, learn entries, history notes and test fixtures in line with the include and namespace format and the other language changes made alongside it: qualify runtime built-ins as `rt.put_s`, `rt.put_n`, `rt.put_c`, `rt.eof` and `rt.get_c`, require the explicit `include` each module needs, and remove the remaining bare builtin names so every shipped sample, every documented snippet and every expected-output fixture compiles and runs against the current compiler. the learn guide and `README.md` snippets are checked by the suite, so they are the gate, but the prose and the non-exercised samples are not.
- [x] add accept, reject and regression coverage for the actor and icon libraries once each is finished: for actors, syntax activation through the include-time extension, spawn/recv/send across input and output types, multiple instances of one actor, parallel execution, implicit `this`, and confirmation that no actor code is reachable from the compiler or anything it reads by default; for icon, find/replace over real syntax, undeclaring names, changing types, adding and rewriting blocks, evaluating parts of an expression, and stepping execution while reading locals and globals, including the failure cases for each.
## handoff: where things stand
this section is written so work can resume from the repository alone. the
branch is `ccr-0054c02e-dfdvnd`; everything listed as done below is committed
and pushed there.
### how the work is organized
- start with `ONBOARD.md` (reading order and exact commands), then
`docs/SPEC.md` (the language reference), then this file.
- the compiler is twelve files listed in order in `src/compiler/parts`; their
concatenation is the old `compiler.zc`. the genome sources (runtime, core,
compiler parts, language server) must stay inside zc0's subset: zc0 has no
structs, enums or `?:` yet (in progress, see below); `u`, `m`, `d` are
ordinary names; parenthesize mixed `&&`/`||`; global arrays as `word[64] x;`.
- focused checks, never the whole suite until the last item:
`make` (ends with `fixpoint: N words`), `bash tests/runerr.sh build/bin/zy
tests/errors` (and `typeerrors`, `langerrors`), `python3
tests/drivers/learn.py build/bin/zy`, `python3 tests/drivers/icon_test.py
build/bin/zy`, `python3 tests/drivers/actor_test.py build/bin/zy`,
`ZC_TEST_TIMEOUT_SCALE=4 node tests/drivers/lsp_test.js build/bin/zclsp`,
`ZC_TEST_TIMEOUT_SCALE=4 node tests/drivers/hint_fixes_test.js
build/bin/zclsp`, `make parity` (zc0 against the self-hosted compiler; by
design differences live in `tests/parity.known`). on x86-64 linux `zy` and
the language server now run on the x86-64 jit (o3 regions still interpreted);
`ZC_TEST_TIMEOUT_SCALE` stretches the deadlines on slower hosts.
`bash tests/drivers/jit_x64_test.sh build/bin/zyi build/bin/zy` compares the
jit with the interpreter (`make build/bin/zyi`) on every program.
- work was done by delegating each item to a separate agent in its own git
worktree (outside the repo, under `/workspace/tinyvm-wt/`) and merging its branch.
every brief started with "read ONBOARD.md first" plus the sections relevant
to the job, stated the invariants (fixpoint, byte-identical images where a
change should not alter output, zc0 subset), the focused checks, and asked
for a report of files changed, tests run and open questions.
- prose everywhere is lowercase with plain punctuation; `//` line comments,
`/* */` only between code; a comment above a declaration is its hover text.
### vision and direction
the coordinator's view of where zc stands and where it should go, recorded so
the reasoning behind the open items survives the session.
- what zc gets right: the compiler lives in memory next to the program, so
`eval`, parse words, immediate functions and library-defined syntax (actors
with no actor code in the compiler) fall out of the architecture: forth's
strength with c's syntax. the ownership analyzer infers most of what other
languages make you annotate (it already infers that `skip(s)` returns a
borrow of `s`). unchecked code falls back to a conservative collector instead
of undefined behavior. the language server is the real compiler in check
mode, so diagnostics cannot drift from compilation. one image runs on the
interpreter, microcore, javascript, wasm, arm64 jit and llvm.
- what holds it back: trust and visibility. until `docs/SPEC.md` there was no
spec, and the two implementations disagreed (`&&`/`||`) without anyone
noticing. run-time failures carry no position and a wild read can exit 0.
"checked" is not purely static: turning the checker on inserts cleanup that
can change behavior (the reverted `unchecked` trial hung `share.zc`).
friction: the toolchain's own sources must stay in zc0's subset; word-per-
character strings cost eight times the memory of `bstr` and split every api
in two (`put_s` and `put_bs`); the compiler's state is global parallel
arrays; on x86 the language server was interpreted (about 18 s to check the
compiler) until the x86-64 jit landed.
- priorities, in order: things that turn wrong behavior into a located,
reproducible failure come first, because everything else (fuzzing, a proving
scaffold, outside users) depends on them: runtime line tables, bounds
checking, and the spec with its parity runner and conformance suite. then the
language features that make zc pleasant (`defer`, destructors, variadics and
`std.print`, test blocks, borrow returns for declarations). then speed (the
x86-64 jit).
- bootstrap philosophy: zc0 is an independent implementation in c that never
retires (like go 1.4, unlike rust's downloaded stage 0 or ocaml's and zig's
committed binaries), so the system builds from source with only a c compiler
and no committed blob. its value is being small, so it grows only where a
feature is cheap and produces no code of its own (tags, recasts, byte
strings, maybe enums and `?:`), never structs. the parity runner is a small
form of wheeler's diverse double-compiling: the two front ends check each
other.
- managing the compiler's global state: (1) a cross-reference report of
writers and readers per global, checked in ci; (2) group each family of
parallel arrays in one place with its invariants written once, and route
snapshot and restore through one pair of functions; (3) the split into phase
files (done). do not try to make the state non-global: the single-pass design
is fast because of it; the goal is visible ownership of state, not removal.
- debugging, beyond line tables and bounds checks: every effect a program has
goes through one syscall gate, so record and replay is cheap here (log each
gate result, replay a run exactly, then add tracing or watchpoints at will).
once line tables exist, a debug adapter for vscode (breakpoints, stepping,
locals) on top of the interpreter is the next step.
- a proving scaffold (the owner asked for an opinion): yes, but check before
prove. a written semantics comes first (the spec, then a small executable
reference evaluator). grow it out of the analyzer that already exists rather
than beside it. order: stack-effect checking for every function and `asm`
block (the commonest invisible bug on a stack machine); declared side
effects at module boundaries (globals touched, syscalls or not) checked
against the effect masks the compiler already computes; `requires`/`ensures`
contracts compiled to run-time checks under a switch, with the easy ones
discharged statically from the value facts. keep the trusted base small and
outside the self-compiling loop (a prover inside the genome can approve its
own mistakes, as the fixpoint approved the precedence bug); prefer
translation validation of each compile over proving the compiler whole.
- native code: the hand-written jit exists for arm64 and (phase 1, no o3
regions or inlined enhancements yet) x86-64 linux; the llvm backend
(`-DZYLLVM`) is the portable native tier. `make` puts it in `zy` on
non-x86_64 hosts whenever a usable llvm is found (`NOLLVM=1` turns it off);
on x86_64 it stays out unless `LLVM=1`, because `-DZYLLVM` replaces the
x86-64 jit (owner's call: keep the jit as the x86-64 default). usable means
`llvm-config` or `llvm-config-19` with `llvm-c/Core.h`, non-empty `--libs`
and a tiny test link; otherwise a one-line note. `build/llvm.mode` (llvm or
nollvm, rewritten only on change) makes switching rebuild `zy` without a
clean. checked on x86_64 with llvm 19: plain make uses the jit (no llvm
regions on o3.zc) and reaches the fixpoint, `make LLVM=1` rebuilds zy with
8 llvm regions, plain make switches back. the non-x86_64 path is untested. the x86-64 port of the hand-written jit is worth it for compile
latency (microseconds per block against milliseconds for llvm), which
matters for short programs and the language server.
- where to pick up (end of session): merge the unmerged agent work in this
order: line tables first (bounds checking and the conformance suite both
want positions in their failures), then bounds checking on top of it, then
the spec conformance suite, then the xref baseline and its ci check. test
blocks and destructors come after, since they are features rather than
safety nets. each agent's state and next step is in "agent work at the end
of the session" below.
### feature decisions (owner's requirements, settled)
- bounds checking is on by default; `unbounded` (used like `unchecked`:
function prefix, block, top-level statement) turns it off; `o3` implies
`unbounded`.
- `defer` runs at the end of the current block on every exit path, like the c
defer proposal, plus ownership awareness; destructors (`fn drop()`) exist as
well.
- borrow returns: inferred from the body wherever possible; any spelling for
declarations must stay unobtrusive (proposed: `borrows s` after the
parameter list).
- `string` is a built-in alias of `char*` (done); c typedefs (done).
- variadics: count and per-argument types available, several packs per
function when element types differ (`void f(...int a, ...char b)`), a mixed
`...any` pack, and `...*any` for comma-free calls; `std.print` is a global,
javascript-like print built on it.
- struct-level untyped constants (done), typo suggestions for members (done).
- every new agent brief starts with "read ONBOARD.md first".
### done in this session (all merged)
- build from source with c23 `#embed`, everything generated under `build/`,
`ONBOARD.md`, the embed declaration (`embed "f" as [type] name;`).
- language: untyped `const` (local, top level, inside structs), c precedence
for `&&`/`||` (the self-hosted compiler had them swapped), `typedef` and the
built-in `string` alias, struct constants (`std.Text.MAX`).
- editor: quick fixes for hints (rename to `_`, make const, `<<`, stack
buffer), missing-import fixes for every module (rt, core, run, loaded files,
and std, icon and libraries listed in `src/lib/modules.txt`, scanned from
source, including static groups like `std.Text.itoa`), typo fixes for
functions, module exports, fields and static members.
- zc0: packed byte strings, `u`/`m`/`d` as names, local `embed`, and the asm
enhancement bug parity found.
- tooling: `make parity` (`src/tools/parity.sh`), the compiler split into
phase files, `docs/SPEC.md`, standard-library api docs and
`examples/std_tour.zc`, temporary notes removed, `PROPOSALS.md`.
### agent work at the end of the session (unmerged; read this first)
the owner stopped the session with seven agents running. each was asked to
commit what it had and report. their worktrees were copied next to the main
code in the final zip (`agents/<name>/`, with `AGENT-COMMITS.txt`,
`AGENT-CHANGES.diff` against the base they branched from, and
`AGENT-UNCOMMITTED.txt`). none of it is on this branch. to resume one, apply its
diff to a fresh branch (or re-create its worktree from the commits listed) and
continue from "next" below; the original architecture is in the matching item
of the open list.
- line tables and backtraces (`agents/line-tables`, 3 commits, last 917bc34).
done and checked on zy, zym and the javascript vm: `lines on;`/`lines off;`,
`rt.trace()`, traces after the unchanged first line of `oob`, bad heap
pointer, use after free/double free, out of memory and frame overflow. per
function line blocks [next, entry, end, name, file, count, offset<<32|line...]
emitted into the code stream and chained from `rt.line_blocks`; compiler side
`lt_*` in base.zc plus hooks in stmt, function definition, closures and eval;
image.zc patches the image's own `rt.line_blocks`; check mode skips it. gate
op 56 returns the return-stack pointer (core.c `case 56: return cb_r;`,
micro.c from `r`, tinyvm.ts `this.rr`; wasm and baremetal go through micro.c;
jit and llvm paths untested). zc0 emits line tables for run-only images, via a
`//:lines NAME` marker that automap.sh puts before the program; the resident
compiler honours the same marker, so `build_image` images get tables too
(`runtime_image_test.py` has a case). stdin source is named `<stdin>`. tests:
trace.zc, trace_more.zc, trace_run.zc with `.expected` files; test.sh fault
checks changed to `head -1`. genome grows 383672 to about 391049 words (+1.9%)
from code alone, with lines off. not done: README and SPEC section 8 docs,
the focused suites on the final build, the measurements (image sizes on/off,
mandel compile time, genome with lines on), the x86-64 jit path for op 56.
limits: o3 frames are not on the return stack; a function straddling code
segments is not recorded; core-level stack overflow messages have no trace;
the frame window (top 24 plus last 4) prints "... N more frames" with a raw
word count. merge note: main has the x86-64 jit since this branched; op 56
must also work there (it reads `cb_r`, check the x64 hook sets it).
resumed on work/line-tables (ported onto main fa35813): done. the x86-64 jit
needed no change: its store hook publishes s and r into S before a gate
store, so `cb_r` is exact there (comment at `hook` in core.c);
`tests/programs/trace_jit.zc` traces from translated code and agrees on zy,
zyi and zym, also with ZYNOO3=1. docs: README "line tables and backtraces"
(after memory debug), SPEC 5.15, 8.3 (`rt.trace`), 8.4 and appendix a.
measurements: genome 388100 words at fa35813, 395477 with this work (+7377,
+1.9%) with lines off, 410937 (+3.9% more) with lines on; run-only images
grow by their tables (`trace_run.zc` 31051 to 31119 words, `gc.zc` 30822 to
30857); `examples/mandel.zc` compiles and runs in about 22 ms on the x86-64
jit with lines on, off, and on main, and the genome compiles itself in
about 1.2 s either way (no measurable difference). the genome no longer fit
the default reservation (0x60000 words), so `preset_default` in genmap.c now
reserves 0x68000 and the map constants in rt.zc moved by 0x8000 (README map
table updated). side effect: the heap starts higher, where many genome code
words look like heap addresses to the conservative collector (it scans
GORG..GORG[6], code included), so `tests/programs/gc.zc` freed 969 of 1000
instead of 998; its threshold is now 950. `make parity` puts `//:lines
<stdin>` before each program, so zc0 images trace like zy. not checked: the
javascript vm on the final build (no tsc on the box), the full suite.
open: scan only the genome's globals as roots, not its code; whether the
genome should ever carry lines (+15460 words).
- bounds checking with `unbounded` (`agents/bounds-checking`, 2 commits, last
46aa911). done, builds to the fixpoint: zc0 accepts and ignores `unbounded`
in all three positions; the compiler has the prefix (any order with
`unchecked`, `immediate`, `o3`; methods and closures), the block and the unit
form. fixed arrays with a variable index get an inline `outside` check and
`oob`; `for (i < K; i++)` with constant K proves a fixed array of at least K
(unverified that the check is really skipped). pointers get a one-instruction
`rt.bnd` enhancement test, then `rt.bchk` to report; interior pointers, stack
arrays and globals are not checked; constant index 0 and analyzer-known sizes
skip it. heap lengths are the buddy block capacity (`rt.new(4)` reports
length 11); exact lengths need a header change. `unbounded` and `o3` also drop
growable-array checks (changes `o3_bounds` and `bounds_hi`, which expected an
abort). `unbounded;` heads rt.zc, core.zc, base.zc, server.zc, run.zc,
build.zc; only actor.zc's header peeks in `encode` are marked. every program
is unchanged except `bounds_hi`, `o3_bounds`, `o3call`; errors, typeerrors,
langerrors, learn, icon and actor pass. cost: pointer kernel user time 0.58 s
to 0.76 s, fixed-array variable-index kernel about 35% slower. next: mark
`main` in tests/programs/o3call.zc `unbounded` (it reads `p[-1]` on purpose);
update the `o3_bounds` and `bounds_hi` expectations; add boundsrules.zc,
bounds_fixed.zc, bounds_heap.zc, zc0unbounded.zc with their test.sh lines;
docs (README, SPEC 2.6.1, 3.7.4, 3.13, keyword table, `kwtext`, vscode
grammar); time the self-compile (a baseline build is in the session
scratchpad `base/`, not in the zip); `make parity`. the 35% fixed-array cost
is worth a look before merging (hoist checks out of loops whose bound is the
array length, extend `proven`).
resumed on work/bounds (ported onto main f75ce6a, conflicts in base, decl
and expr; the context save is 50 words, `fbnd` at 49 after `dfr_base`).
done: o3call's main is `unbounded`; `o3_bounds` and `bounds_hi` now expect
no check (o3 implies unbounded); new boundsrules, bounds_fixed, bounds_heap,
zc0unbounded (also a run-only image) with test.sh lines; bounds_fixed and
bounds_heap are in parity.known; docs in README ("bounds checks"), SPEC
2.6.1, 3.7.4, 3.13.4, 4.10.1, new 4.15, 5.1.1 and the keyword table, the
server's `kwtext` and `iskeyword`, the vscode grammar. genome 395477 to
399590 words (+4113, +1.0%); the self-compile takes about 1.1 to 1.2 s
before and after. the fixed-array cost: the `for (i < K)` proof never fired
(the loop shape needed a local on the right of `<`), fixed now, so a
constant-bound loop costs nothing; the inline check is now plain ops
(`i | (n - 1 - i)` < 0) instead of the `outside` enhancement, which the
x86-64 jit calls rather than inlines. best of 3, ms, main / port / final:
hash-index kernel jit 37 / 72 / 44, interpreter 412 / 504 / 542; loop with a
variable bound jit 55 / 113 / 69, interpreter 741 / 907 / 865; loop with a
constant bound jit 46 / 100 / 53, interpreter 663 / 822 / 690. the plain-op
check is slightly slower than the enhancement on the interpreter for an
unprovable index, and probably on arm64, where enhancements are inlined (not
measured here). still open: pointers, jit 52 to 279 ms and interpreter 710
to 1141 ms on a 256-word sweep (`rt.bnd` is an enhancement call on x86-64);
a variable loop bound is not hoisted (needs a loop-entry test of `n <= N`, or
of the heap block's payload for a pointer, and a flag the body tests).
inlining enhancements in the x86-64 jit would help both. also fixed here:
four conformance expectations on main lacked the backtrace line that line
tables print (3-expressions/dyn_array_*), and test.sh's run-only gc sed
still looked for `f >= 990`.
- spec conformance (`agents/spec-conformance`, last 808886b; the first,
interrupted attempt is `agents/spec-conformance-first-attempt`, already
folded in). done: sections 1 source (114 programs, zy 103 ok), 2 types
(142), 3 expressions (167, zy 165 ok). real self-hosted deviations found
(class a): a bad float or character literal in the first statement of a
function body is not reported (the lexer error is swallowed by lookahead);
`u*d` with `u`, `d` visible variables is taken as a recast (spec 1.6.3); the
bstr flag leaks into `int*` locals so `p[0]` reads a byte; the parameter
limit is 23 not 24 (decl.zc:326; closures 19, expr.zc:671); `void f() {
return 3; }` yields 3 not 0. spec issues (class c): 5.2.2 against 2.6.6 on
array parameters; `_` is an undocumented discard with a warning; an untyped
word stored into a float converts by value. zc0 (class b): mostly dropped
types (appendix a); b.1.11 literal saturation confirmed; new: INT_MIN / -1
gives 0 and % gives INT_MIN in zc0. partly done: section 4 statements (129
programs, zy 119 ok); six expectations are the agent's own mistakes to fix
(switch_break_in_loop_leaves_switch_only should be 5,
switch_expression_evaluated_once "hit1/", switch_label_forms calls t(-1),
t('a'), t(6), t(1), memdebug_report uses pn(rt.memreport()*100+7) substring
107, memdebug_report_line line 9, asm_float_literal uses `word w`); real:
`memdebug scrub;` rejected (b.1.1/b.2.1 confirmed), top-level `o3 { }`
rejected (c), `case rect():` with fields possibly rejected (unresolved).
resumed on work/spec and finished; see the closed item in the list below.
- record and replay (no code; agent stopped during research). findings: every
syscall goes through `hostcall(op, a, b, c)` (core.c ~264) with three callers
(interpreter ~6324, jit `hook()` ~3617 which the x86-64 helpers also reach,
llvm ~5060): rename it `hostcall_real` and wrap it. stdin does not go through
the gate: the vm reads it with the Q opcode (io_linux.c; `J` is the end flag),
stdout with G, so log Q as (byte | J<<8) chunks flushed before each gate
record. the program source and its input share stdin (the compiler reads up
to the backtick with `srcend` in lex.zc), so replaying a modified program
needs a c mirror of `srcend`: serve the new source live, then answer the rest
from the log after skipping the old source. ops writing vm memory: 3 and 11
(read), 5 and 13 (getenv), 22 and 24 (argv, env via `vecstr`), 9 sub-ops, 41
(float format), 20 (scan), 43 to 45 (fork, channels); 18 and 19 are pure;
14 to 17 (c abi) and 51 to 54 (contexts) hand out native handles: stop in
replay, and detect thread start (51, 43) in record. micro.c has its own gate.
- struct destructors and the `defer` single-statement form (no code; research
only). findings: `freevar(i)` (analyzer.zc ~611) is the only place an owned
local is freed, called from `exit_vars` (block end, return, break, continue,
deferred frees) and `fixups` (join paths); `borrow_cleanup_since` (~136) frees
fresh temporaries passed to borrow parameters (needs the argument type kept in
`emit_borrow_hold` to drop them); explicit `rt.free`/`rt.recycle` go through
`call()` in expr.zc, where `xty` is the argument type after `args()`;
reassignment never frees implicitly (`assigncheck` rejects overwriting a live
owner). a struct is `TY_STRUCT` with its name in `t[2]`, a pointer `TY_PTR`
with the struct in `t[4]`; find `drop` with `lookup(methname(name, "drop"))`
and `skind == K_FN`; reject generic, static, argument-taking or
result-returning `drop` in `kw_struct`. planned emission before the free,
when `fchk && !o3on && !dry`: `lit(0); op(O_PICK); skip = hole(); op(O_JZ);
lit(0); op(O_PICK); fnref(dropsym); op(O_CALL); op(O_DROP); setlit(skip,
here);`. the defer form: `defer_tokens`/`kw_defer` in stmt.zc.
- test blocks (no code; research only). findings: the native core never reads
a file argument (`zy foo.zc < src` keeps reading stdin; only the javascript
runner reads argv[2]), and any `--option` as argv[1] is rejected as unknown
in core.c main (~6743), micro/port_linux.c main (~375) and tinyvm.ts (~960),
so `--test` needs a small change in all three (accept it and keep it in
argv). `main` is called in `start()` in setup.zc (~187) through
`lookup(s_main)`, and `s_main` is a global of namespace `run`, so the library
can point it at `test.run_all` with no compiler change. plan: `src/lib/test.zc`
like actor.zc (`run.syntax("test", parse_test)`); without `--test` skip the
block by brace matching; with it compile the block through `run.tokens` and
`run.compile_list_context` as `test.case_N` and record [fn, name]; arm the
runner at include time; rewrite `expect(c)` to `if (!(c)) { test.fail(line,
text); return; }`. open: whether the keyword handler can decline when the
next token is not a string (so `test` stays usable as a name). diagnostics
print `zc: line N:` with no file name.
- compiler cross-reference report (`agent/xref-report`, cherry-picked onto
work/xref without conflicts). done: `src/tools/xref.js` and
`src/tools/xref.sh` (`--help`, unknown option exits 2), the `make xref`
target, the readme and onboard.md sections, and the xref.sh entry in
`tests/drivers/cli_help_test.py` (passes with `--group scripts`).
`tests/xref.baseline` is checked in, written by `xref.sh --update` (658
globals, 633 functions, one line per global with its sorted writers); a run
took about 10 s with 8 servers, not 12 minutes, so the readme now says tens
of seconds. `--check` against it exits 0, and exits 1 when one writer is
dropped from one baseline line. `src/tools/xref_selftest.sh [server]` does
both cases in a scratch directory and runs in the lsp section of
`tests/test.sh` next to the other node drivers (not in -q). known limits: a
store through a pointer (`*p = ...`), an array written by a callee
(`copy_w`, `memset`) and a global reached only through its `_initial` array
are not counted as writes. the baseline is not yet reviewed by a person;
the 60 "no writer" globals still need checking against those limits. the self-test also
copies src/compiler to a scratch dir, adds a new function writing
`namecount`, and checks that `--compiler-dir ... --check` exits 1 naming it;
it rejects unknown options and is in cli_help_test.py. open: deciding whether ci runs it on every job or only
when `src/compiler/` changes.
### decisions
- no structs in zc0 (owner's call): zc0 stays the small trusted root of the
bootstrap. so the compiler keeps its parallel arrays; the records refactor
becomes grouping and documenting them (see that item). enums and `?:` in zc0
stay optional and low priority.
- agents: sonnet 5.5 for the jobs; do not cancel running agents.
## open, easiest first
sorted by how hard each is to implement; the final full suite stays last.
items with an architecture note carry the design worked out so far, enough to
brief someone without re-reading the code.
- [x] remaining zc0 gaps, one commit each on work/xref: string literals have no length limit (the token text grows; `1-source/lit_string_over_4095_bytes.zc`); `.u` on a typed parameter is rejected with the compiler's message, "a parameter tag is .m or .d" (`5-declarations/fn_param_tag_u_rejected.zc`); `x as T` reads a whole type, so `x as int*` and `x as char**` work (`3-expressions/cast_as_pointer_type.zc`); `embed` of an array type inside a function puts the elements in the code and the local holds their address (`tests/programs/embed_local_array.zc`, a parity program); integer literals as the compiler reads them, so `010` is 10, `0x` is 0 and values wrap modulo 2^64 (the eleven class b literal entries left `known.txt`). `docs/history/43-stdlib/notes.md` has a forward note that zc0 no longer reserves u/m/d. the genome is byte-identical to main's.
- [x] verify both compilers against `docs/SPEC.md`: `tests/conformance/<section>/` (sections 1 to 8 plus `9-appendix-b`, 657 programs), runner `tests/conformance/run.sh` (`--zc0`, `--zym`, `--zy`, `-w`), understood failures in `tests/conformance/known.txt` (class a self-hosted bug, b zc0, c spec issue), wired into the interp section of `tests/test.sh`. zy 626 ok, 31 known; zc0 221 ok, 38 known, 398 skipped (outside appendix a). appendix b.2 items 1 to 16 confirmed except b.2.8, which is documented behavior (5.3.2, 3.14.2; the spec wins). top-level `o3 { }` is rejected (4.10.2, class c, left alone). the seven class a bugs are fixed, one commit each: a bad literal in a function's first statement (capture_scan left panic set), `u*d` with visible `u`/`d`, the bstr flag surviving a cast, the 24-parameter limit, `return e;` in a void function (now dropped, yields 0), `memdebug scrub;`, and `case member():` on a member with fields. the conformance run is in `tests/test.sh` but not in `-q`. open: `rt.memreport()` reports buddy capacity (`rt.new(2)` is 3 words), whether that is the spec's W; whether top-level `unchecked;` is in the zc0 subset.
- [ ] fix the implementation bugs still open in `docs/SPEC.md` appendix b: unterminated string literals are only reported as a later syntax error (b.2.11 fixed the `/*` half); duplicate `case` labels are not diagnosed (b.2.15); a local `const` cannot be used in a constant expression (b.2.16, b.1.7); constant expressions have no precedence, so `1+2*3` is 9 (b.1.3); syntax-error recovery hides errors in the skipped statements (b.2.10, by design; decide whether to keep it). fixed and merged from work/spec: enum members naming earlier members (b.2.2), function redefinition (b.2.3), prototype after the definition (b.2.4), `?:` with int and float branches (b.2.5), `bool` normalisation (b.2.6), float global initializers (b.2.7, integer constants too via 46825f2), `include std;` in a function body (b.2.9), unterminated `/*` (b.2.11), `x as int * y` settled as a syntax error (b.2.12, doc only), unnamed prototype parameters (b.2.13), `len(char*)` with typecheck off (b.2.14). fix the prose items appendix b.1 lists too (verify which are still open).
- [ ] optional, low priority: plain enums and `?:` in zc0 (both are small and produce ordinary code), so parity covers more programs. no structs in zc0 (decided).
- [ ] group the compiler's parallel per-local arrays (`lname`, `ltype`, `lstate`, `lhome`, `lown`, `lslot`, `ltag`, `lpos`, `lnsz`, `lleaf`, ...) without structs: declare each family together in one place, with one comment block stating each array's meaning and invariant (for example when `lhome` is meaningful), route snapshot and restore of the family through one pair of functions, and use the cross-reference report to confirm nothing else writes them. then symbols and types the same way. fixpoint and analyzer behavior unchanged.
- [ ] continue removing `unchecked` from the runtime. a reverted trial showed: `copy_w`/`fill_w` pass the analyzer when their gate stores go through `syscall(SYS_COPY, ...)`, `dgrow` with `own(w, it)` instead of `w[-2] = it`, `dpush` with `own` before the store, `dnew` as is; but the checked versions changed behavior (7 program checks failed, `tests/programs/share.zc` hung with a checked `cons`), so checked code inserts cleanup these functions can't afford. each needs its contract taught to the analyzer. `free`, `recycle`, `recycle_leaf`, `gc` are the allocator and collector the analyzer's own cleanup calls: document them as the irreducible escapes. `src/lib/actor.zc` has about 20 more.
- [ ] revise diagnostics, comments and learning/reference prose using professional compiler documentation guidance, preserving lowercase and adding message regressions. touches messages everywhere, so do it when few other compiler changes are in flight.
- [ ] (partly done, see "agent work at the end of the session") test blocks: `test "name" { ... }` with `expect(cond)`, compiled only by `zy --test file.zc`, which runs each block and prints one line per test with the position of the first failed `expect`; skipped at parse time otherwise. build it as a library through `run.syntax` and `run.statement_hook`, like `src/lib/actor.zc`.
- [x] (done on work/bounds, see "agent work at the end of the session") bounds checking on by default, `unbounded` to turn it off (function prefix, block, top-level `unbounded;`, mirroring `unchecked`), implied by `o3`. architecture: indexing is compiled in the postfix `[` handling of the expressions part; growable arrays already check (`dat`, `range_jz`, the `outside` enhancement, `proven()` removing checks a loop test already did). fixed arrays `T[n]` with a variable index get the same inline check against the constant `n`. pointers get a runtime call (for example `rt.bchk(p, i)`) that checks only when `p` is the start of a live heap block (header check word `p ^ MAGIC`, `paylen`), skipping constant index 0 and sizes the analyzer knows (`lnsz`). reports go through `oob`. the runtime, core, compiler parts, server, `run.zc` and `build.zc` start with `unbounded;`, which zc0 accepts and ignores (genome bytes unchanged). keep std, icon and actor checked unless a function touches raw memory on purpose. measure interpreter time before and after.
- [ ] after the fuzzing work, add more general functionality to the standard library.
- [ ] (partly done, see "agent work at the end of the session") struct destructors: a struct may declare `fn drop()`, called wherever the compiler frees an owned value of that type, before the memory is released. architecture: the compiler frees owned locals in `freevar` (analyzer part), called from `exit_vars`, reassignment and explicit `rt.free`/`rt.recycle` calls (`s_free`, `s_recycle` handling in the call code); when the local's static type is a pointer to a struct with a `Name.drop` method, emit a call to it first. objects freed by the collector or inside unchecked code do not run `drop`: document that. first users: file descriptors in `std.Host`, processes in `actor.zc`.
- [x] (done on work/line-tables, see "agent work at the end of the session") runtime source positions. architecture: at each statement start (`stmt()`) record (code address `here`, line); per function a permanent data block [entry, end, name, file, count, pairs...] linked into a runtime list (`rt.line_blocks`), or appended to run-only images by `build_image` like the symbol blob. `lines on;`/`lines off;` like `docs on;`; on for user programs, off for the genome unless the growth is small. a new syscall op returns the return-stack pointer (the return stack is in vm memory, 0x1A4100..0x1B40FF, growing up); implement it in `src/core/core.c`, `src/micro/micro.c`, `src/js/tinyvm.ts` and the wasm port. `rt.trace()` walks the return stack, printing ` at file:line in function` for words inside a known function range. the runtime's fatal reports (`oob`, bad heap pointer, double free, out of memory, frame overflow) print the trace after their unchanged first line. check mode skips all of it.
- [ ] borrow returns tied to an argument. inference already works for a callee defined before the call: `char* skip(char* s) { ...; return s + i; }` makes `let r = skip(t); rt.free(t); r[0]` an error. the gap: when the call comes before the callee's definition, nothing is reported and the program reads freed memory. extend the deferred effect requirements (`effect_require`, `effect_finish` at unit end) so a forward-called function whose return turns out to alias a parameter produces the same error. for bodyless declarations, externs and function types, add an unobtrusive spelling, for example a `borrows s` clause after the parameter list (`char* skip(char* s) borrows s;`), checked against the body when one exists.
- [ ] variadic arguments and `std.print`. design: a pack is passed as one hidden argument, a caller-built read-only growable-array handle (`fixed T[?]` view: [len, cap, items, element size] in the caller's frame, so `len(a)` and `a[i]` work unchanged and nothing is allocated). `...int a` is a typed pack; several packs per function when their element types differ (`void f(...int a, ...char b)`): each extra argument goes to the pack whose element type matches its static type, an ambiguous match is an error, packs come after the fixed parameters. `...any args` packs each value with a kind code (int, float, char, string, bstr, pointer, ...) and its type record: `any` is a small built-in record { kind, value, type }. `...*any args` makes the call comma-free: after the fixed arguments, the parser reads unary expressions until `)`, commas optional, so `std.print("x = " name ", age " age "\n");` works and binary expressions must be parenthesized. `std.print(...*any args)` prints each argument by its kind (`put_n`, the float printer, `put_s`, `put_bs`, a character for `char`, hex for pointers).
- [x] `memdebug`'s `obj_base` used to search only 64 words back for a block header, so `buf[255]` on a 256-word block reported "bad heap pointer". it now rounds the pointer down to each size class's alignment (largest first) and takes the first header whose check word, size and class agree and whose payload holds the pointer, so interiors at any depth work, a deep pointer into a freed block reports use after free, and header words and free space still report "bad heap pointer". regression: `tests/programs/memdebug_interior.zc` (in `tests/test.sh` beside the other memdebug checks). note that a stray pointer that lands inside another live block is accepted, since memdebug cannot know which block it came from.
- [ ] (partly done, see "agent work at the end of the session") `defer` follow-ups (small): done on work/defer: `defer` takes any single statement (if/else, loops, switch, unchecked, unbounded). still open: decide whether deferred code may name an owner declared after the `defer` (now rejected by the lexical rule; one line in `defer_one`).
- [ ] stack-effect checking: every function and `asm` block declares or is inferred to have a fixed data-stack effect, and the compiler reports a block whose paths leave the stack unbalanced. the first step of a proving scaffold.
- [ ] declared side effects at module boundaries (globals touched, syscalls or not), checked against the effect masks the compiler already computes.
- [ ] `requires`/`ensures` contracts on functions, compiled to run-time checks under a switch, with the cases the value facts settle (constants, simple ranges) checked statically.
- [ ] (partly done, see "agent work at the end of the session") record and replay: a core option that logs every syscall gate result to a file and replays a run from the log exactly, for reproducing failures and adding tracing after the fact.
- [ ] a debug adapter for vscode (breakpoints, stepping, locals) on the interpreter, built on the line tables. goes to whoever finishes the bounds hints.
- [ ] reduce the cost of word-per-character strings: measure, then decide between making `bstr` the default string, unifying the two apis behind one name, or both.
- [ ] continue fuzzing the compiler, analyzer, runtime, standard library, lsp, build tools and generated artifacts, including analyzer properties, memory-limit edges, compiler image mutation and long-running lsp memory use; feed generated programs to `make parity` for differential fuzzing between zc0 and the self-hosted compiler; fix discovered bugs.
- [ ] keep improving the borrow checker: fewer annotations in ordinary code, clearer messages naming both the owner and the conflicting use, and new fixtures for each rule that changes.
- [ ] x86-64 jit, phase 2 follow-ups (phase 2 is in the done list). we are not chasing llvm parity on the jit (owner's decision).
- region 474..692 in `tests/programs/o3.zc` falls back to the region interpreter; find out why and compile it.
- fib is slightly slower under o3 than plain (0.86 against 0.78 ms), likely region entry and exit on every recursive call; related to direct o3-to-o3 calls.
- a compiled o3 region that stores into its own code finishes the current activation on the old code; the change takes effect on the next entry. either document it or bail to the region interpreter when the hook drops the running region.
- [ ] (in progress on work/exactsize) exact-size bounds checks: n is stored in p[-4] bits 32..62 and the check is 2 loads.
- [ ] (on work/checks) bounds checks out of loops and `--explain-checks`. done: a for loop's body is read ahead (`body_scan` in analyzer.zc) for the names it assigns, increments or takes the address of, closures, and anything that might free a block (calls, push/pop/resize/defer/asm/quote/new, stores to locals that can own); loop proofs now need a body that leaves the index, the bound and the array alone (before, a body that moved the index after a proved access was unchecked on the next iteration). `i < n` with an int n hoists the check to a flag computed at the loop entry (`hoist_guard`, `hoist_stub`): the check still runs, in place, when the flag says it can fail, so faults and side effects keep their order. a pointer set from `rt.new(n)` with a constant n is proved by `i < k` (k <= n) or hoisted, using the requested size (never the rounded block size), and only when the body cannot free or resize anything. `zy --explain-checks [file] < src` prints the kept checks as json lines from the same decisions (format v1 in docs/SPEC.md 4.15.4; consumers ignore unknown kinds, reasons and fields; `o3-candidate` is a possible future kind). later: hoisting for growable arrays with a variable bound, `while` loops, pointer sizes from `len()` once exact sizes land, mechanical `edit`s, tracking ownership precisely enough to hoist past calls that cannot free. cleanup: let the `bs_*` body scanner reuse capture_scan's token skipping instead of its own (about 0.5 to 1k genome words). the `--explain-checks` writer is `src/lib/explain.zc`, embedded and compiled only in that mode (a unit that defines its own `explain` namespace would clash with it).
- [ ] (in progress on work/bench) `make bench`, `bench-check` and `bench-selftest`: baseline from ec49c6d, jit and llvm rows, interp rows only with `INTERP=1`. follow-up: fill the checks column from `--explain-checks` and re-save the baseline; the rule that fails when the check count rises is already in bench-check.
- [ ] lsp and vscode bounds hints: diagnostics for kept checks with an all / loops / off setting, quick fixes, and a hot-checks command; the vscode extension supports all of it. later: `o3-candidate` hints for hot loops outside o3.
- [ ] hoisting v2: use the u/m/d ownership tags to prove a loop can't free a pointer (v1 refuses to hoist when the loop body has any free, release or realloc, or any call that might free).
- [ ] gc roots change: its own branch, memdebug scrub, full suite, bench comparison, and the owner reviews it before merge.
- [ ] shrink the 500 genome words added by b.2.13.
- [ ] file-level `o3;` and `unchecked;` directives that apply to every function after them, like `memdebug scrub;`. they replace the deferred top-level `o3 {}` and `unchecked {}` blocks.
- [ ] tests/bench/run.py: the checksum step fails to compile the zc sieve on main, so run.py's sieve/trees timings don't run; bench.sh covers sieve only.
- [ ] run the full suite once everything above is done (`make test JOBS=4`; `make check` is the quick one), fix what it finds, and only then tick this item.
## done in the sorted list
- [x] a cross-reference report: for every global of the compiler, the functions that write it and the ones that only read it. build it from the language server's references (`textDocument/references` on `build/bin/zclsp`, so scoping is exact) and classify each reference as a write when an assignment operator, `++` or `--` follows it. output under `build/xref/`; a committed baseline and a `--check` mode that fails when a new writer of a global appears, so it can run in ci. done: `--check` runs in `tests/test.sh` through `xref_selftest.sh` against `tests/xref.baseline`.
- [x] x86-64 jit phase 2: compile o3 regions to x86-64 (port `rcompile` and the region register allocation; regions now run on the region interpreter and are 2 to 4 times slower than their plain versions on this jit) and inline enhancements (lower the optimized micro-ops to x86 registers as the arm64 path does, instead of calling `xrun`). validate with `tests/drivers/jit_x64_test.sh` and `tests/bench/bench.sh`, which now has an x86-64 jit column. add the driver to `tests/test.sh` on x86-64 linux. done: inline enhancements merged as 1f4aceb, compiled o3 regions merged as ec49c6d.
done so far: enhancements are inlined (`x64_spec_enh`, `x64_enh`, `x64_uop` in core.c; seven value registers rsi, rdi, r8..r11, rbp; rax, rcx, rdx scratch; floats, byte loads and stores, native calls, popcnt and f2s/s2f still go through `xrun`; `ZYNOENH=1` calls everything as before). bench.sh plain kernels on the x86-64 jit, ms before/after: collatz 49.1/32.1, sieve 4.4/4.1, digits 41.9/14.7, fib 0.83/0.77; o3 unchanged (regions still interpreted). a 256-word pointer sweep with bounds checks 291 to 72 ms. `tests/programs/enh_ops.zc` covers every lowered op on edge values; jit_x64_test runs in test.sh's jit section (not in the quick path). o3 regions now compile (`rjit_x64`, `x64_ralloc`, `remit_x64` in core.c): vars stay in their frame slots, values in six registers, calls go through x_disp; `ZYO3INTERP=1` keeps the region interpreter. bench.sh o3 kernels on the x86-64 jit, ms interpreted/compiled: collatz 289/10.4, sieve 8.7/0.40, digits 89/3.0, fib 3.0/0.92 (fib is still slower than its plain version: calls and returns go through the dispatch stub). still interpreted: regions with more than six live values (one in tests/programs/o3.zc) or native calls. later: vars in registers across a loop, direct o3 calls, floats in xmm registers, spilling instead of giving up. caveat: a store from a region's native code into its own words drops the region, but the block that stored finishes on the old code (arm64 does the same). `-DZYRDUMP` dumps each region's blocks (off by default). the loop-entry flag is done on work/checks (see below). next, after `--explain-checks` v1: direct o3-to-o3 calls on x86-64 (like arm64's `rcall`), so fib stops paying for the dispatch stub. a direct o3-to-o3 call must keep a callee generation guard (like llvm's `inline_fn` checking `C->gen`) so a function rewritten by `icon.find_replace` is not bypassed; `tests/icon/regression/o3_caller_replace.zc` calls an o3 callee from an o3 caller before and after a replacement. region 474..692 of tests/programs/o3.zc stays on the region interpreter because it has more than six live values (`x64_ralloc` gives up; needs spilling, dropped from this round, so not a small fix). fib under o3 is slower than plain because each o3 call and return goes through the dispatch stub (`x_disp`) while the plain jit calls directly; the fix is the direct o3 calls above, not small.
- [x] llvm by default (decided): llvm is turned on automatically on non-x86-64 hosts; on x86-64 the jit stays the default and llvm is opt-in with `LLVM=1`.
- [x] stale known-line check: conformance and parity fail when a known entry passes again (merged in c1072ab).
- [x] `defer expr;` and `defer { ... }`: tokens grabbed at the `defer`, recompiled inline at every exit (block end through `block_close`, `return` after its value, `break`/`continue` for entries deeper than the loop), newest first; only defers reached so far register. names are lexical (a local declared after the `defer` is hidden in the replay). a deferred `rt.free(p)` is `p`'s transfer to the analyzer. each entry replays in the analyzer mode it was written in; closures get their own base through `ctx_push`. `return`, `break` and `continue` that would leave a deferred statement are rejected. a failed entry reports once. limit 256 pending per function (`DFR_MAX`). spec 4.14, learn section 34, `tests/programs/defer.zc`, seven reject fixtures.
- [x] four small fixes: hover on standard-library members (`hover_embedded_sym` sends std/icon symbols to the library lookup; `bs_hover` renders a member hit), `capture.end`'s terminator (the buffer has `cap + 1` words; latent, since the allocator's power-of-two slack hid it; `tests/programs/capture_long.zc`), bare sibling static calls inside a struct's methods (`mstruct` set for every non-generic struct; the struct's own constant or static beats a top-level name, a local beats both; `tests/programs/struct_static_call.zc`, spec 5.5.5), and `void main()` in run-only images (`run.zc` declares an untyped `fn main();`; spec 5.2.5).
- [x] a hand-written x86-64 jit, phase 1 (`ZYJIT_X64` in `src/core/core.c`, default on x86-64 linux; `-DZYINTERP` and `make build/bin/zyi` give the interpreter). r12 memory, r13 data stack, r14 return stack, r15 table, rbx state; stack slots addressed with a virtual offset so r13 moves only at calls and exits; shares the table, coverage map, store hook and invalidation with the arm64 jit. enhancements are called through `xrun`, o3 regions run on the region interpreter (`ZYNOO3=1` runs their vm code through the jit instead). 179 of 179 programs agree with the interpreter (`tests/drivers/jit_x64_test.sh`), the self-compile is byte-identical and about 7x faster (1.5 s against 11 s), and the rejection, learn, icon, actor, lsp and hint suites pass on it.
- [x] remove temporary markdown files (`TASKLIST.md`, `docs/design/handoff.md` deleted; `BOOTSTRAP-OWNERSHIP.md` moved to `docs/history/`).
- [x] verify every command in `ONBOARD.md` from a clean checkout and fix the ones that drifted.
- [x] add `string` as a built-in alias of `char*`.
- [x] untyped constants inside structs, reached as `Text.MAX` and `std.Text.MAX`.
- [x] typo suggestions for field and member names, with the change-to quick fix.
- [x] packed byte strings and `u`/`m`/`d` in zc0, byte-identical genome.
- [x] `docs/SPEC.md`, with the zc0 subset (appendix a) and disagreements and bugs (appendix b).
- [x] `make parity`: zc0 against the self-hosted compiler on every program zc0 accepts.
- [x] c typedefs.
- [x] the compiler split into phase files listed in `src/compiler/parts`.
- [x] standard-library api documentation, `examples/std_tour.zc`.
|