diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/ACTOR.md | 59 | ||||
| -rw-r--r-- | docs/BORROW.md | 100 | ||||
| -rw-r--r-- | docs/ICON.md | 112 | ||||
| -rw-r--r-- | docs/LEARN.md | 490 | ||||
| -rw-r--r-- | docs/design/diagnostic-style.md | 3 | ||||
| -rw-r--r-- | docs/design/memory.md | 2 | ||||
| -rw-r--r-- | docs/design/storage-analysis.md | 4 | ||||
| -rw-r--r-- | docs/design/test-coverage.md | 121 | ||||
| -rw-r--r-- | docs/history/maintenance-notes.md | 16 |
9 files changed, 712 insertions, 195 deletions
diff --git a/docs/ACTOR.md b/docs/ACTOR.md new file mode 100644 index 0000000..913c6bc --- /dev/null +++ b/docs/ACTOR.md @@ -0,0 +1,59 @@ +# actors + +`include actor;` loads `src/lib/actor.zc` from the project library directory. the compiler contains generic module loading, syntax registration and statement hooks. it does not embed or recognize actor syntax until the library registers its parser. + +actors run asynchronously in separate operating-system processes on the native and portable micro interpreters on posix hosts. each worker has its own vm stack, heap, globals and captured handler state. workers can execute in parallel. these are isolated worker processes, rather than shared-memory threads. + +```zc +namespace; +include rt, actor; +typecheck on; + +actor Counter { + int total; + int handle(int value) { + this.total += value; + return this.total; + } +} + +fn main() { + Counter* counter = rt.new(Counter); + counter.spawn(); + counter.recv(fn(int value) { + rt.put_n(value); + rt.put_c(10); + return 0; + }); + counter.send(2); + counter.send(3); + counter.await(); + counter.close(); + return 0; +} +``` + +the example prints `2` and `5`. the worker's `total` changes independently of the caller's `counter.total`. `this` is implicit in actor methods; declaring it as an explicit first parameter is rejected. + +a declared actor requires a `handle(input)` method with one input parameter and a typed result. the library adds `spawn`, `send`, `recv`, `poll`, `await` and `close`. allocate actor instances with `rt.new` to initialize their storage. + +for a handler function or closure without a declared actor, allocate `actor.Actor<I,O>` and call `spawn(handler)`. for example, `actor.Actor<char*,int>` accepts character strings and returns integers. `typecheck on;` checks the handler, message and receiver types. receive callbacks currently return `int`. + +`send` snapshots a message and queues it for the worker. it does not run the handler on the caller's stack. `recv(callback)` registers a callback without waiting. ordinary caller statements poll for completed results; `poll()` requests delivery explicitly and `await()` waits for all submitted messages to reach the registered receiver. callbacks execute on the caller's vm at statement boundaries and retain fifo order. a long blocking host call or an optimized `o3` region delays automatic delivery until the caller reaches another boundary. + +messages copy their typed object graphs into the receiving context. supported values include scalars, enums, strings, packed fields, byte strings, growable arrays, pointers to structs, and function values with typed captures. repeated references and cycles preserve identity within a copied graph. integer payloads that happen to equal heap addresses remain integers. changes to a received graph do not modify the sender's graph. ordinary language rules for aggregate parameters still apply. + +`await()` requires a registered receiver when messages are pending. `close()` cancels remaining work, closes the channel and joins the worker; repeated closes succeed. later sends and receives return `-1`. close every spawned actor before discarding it. sends may wait for channel space under backpressure, and polling a large result can wait for its remaining bytes. + +## current limits + +- custom generic actor declarations are rejected. use `actor.Actor<I,O>` with a typed handler instead. +- javascript, wasm, bare-metal and windows hosts do not provide the required context primitives. +- compiler-free emitted images cannot import this source parser: a syntax handler must already reside in the compiling host. the actor library reports this limitation through the generic syntax-registration interface. +- statement hooks run outside included library code and `o3` regions. automatic callbacks therefore require the caller to return to an ordinary statement; explicit `poll` and `await` remain available. +- channels stay within the spawning vm process family. function messages require code and type descriptors already present when the worker was spawned; code compiled dynamically afterward is not transferred. +- raw pointers lack bounds. a pointer to a non-heap object copies its declared pointee; it does not infer the extent of an arbitrary interior pointer or external native allocation. + +## checks + +`python3 tests/drivers/actor_test.py bin/zy` and `python3 tests/drivers/actor_test.py bin/zym` exercise library-only syntax installation, declared actors, independent captures, two-instance parallelism, callback registration before send, statement delivery, fifo ordering, typed graph copies, closure messages, lifecycle behavior and rejection fixtures. the parallel fixture measures worker time inside the vm: two 300 ms tasks must complete in less than 580 ms. diff --git a/docs/BORROW.md b/docs/BORROW.md index 78b28e9..818eea4 100644 --- a/docs/BORROW.md +++ b/docs/BORROW.md @@ -1,89 +1,59 @@ -# a quieter ownership and lifetime checker +# borrowing, ownership, and lifetime rules -## objective +the compiler records ownership, aliases, parameter effects, local lifetime events, array projections, and source locations while it parses and emits code. these facts allow common borrows and transfers without spelling `.m`, `.d`, or `u*` at every call site. unknown facts remain conservative; low-level word and native boundaries still need explicit contracts or unchecked code. -accept ordinary, safe code with little annotation. infer ownership transfers and borrow duration when the compiler can prove them, and explain the cases it cannot prove. measure this on a corpus of small programs rather than claiming the checker is generally smarter than rust's. +## borrowed arguments and temporary results -rust already infers non-lexical lifetimes from control flow; reproducing that idea would not establish an advantage. its checker also tracks initialization and moves. zc can choose different ergonomics for its single-threaded ownership model and collector, while documenting where that model provides fewer guarantees. see the [rust compiler guide](https://rustc-dev-guide.rust-lang.org/borrow-check.html). - -## current behavior - -the analyzer runs alongside parsing and code emission. a local records an owner, scope, and state: held here, moving up, moving down, borrowed, or dead. snapshots reconcile branches and loops; derived pointer aliases retain their owner. explicit `.m`, `.d`, and `.u` tags describe borrowing and transfers. - -passing a fresh owned result directly to a borrowing parameter is deliberately rejected by `argcheck`. only a consuming `.d` parameter can currently receive it without a named local. naming the result gives the caller a place to insert cleanup. accepting the same call without generating that cleanup would change lifetime behavior and can leak the temporary until collection. +a function body records whether each parameter is read, written, retained, consumed, or returned as an alias. summaries for checked functions are propagated through calls to a fixed point, including recursion. when a fresh owned result is passed to a parameter proved not to retain or consume it, the compiler stores the result in a hidden frame slot for the call and releases it on every exit from the containing expression. this makes calls such as `rt.put_s(message())` safe without adding an explicit local. nested calls, short-circuit branches, conditional expressions, loops, and early returns use the same cleanup path rules. ```zc -char*.u message() { char* s = new(2); s[0] = 'x'; s[1] = 0; return u*s; } +namespace; +include rt; +char*.u message() { char* s = rt.new(2); s[0] = 'x'; s[1] = 0; return s; } fn main() { - let text = message(); - put_s(text); - return 0; + rt.put_s(message()); } ``` -plain borrowing parameters do not provide a complete interprocedural escape contract. unchecked code, raw host calls, integer-to-pointer conversions, and unknown native functions remain boundaries of static reasoning. - -## proposed rules - -### temporary borrows - -allow `put_s(message())` when the callee is proved to borrow without retaining, freeing, or transferring the argument. evaluate arguments in their existing order, hold each owned temporary in a compiler-generated slot, call the function, then free the temporary after its last dependent use. keep a returned alias alive through the enclosing expression when necessary. - -do not substitute `.d`: the output function does not consume its argument. nested calls, short-circuit expressions, early returns, and error recovery need their own cleanup paths. a temporary cannot be destroyed while another argument or the result still refers to it. unknown or escaping callees keep the explicit binding requirement. - -rust also has temporary scopes and lifetime extension rules; the proposed benefit here is a smaller set of user-facing rules for zc, not the invention of temporary lifetimes. see the [rust reference](https://doc.rust-lang.org/reference/destructors.html#temporary-scopes). - -### function effects - -infer a summary for each parameter: read, write, retain, consume, or return an alias. distinguish a returned owned result from a returned borrow and record which argument the borrow depends on. compute summaries to a fixed point for recursive functions; cache them by function body and relevant callee summaries. +an alias returned from an argument keeps that temporary alive until the alias's last use in the expression. passing a temporary to a known retaining or consuming parameter is rejected; use an owned `.d` parameter for a transfer. a checked borrower may return a borrow of its parameter, but it cannot return a local object's borrow as if it outlived that object. unchecked, external, indirect, or otherwise unknown callees need an explicit borrow/transfer contract or a named binding where lifetime is visible. -start with the runtime and ordinary functions whose bodies are available. annotations remain useful for externs, separate compilation, and deliberately unchecked code. a missing summary means unknown effects, not a promise that the call is safe. +## inferred parameter effects and transfers -### borrows end at their last use +the compiler tracks parameter origins through typed pointer expressions and uses their effect summaries to decide what a call may do. a call to an explicit `.m` parameter is a borrow contract; `.d` is a transfer contract. for checked bodies, a plain parameter that retains or consumes an owner is diagnosed at the caller. owned values passed to `.d` parameters transfer implicitly, and definite later uses identify the transfer and use locations. a pointer returned as an alias carries the source owner when the compiler can prove one origin. -derive borrow duration from control flow. a temporary alias that is no longer used should not prevent a later move of its owner. infer ordinary moves into consuming calls and returns where there is one unambiguous owner; retain explicit tags for ambiguous transfers and low-level code. +effects are cached for the current compiler analysis and participate in the language server's bounded result cache. the server cache is invalidated by source and included-file changes. summaries are not persisted as reusable per-function compiler records across arbitrary new compilations. externs, unchecked code, raw assembly, integer-to-pointer casts, untyped word storage, unknown indirect targets, and multiple ambiguous owners remain analysis boundaries. -use one fact graph for ownership, initialization, aliases, and known bounds. keep the existing parser and parse words: record analysis events from the real compiler, rather than adding an independent source parser that can disagree with it. +## last uses, fields, and elements -### fields, elements, and mutation +aliases are tied to their owner. when a tracked alias has no later use, a subsequent move or free can proceed; uses after a definite transfer or free produce an error that points to both the transfer and the later use. independent struct fields and constant array elements can be used when their projections prove they do not overlap. unknown indices and overlapping projections are not treated as disjoint. -track accesses by owner and projection: a struct field, a constant array index, or an element range. allow disjoint accesses when their projections are proved separate. invalidate those facts after an overlapping write, resize, unknown call, or unchecked operation. +after an owned local is released, assignment from another owner in the same declaring scope moves the replacement into that empty slot. the source and its aliases expire, the destination receives its cleanup responsibility, and loop paths still have to agree. this supports buffer-growth code such as `rt.free(buffer); buffer = replacement;` without a cast. initial declarations still cannot copy an owned local, overwriting a live owner remains an error, and captured owners are excluded from this replacement rule. -ordinary aliases may coexist in zc; mutation is safe only while the owner remains alive and the access stays valid. permitting aliases does not provide rust's concurrency guarantees. do not infer disjointness from uncertain indices, and do not treat a quarantined block's intact bytes as a valid lifetime. +an element borrow records its base array and known range. a constant index outside a known length gets a bounds diagnostic. mutations such as `push`, `pop`, and `resize` advance the array generation and invalidate element borrows; a later use reports the mutation and where the borrow was made. uncertain branch and loop facts are not promoted to definite errors. runtime checks remain for dynamic indices. -### useful diagnostics +## closure captures -report the operation that makes a later use invalid, the later use, and the relevant owner or bound. show known size, ownership, borrow source, and lifetime end in hover text. use "unknown" when a fact is unavailable. +closures share captured local and parameter cells with their enclosing function and sibling closures. updates are visible through every closure that captured the variable, including after collection and after the creating function returns. captured owned objects are kept alive by those cells. captures of short-lived borrowed or temporary values are rejected, and aggregate captures remain in place rather than being copied into a closure. -preserve hard errors for existing definite lifetime violations. add warnings for newly detected definite out-of-bounds access or use after free in code the old analyzer did not track. keep heuristic suggestions separate from definite findings, and avoid claiming an uncertain path is definitely invalid. +## limits and acceptance examples -## implementation order +the compiler functions, standard library, and language server currently run with lifetime checking enabled. compiler table allocations use an explicit `m`-tagged `rt.new(...)` collector-managed lifetime: there is no scope cleanup for those buffers, and live local or global roots keep them available to collection. owned source and token buffers return `.u`, while generic-token attachment consumes a `.d` buffer. generated native struct-return stubs keep their result owner checked and lend its numeric address to the native descriptor only for the call. -1. finish the existing maintenance and memory work in `TODO.md`; analysis tables must grow without silently dropping facts. -2. add compiler events with source ranges and a small control-flow representation. reproduce the current analyzer's accepted and rejected corpus before changing its rules. -3. infer non-escaping summaries for the runtime and add temporary cleanup for proved borrowing calls. -4. implement last-use borrow endings and unambiguous ownership transfer inference. -5. add projection facts, bounds facts, and explanatory editor previews. perform the final warning phase only after the preceding maintenance is complete. -6. compare the old and new checker on the same programs. expose unexplained behavior changes as regressions, not as an improvement count. +the analysis runs with the existing parser, ownership checker, and lifetime records; it does not have a separate source parser or a complete path-sensitive proof. snapshots conservatively join branches and loops. low-level and unknown boundaries can hide behavior, so a quiet diagnostic set is not a whole-program safety proof. zc is single-threaded; shared closure cells do not define concurrent mutation rules. -## acceptance cases +the maintained regressions cover these cases: -| case | required behavior | +| case | behavior | |---|---| -| an owned temporary passed to a proved non-escaping output function | accepted; one cleanup after its last use | -| a temporary passed to a callee that stores it globally | rejected unless ownership is transferred | -| a returned alias of a temporary used later in the same expression | accepted only with a proved lifetime extension | -| two arguments depending on the same owner | evaluate without freeing or moving that owner too early | -| an alias whose last use precedes an ownership transfer | accepted when no surviving alias escapes | -| independent struct fields or constant array elements | permit proved disjoint accesses | -| resize while an element alias remains live | reject or invalidate the alias according to an explicit rule | -| a use after a definite free or move | retain the error, with both source locations | -| an index definitely outside a known length | show a definite bounds finding | -| a dynamic index with no useful bound | retain runtime checks; no definite warning | -| unknown native or unchecked effects | require a contract or explicit escape hatch | - -verify cleanup counts, allocation counts, alias lifetimes, and output on the interpreter, microcore, and available native backends. tests must include safe neighbors of every rejected example. track annotations removed, safe programs accepted, compile time, and diagnostic precision as separate outcomes. - -## scope - -this is a design plan, not an implemented guarantee. lifetime inference must not change evaluation order or silently choose a different ownership contract. the initial work targets zc's present single-threaded execution model; shared mutable concurrency needs a separate model. +| owned temporary passed to a known non-retaining output function | held in a hidden frame slot and released after the call | +| temporary passed to a retaining or consuming callee | rejected unless ownership is transferred | +| returned alias of a temporary used later in the same expression | temporary stays alive through the dependent use | +| alias whose last use precedes an ownership transfer | transfer is accepted | +| released local receives a same-scope owned replacement | ownership and cleanup move; source and old aliases expire | +| two independent struct fields or constant array elements | permitted when projections prove separation | +| element alias used after array mutation | rejected with borrow and mutation locations | +| use after definite free or move | rejected with both source locations | +| definitely out-of-range constant index | diagnosed; unknown dynamic indices retain runtime checks | +| counter updated by one or more closures | all closures observe the shared state | + +`tests/programs/temporary_borrows.zc`, `tests/programs/borrow_rules.zc`, `tests/programs/replace_owner.zc`, and the focused error fixtures exercise these rules. the checker deliberately leaves ambiguous native, unchecked, opaque-word, and unknown-indirect effects conservative. diff --git a/docs/ICON.md b/docs/ICON.md new file mode 100644 index 0000000..9c222c8 --- /dev/null +++ b/docs/ICON.md @@ -0,0 +1,112 @@ +# runtime syntax transformations + +`src/lib/icon.zc` is a prototype for `icon` operations that transform compiler token lists and recompile retained functions. its current tests and examples do not constitute the full integration planned in TODO.md. include `run` when calling compiler operations directly, and `rt` for runtime output: + +```zc +include rt; +include run; +include icon; + +int addition() { return 7 + 3; } + +fn main() { + icon.find_replace(addition, ${ $int + $int }, ${ $int - $int }); + rt.put_n(addition()); // 4 + return 0; +} +``` + +the templates are compiler token lists. identifiers, strings, operators and numeric values remain distinct token kinds; a pattern cannot accidentally match text inside a string or comment. edits preserve token source lines for compiler diagnostics. + +## pattern captures + +a pattern contains ordinary tokens and capture markers. ordinary tokens match both token kind and value. capture markers accept the following ranges: + +| marker | accepted range | +| --- | --- | +| `$int` | one integer literal token | +| `$name` | one identifier token | +| `$expr` | one complete expression in the supported expression grammar | +| `$stmt` | one semicolon-terminated token range, brace block or braced `if`/`while` statement | +| `$block` | one balanced brace block | + +name a capture with `:name`: `$expr:left`. use `$left` in the replacement to insert its captured tokens. capture-class names are reserved and cannot also name a capture. repeated named captures require identical token sequences. unnamed captures are substituted by class and occurrence order: the first replacement `$int` uses the first captured integer, and the next uses the second. + +```zc +let source = run.tokens("return (9 + 2) * 3;"); +let pattern = run.tokens("$expr:left * $int:right"); +let replacement = run.tokens("$left + $right"); +let edited = icon.rewrite(source, pattern, replacement); +rt.put_n(run.eval(edited)); // 14 +``` + +expression recognition covers literals, names, grouping, calls, indexing, member access, unary `!`, `~`, `+`, `-`, `*` and `&`, postfix increment/decrement, ordinary binary operators and conditional expressions. capture classes describe token structure, rather than performing type inference: `$int` matches integer literals, while `$expr` also accepts variable expressions. `$stmt` checks a supported statement boundary rather than validating its entire grammar; blocks and braced `if`/`while` captures stop at their closing boundary. casts, custom operators, closure literals and compound initializers require exact token patterns or block captures. + +matching operates on balanced token ranges. it is not a type-directed tree rewrite. preserve parentheses when a replacement changes operator precedence. matches are nonoverlapping, replacement tokens are not searched again, and patterns cannot match an empty range. originals and templates remain unchanged. + +## editing token lists + +the edit operations return independent lists. a null result indicates a rejected edit; `icon.error()` supplies its status. + +| operation | behavior | +| --- | --- | +| `rewrite(tokens, pattern, replacement)` | replace every matching range | +| `splice(tokens, start, end, insert)` | replace the token range `[start, end)` | +| `undeclare(tokens, name)` | remove a complete, single-binding declaration statement | +| `change_type(tokens, name, type_tokens)` | replace the type prefix of the first simple declaration | +| `insert_block(tokens, position, body_tokens)` | insert a new scoped block | +| `calculate(tokens, start, end)` | compile and evaluate the selected expression | +| `validate(tokens)` | check delimiter structure | + +`undeclare` rejects declarations containing multiple bindings and statements that merely reference the name. simple declarations inside nested blocks are included in the search, and `change_type` preserves a preceding `const` qualifier. edits select the first declaration with the requested spelling; they do not perform name-resolution-based rewriting of every declaration or use. removing a declaration leaves uses elsewhere for the compiler to diagnose. `validate` checks delimiters, rather than all language rules; the compiler remains responsible for type, ownership and syntax checking before execution. + +`calculate` executes an expression through the compiler. calls in that expression retain their ordinary effects. it is an evaluator, rather than a purity or constant-folding guarantee. + +## publishing a function replacement + +`find_replace(function, pattern, replacement)` obtains the function's retained body tokens, transforms them, and asks the compiler to publish the replacement. it returns the number of matched ranges after successful publication, or zero when there is no match or publication is rejected. existing references to the original function entry use the published version. + +publication must preserve the original parameter and return contract. it does not retroactively alter frames that are already executing the original body. closure environment mutation and inspection of native execution frames require compiler or vm support beyond these token-list operations. + +## resumable execution and inspection + +`begin(tokens)` creates a resumable frame with its own token list, local binding map and global binding map. `begin_function(function)` starts from retained body tokens; provide each argument with `set_local` before execution. `step(frame)` executes one statement or one control-flow decision. `resume(frame, budget)` executes at most the requested number of steps, leaving an unfinished frame available for inspection or later execution. + +```zc +let code = run.tokens("int n = 4; n = n * 3; return n + bias;"); +let frame = icon.begin(code); +icon.set_global(frame, "bias", 2); +icon.step(frame); // n is 4 +icon.step(frame); // n is 12 +icon.step(frame); // completed result is 14 +rt.put_n(icon.result(frame)); +icon.close(d*frame); +``` + +the frame evaluator supports single-binding `int`, `word` and `let` declarations containing integer values, assignments and `+=`, `-=`, `*=`, `/=`, standalone increment and decrement, return, blocks, and braced `if`, `else` and `while`. expression evaluation uses the compiler after substituting the frame's integer bindings. increment or decrement inside an expression is rejected because substitution does not preserve an assignable location. block-local declarations are removed at block exit, and shadowed bindings become visible again. + +use `local(frame, name)` and `global(frame, name)` to read bindings; `set_local` and `set_global` update them. `local_count` and `local_name` enumerate the currently visible storage records, including shadowed records within an active block. `position` returns the next token position and `steps` reports executed steps. `result` is available after a return or normal completion. + +frame states are `0` for resumable execution, `1` for completion and `2` for an unsupported statement. unsupported syntax sets status `5` and does not execute the statement. `for`, switch and pattern matching, pointer or aggregate mutation, exceptions, and arbitrary native-frame suspension are not supported by this evaluator. direct reads of program globals require an explicit frame binding. expressions may call ordinary program functions and use constants; those calls retain their ordinary effects and run to completion within the step. + +`bind_global(frame, name)` explicitly connects one global binding to a program global of type `int` or `word`. reads observe its current value and frame assignments update its actual storage. binding a function, array, pointer, packed integer, floating value or missing symbol is rejected. this operation changes the isolation boundary for that one binding, so use it only when updating the program global is intended. + +`close(d*frame)` releases the frame's tokens, bindings and control-flow storage. the caller must release independently created token lists according to their normal ownership rules. + +## status values + +| status | meaning | +| --- | --- | +| `0` | operation completed | +| `1` | invalid delimiters or malformed or empty pattern | +| `2` | invalid range, declaration or missing binding | +| `3` | replacement references an unavailable capture | +| `4` | function tokens are unavailable | +| `5` | unsupported frame statement | +| `6` | function publication rejected | + +the status applies to the most recent transformation or binding operation. successful binding reads and writes clear it; inspection of frame state does not clear it. function publication is staged: a parse, type or signature failure rejects the replacement and retains the previous function entry. compiler errors during ordinary expression evaluation follow the compiler's normal diagnostic and termination behavior; these status codes do not intercept them. + +## focused tests + +the acceptance, rejection and regression fixtures live under `tests/icon`. run them with `python3 tests/drivers/icon_test.py bin/zy`; use `--js` for the javascript vm. select one fixture or category with `--case`, for example `python3 tests/drivers/icon_test.py bin/zym --case loop_step`. add `--source` to test the working library source before rebuilding the embedded compiler module; omit it when checking the built image. diff --git a/docs/LEARN.md b/docs/LEARN.md index 676358a..8b0e1e6 100644 --- a/docs/LEARN.md +++ b/docs/LEARN.md @@ -1,6 +1,6 @@ # learning zc -zc is a small c-like language for a stack machine with 64-bit words. this guide starts with ordinary programs, then covers memory, types, native calls, and metaprogramming. every zc code block is a complete file in [examples/learn/](../examples/learn/). the output below it comes from the matching `.out` file. +zc is a small C-like language for a stack machine with 64-bit words. this guide starts with ordinary programs, then covers memory, types, native calls, and metaprogramming. every zc code block is a complete file in [examples/learn/](../examples/learn/). the output below it comes from the matching `.out` file. ## setup @@ -13,12 +13,14 @@ bin/zy < examples/learn/01-hello.zc `bin/zy` is the native core: it uses the arm64 jit where available and otherwise interprets. `bin/zym` is the portable microcore, `bin/zc0` is the bootstrap compiler, and `bin/zclsp` checks source for the editor. the interactive runners are `./tinyvm.sh` on linux and `tinyvm.bat` on windows. -run from the root throughout this guide. paths in `include` and `readfile` are resolved from the process's working directory. `tests/drivers/learn.py` checks the examples and verifies that the code blocks here match their source files: +run from the root throughout this guide. paths in `rt.readfile` are resolved from the process's working directory, and so are those in `include` when nothing nearer has the file. `tests/drivers/learn.py` checks the examples and verifies that the code blocks here match their source files: ```sh python3 tests/drivers/learn.py bin/zy ``` +example 23 uses the native C ABI and requires host FFI support. the microcore does not provide that support, so skip example 23 when running the guide on a micro-only host; the learning driver does not currently select examples by backend capability. + native c calls need the full native core. the raw syscall example needs linux on x86-64. the other examples use ordinary language features; availability of host services still depends on the port. ## contents @@ -28,24 +30,28 @@ native c calls need the full native core. the raw syscall example needs linux on - [ownership and the heap](#ownership-and-the-heap) - [libraries and the host](#libraries-and-the-host) - [types and containers](#types-and-containers) +- [inferred call types](#inferred-call-types) +- [const fixed arrays](#const-fixed-arrays) - [optimized regions](#optimized-regions) - [closures](#closures) +- [editor hints](#editor-hints) - [further reading](#further-reading) ## programs and control flow ### the first program -a unit is read to eof or a backtick outside a comment or literal. after compiling the unit, the runtime calls `main`. an ordinary function returns one word; falling off its end returns 0. the return value of `main` is not the process exit status. `halt()` stops the machine. +a unit is read to eof or a backtick outside a comment or literal. after compiling the unit, the runtime calls `main`. an ordinary function returns one word; falling off its end returns 0. the return value of `main` is not the process exit status. `rt.halt()` stops the machine. -`put_s(s)` prints a string, `put_n(n)` prints an integer, and `put_c(c)` prints a character code. `put_c(10)` prints a newline. +`rt.put_s(s)` prints a string, `rt.put_n(n)` prints an integer, and `rt.put_c(c)` prints a character code. `rt.put_c(10)` prints a newline. [example source](../examples/learn/01-hello.zc): ```zc +namespace; +include rt; fn main() { - put_s("hello, world!\n"); - return 0; + rt.put_s("hello, world!\n"); } ``` @@ -64,11 +70,12 @@ hello, world! [example source](../examples/learn/02-loops.zc): ```zc +namespace; +include rt; fn main() { let i = 0, n = 0; while (i < 4) { n = n + i; i++; } - put_n(n); put_c(10); - return 0; + rt.put_n(n); rt.put_c(10); } ``` @@ -78,6 +85,28 @@ output: 6 ``` +### file namespaces and multiple includes + +`namespace app;` scopes declarations in this file. `include rt, std;` makes both namespaces available here; an included library keeps its own namespace and does not import its dependencies into `app`. `let` infers the word-string type from `"21"`, and the qualified standard-library call parses it. + +[example source](../examples/learn/27-namespaces.zc): + +```zc +namespace app; +include rt, std; +int twice(int n) { return n * 2; } +fn main() { + let text = "21"; + rt.put_n(twice(std.Text.atoi(text))); rt.put_c(10); +} +``` + +output: + +```text +42 +``` + ### do and continue `do` runs the body once before testing the condition. in a `do` loop, `continue` jumps to the condition; in a `for` loop, it jumps to the increment step. this example skips adding 2. @@ -85,11 +114,12 @@ output: [example source](../examples/learn/03-do-continue.zc): ```zc +namespace; +include rt; fn main() { int n = 0, i = 0; do { i++; if (i == 2) continue; n = n + i; } while (i < 5); - put_n(n); put_c(10); - return 0; + rt.put_n(n); rt.put_c(10); } ``` @@ -106,11 +136,12 @@ output: [example source](../examples/learn/17-switch.zc): ```zc +namespace; +include rt; fn main() { int k = 2; - switch (k) { case 1: put_s("one"); break; case 2: put_s("two"); break; default: put_s("other"); } - put_c(10); - return 0; + switch (k) { case 1: rt.put_s("one"); break; case 2: rt.put_s("two"); break; default: rt.put_s("other"); } + rt.put_c(10); } ``` @@ -129,10 +160,11 @@ with the typechecker enabled, `color` is its own type: members have that type, a [example source](../examples/learn/04-enums.zc): ```zc +namespace; +include rt; enum color { RED, GREEN = 5, BLUE }; fn main() { - put_n(RED); put_c(32); put_n(GREEN); put_c(32); put_n(BLUE); put_c(10); - return 0; + rt.put_n(RED); rt.put_c(32); rt.put_n(GREEN); rt.put_c(32); rt.put_n(BLUE); rt.put_c(10); } ``` @@ -148,19 +180,20 @@ output: `u8`, `i8`, `u16`, `i16`, `u32`, and `i32` narrow on assignment. adding 10 to an unsigned byte holding 250 leaves 4. arrays and pointers of these types use packed storage. -addresses normally count words. `sizeof` also counts words, so `[100]u8` occupies 13 words. an `int*` points to word-sized integers; `p + 1` advances one word. a sized pointer such as `u8*` indexes packed elements. +addresses normally count words. `sizeof` also counts words, so `u8[100]` occupies 13 words. an `int*` points to word-sized integers; `p + 1` advances one word. a sized pointer such as `u8*` indexes packed elements. [example source](../examples/learn/05-small-ints.zc): ```zc +namespace; +include rt; fn main() { u8 a = 250; a = a + 10; - put_n(a); put_c(10); - put_n(sizeof([100]u8)); put_c(10); - u8* px = new(u8, 4096); + rt.put_n(a); rt.put_c(10); + rt.put_n(sizeof(u8[100])); rt.put_c(10); + u8* px = rt.new(u8, 4096); px[10] = 300; - put_n(px[10]); put_c(10); - return 0; + rt.put_n(px[10]); rt.put_c(10); } ``` @@ -176,16 +209,17 @@ output: a literal such as `1.5` is an ieee binary64 value in one word. `float` and `f64` name that type; `f32` rounds on store. integer operands are promoted when mixed with floats in arithmetic. -`put_n` prints integers. the helper `put_f` below calls the vm's float printer with encoded instruction `#1220`. division by zero gives `inf`; `0.0 / 0.0` gives `nan`. both `0.0` and `-0.0` are false in conditions. +`rt.put_n` prints integers. the helper `put_f` below calls the vm's float printer with encoded instruction `#1220`. division by zero gives `inf`; `0.0 / 0.0` gives `nan`. both `0.0` and `-0.0` are false in conditions. [example source](../examples/learn/06-floats.zc): ```zc -fn put_f(x) { asm { x @ #1220 } return 0; } +namespace; +include rt; +void put_f(float x) { asm { x @ #1220 } } fn main() { let a = 1.5; - put_f(a + 2.5); put_c(10); - return 0; + put_f(a + 2.5); rt.put_c(10); } ``` @@ -197,19 +231,20 @@ output: ### byte addresses -`addr_b(p)` converts a word address to a byte address; `addr_w(b)` converts back. `ld8`, `ld16`, and `ld32` load little-endian values; `st8`, `st16`, and `st32` store them. byte addresses name the same memory as word addresses. +`rt.addr_b(p)` converts a word address to a byte address; `rt.addr_w(b)` converts back. `ld8`, `ld16`, and `ld32` load little-endian values; `st8`, `st16`, and `st32` store them. byte addresses name the same memory as word addresses. this example stores `0x11` in the low byte and `0x22` in the next byte. the resulting word is `0x2211`, or 8721. [example source](../examples/learn/07-bytes.zc): ```zc +namespace; +include rt; fn main() { let w = 0; - st8(addr_b(&w), 0x11); - st8(addr_b(&w) + 1, 0x22); - put_n(ld8(addr_b(&w))); put_c(32); put_n(w); put_c(10); - return 0; + rt.st8(rt.addr_b(&w), 0x11); + rt.st8(rt.addr_b(&w) + 1, 0x22); + rt.put_n(rt.ld8(rt.addr_b(&w))); rt.put_c(32); rt.put_n(w); rt.put_c(10); } ``` @@ -221,20 +256,21 @@ output: ### packed byte strings -`"text"` is a word string: one character per word, followed by a zero word. `b"text"` is a packed byte string: a length word, then eight characters per word and a terminating zero byte. write the `b` directly against the quote. +`"text"` is a word string of type `char*`: one character per word, followed by a zero word. `b"text"` is a packed byte string of type `bstr`: a length word, then eight characters per word and a terminating zero byte. write the `b` directly against the quote. -for byte strings, `len`, indexing, and `put_s` use the packed representation. indexing reads a byte and cannot be assigned through. `+` concatenates strings, `s[i:j]` makes a slice with an exclusive end, and comparisons order unsigned bytes. `s_to_n` parses an integer; `n_to_s` creates a byte string containing its decimal form. +for byte strings, `len` and indexing use the packed representation, and `rt.put_s` writes the packed bytes. indexing reads a byte and cannot be assigned through. `+` concatenates strings, `s[i:j]` makes a slice with an exclusive end, and comparisons order unsigned bytes. `rt.s_to_n` parses an integer; `rt.n_to_s` creates a byte string containing its decimal form. [example source](../examples/learn/08-byte-strings.zc): ```zc +namespace; +include rt; fn main() { let name = b"zc"; let hi = b"hello, " + name; - put_s(hi); put_c(10); - put_n(len(hi)); put_c(32); - put_n(s_to_n(b"40") + 2); put_c(10); - return 0; + rt.put_s(hi); rt.put_c(10); + rt.put_n(len(hi)); rt.put_c(32); + rt.put_n(rt.s_to_n(b"40") + 2); rt.put_c(10); } ``` @@ -245,24 +281,48 @@ hello, zc 9 42 ``` +### inferred string types + +`let` preserves the distinct types of the two string literal forms: `"..."` infers `char*`, while `b"..."` infers `bstr`. the standard-library groups accept the matching type, so each length call below uses the correct representation. + +[example source](../examples/learn/28-string-inference.zc): + +```zc +namespace; +include rt, std; +fn main() { + let word_text = "word string"; + let packed_text = b"packed bytes"; + rt.put_n(std.Text.len(word_text)); rt.put_c(32); + rt.put_n(std.Bytes.len(packed_text)); rt.put_c(10); +} +``` + +output: + +```text +11 12 +``` + ## ownership and the heap ### shared cells -`cons(a, b)` creates an immutable pair, read with `car` and `cdr`. `triple` and `tget` do the same for three values. several pointers may share a cell. `share(p)` publishes an existing object, and `is_shared(p)` reports whether it is shared. +`rt.cons(a, b)` creates an immutable pair, read with `rt.car` and `rt.cdr`. `rt.triple` and `rt.tget` do the same for three values. several pointers may share a cell. `rt.share(p)` publishes an existing object, and `rt.is_shared(p)` reports whether it is shared. -`free` leaves shared cells intact. `gc()` reclaims them once no root points to them, including cycles. `scrub()` clears unused stack words that would otherwise look like roots. here, `left` and `right` share the same tail. +`rt.free` leaves shared cells intact. `rt.gc()` reclaims them once no root points to them, including cycles. `rt.scrub()` clears unused stack words that would otherwise look like roots. in this example, `left` and `right` share the same tail. [example source](../examples/learn/09-shared.zc): ```zc +namespace; +include rt; fn main() { - let tail = cons(1, 2); - let left = cons(tail, 3); - let right = cons(tail, 4); - put_n(car(car(left))); put_c(32); - put_n(car(left) == car(right)); put_c(10); - return 0; + let tail = rt.cons(1, 2); + let left = rt.cons(tail, 3); + let right = rt.cons(tail, 4); + rt.put_n(rt.car(rt.car(left))); rt.put_c(32); + rt.put_n(rt.car(left) == rt.car(right)); rt.put_c(10); } ``` @@ -274,18 +334,19 @@ output: ### owned objects -binding a fresh `new` result to a local makes that local its owner. the analyzer inserts a free when the scope ends and reports an owned object that is never used. you normally do not write the free yourself. +binding a fresh `rt.new` result to a local makes that local its owner. the analyzer inserts `rt.free` when the scope ends and reports an owned object that is never used. explicit `rt.free` calls are usually unnecessary for owned locals. `int*` is a pointer to an integer. `*p` accesses that integer. [example source](../examples/learn/10-owned.zc): ```zc +namespace; +include rt; fn main() { - int* p = new(int); + int* p = rt.new(int); *p = 7; - put_n(*p); put_c(10); - return 0; + rt.put_n(*p); rt.put_c(10); } ``` @@ -300,19 +361,22 @@ output: tags describe where an owned object goes: - `.m` keeps it in the current block. -- `.d` hands it down to a callee or container; the previous owner cannot use it afterward. -- `.u` hands it up through a return, block value, or assignment to an enclosing scope. +- `.d` transfers it to a callee or container; the previous owner cannot use it afterward. +- `.u` transfers it through a return, block value, or assignment to an enclosing scope. -`m*p`, `d*p`, and `u*p` recast the tag. a plain parameter lends the object without transferring ownership; `.m` on a parameter adds a checked borrow. `free(d*p)` hands the object to `free`. an owned result can be declared with `.u` or inferred from its return. +assigning an owner to a global transfers it to the collector automatically; the local cannot be used afterward, and a borrowed reference or a child of a container is rejected. + +`m*p`, `d*p`, and `u*p` remain available for explicit low-level recasts. a plain parameter lends the object without transferring ownership; `.m` on a parameter adds a checked borrow. a `.d` parameter and `rt.free` consume an owned argument automatically, and returning an owned local transfers it to the caller without `u*`. a fresh object passed to a proved borrower, such as `rt.put_s(message())`, is held through the call and freed afterward. an owned result can be declared with `.u` or inferred from its return. [example source](../examples/learn/11-tags.zc): ```zc -int*.u box(int v) { int* b = new(int); *b = v; return u*b; } +namespace; +include rt; +int*.u box(int v) { int* b = rt.new(int); *b = v; return b; } fn main() { int* p = box(4); - put_n(*p); put_c(10); - return 0; + rt.put_n(*p); rt.put_c(10); } ``` @@ -331,13 +395,14 @@ in this example, `a` owns `b` after `a.next = b`, and `c` borrows the child. ind [example source](../examples/learn/12-aliases.zc): ```zc +namespace; +include rt; struct node { int v; node* next; }; fn main() { - node* a = new(node), b = new(node); + node* a = rt.new(node), b = rt.new(node); a.v = 3; b.v = 4; a.next = b; node* c = a.next; - put_n(c.v); put_c(10); - return 0; + rt.put_n(c.v); rt.put_c(10); } ``` @@ -351,29 +416,30 @@ output: `unchecked fn` and `unchecked { }` disable lifetime analysis for their scope. allocations there rely on the collector. checked variables cannot be accessed from an unchecked block. -`memdebug on;` adds runtime checks for heap loads and stores, use after free, and double free. `memreport()` lists live allocations made while debugging was on. the allocation line below is the line containing `new`; the reported size is the buddy block's payload capacity. +`memdebug on;` adds runtime checks for heap loads and stores, use after free, and double free. `rt.memreport()` lists live allocations made while debugging was on. the allocation line below is the line containing `rt.new`; the reported size is the buddy block's payload capacity. -`memdebug scrub;` also fills freed payloads with `0xDEAD`; `memdebug off;` disables debugging. ordinary `free` quarantines an owned block until collection can safely reuse it. stale contents surviving quarantine are not permission to use a dead object. +`memdebug scrub;` also fills freed payloads with `0xDEAD`; `memdebug off;` disables debugging. ordinary `rt.free` quarantines an owned block until collection can safely reuse it. stale contents surviving quarantine are not permission to use a dead object. [example source](../examples/learn/13-memdebug.zc): ```zc +namespace; +include rt; memdebug on; int* g; fn main() { - int* p = new(int); + int* p = rt.new(int); *p = 1; g = d*p; - put_n(memreport()); - put_c(10); - return 0; + rt.put_n(rt.memreport()); + rt.put_c(10); } ``` output: ```text -leak: line 4 3 words +leak: line 6 3 words 1 ``` @@ -381,19 +447,19 @@ leak: line 4 3 words ### including a library -`include "src/lib/std.zc";` reads and compiles that file at the include. place it before code that uses the library. the path is relative to the process's working directory. +`include std;` compiles the standard library at that point. it is part of the compiler, so no file is read, and it must appear before code that uses it. the library uses the `std` namespace, so call its helpers as `std.Text.itoa(...)` or `std.Host.now()`. each file has its own namespace and imports: an included file does not import its dependencies into the caller. `include rt, std;` imports both namespaces into the current file. `include "path";` reads and compiles another file the same way. the file is looked for next to the file that has the include (for a program on stdin, that step is skipped), then in the workspace folder when an editor gives one, then relative to the process's working directory. -the library groups related helpers in structs and calls them as `Group.name(args)`. for example, `Text.itoa` writes a decimal integer into a caller-provided word-string buffer. +the library groups related helpers in structs and calls them as `std.Group.name(args)`. for example, `std.Text.itoa` writes a decimal integer into a caller-provided word-string buffer. [example source](../examples/learn/14-include.zc): ```zc -include "src/lib/std.zc"; +namespace; +include rt, std; fn main() { - let s = readfile("tests/data/hi.txt"); - put_n(len_w(s)); put_c(10); - free(d*s); - return 0; + let s = rt.readfile("tests/data/hi.txt"); + rt.put_n(rt.len_w(s)); rt.put_c(10); + rt.free(s); } ``` @@ -405,18 +471,18 @@ output: ### text and number helpers -`Text` provides word-string parsing, formatting, copying, search, and case conversion. `Num` provides helpers such as `gcd`, `pow`, `isqrt`, and `clamp`. `Bytes` provides packed-string helpers. `Host`, `File`, `Arena`, and `Buf` cover host services, files, allocation arenas, and byte buffers. +`std.Text` provides word-string parsing, formatting, copying, search, and case conversion. `std.Num` provides helpers such as `gcd`, `pow`, `isqrt`, and `clamp`. `std.Bytes` provides packed-string helpers. `std.Host`, `std.Arena`, and `std.Buf` cover host services, files, allocation arenas, and byte buffers. check the signatures in `src/lib/std.zc` before passing a buffer: many text functions write into storage supplied by the caller. [example source](../examples/learn/20-text-num.zc): ```zc -include "src/lib/std.zc"; +namespace; +include rt, std; fn main() { - put_n(Num.gcd(12, 18)); put_c(32); put_n(Num.pow(2, 10)); put_c(32); put_n(Num.isqrt(50)); put_c(10); - put_n(Text.atoi("123") + 1); put_c(10); - return 0; + rt.put_n(std.Num.gcd(12, 18)); rt.put_c(32); rt.put_n(std.Num.pow(2, 10)); rt.put_c(32); rt.put_n(std.Num.isqrt(50)); rt.put_c(10); + rt.put_n(std.Text.atoi("123") + 1); rt.put_c(10); } ``` @@ -429,17 +495,17 @@ output: ### host services -`syscall(op, a, b, c)` passes a request through the vm's syscall gate. the runtime defines names such as `SYS_OPEN`, `SYS_READ`, `SYS_WRITE`, `SYS_GETENV`, and `SYS_TIME`. unsupported operations return -1. `Host.now()` wraps the clock request. +`rt.syscall(op, a, b, c)` calls the runtime interface to the vm's syscall gate. the runtime defines names such as `SYS_OPEN`, `SYS_READ`, `SYS_WRITE`, `SYS_GETENV`, and `SYS_TIME`. unsupported operations return -1. `std.Host.now()` wraps the clock request. -`getenvs(name)` returns a fresh word string or 0. the library also supplies `argc`, `argv`, `envc`, and `envp`; a program read on stdin has `<eval>` as its first argument. gate calls must be made from ordinary code, outside an `o3` region. +`std.Host.env(name, dst, n)` writes an environment value into a word-string buffer and returns its length, or -1 when it is unset. the library also supplies `std.argc`, `std.argv`, `std.envc`, and `std.envp`; a program read on stdin has `<eval>` as its first argument. gate calls must be made from ordinary code, outside an `o3` region. [example source](../examples/learn/21-host.zc): ```zc -include "src/lib/std.zc"; +namespace; +include rt, std; fn main() { - put_n(Host.now() > 0); put_c(10); - return 0; + rt.put_n(std.Host.now() > 0); rt.put_c(10); } ``` @@ -451,17 +517,17 @@ output: ### raw linux syscalls -`Host.raw` uses the kernel's own syscall number and argument convention. it returns the result or a negated errno. the syscall numbers differ by architecture: this example uses x86-64 linux `getpid`, number 39. +`std.Host.raw` uses the kernel's own syscall number and argument convention. it returns the result or a negated errno. the syscall numbers differ by architecture: this example uses x86-64 linux `getpid`, number 39. this path bypasses the vm's syscall checks. pointer arguments must use the host addresses required by that syscall. the example does not take pointer arguments and is skipped on other hosts. [example source](../examples/learn/22-raw-x86.zc): ```zc -include "src/lib/std.zc"; +namespace; +include rt, std; fn main() { - put_n(Host.raw(39, 0, 0, 0, 0, 0, 0) > 0); put_c(10); - return 0; + rt.put_n(std.Host.raw(39, 0, 0, 0, 0, 0, 0) > 0); rt.put_c(10); } ``` @@ -473,14 +539,15 @@ output: ### input after the source -a backtick outside a comment or literal ends the source unit. bytes after it are the program's input, read with `get_c`. `eof()` reports the end. this complete file includes both the program and the input `abc`; the program reads only the first two characters. +a backtick outside a comment or literal ends the source unit. bytes after it are the program's input, read with `rt.get_c`. `rt.eof()` reports the end. this complete file includes both the program and the input `abc`; the program reads only the first two characters. [example source](../examples/learn/24-backtick.zc): ```zc +namespace; +include rt; fn main() { - put_c(get_c()); put_c(get_c()); put_c(10); - return 0; + rt.put_c(rt.get_c()); rt.put_c(rt.get_c()); rt.put_c(10); } `abc ``` @@ -500,17 +567,18 @@ this example calls `sqrt`, then calls `qsort` with a zc comparison callback. nat [example source](../examples/learn/23-c-abi.zc): ```zc -fn put_f(x) { asm { x @ #1220 } return 0; } +namespace; +include rt; +void put_f(float x) { asm { x @ #1220 } } extern double sqrt(double); extern void qsort(word *base, long n, long size, void *cmp); int cmp(int* a, int* b) { return *a - *b; } fn main() { let a[4]; a[0] = 5; a[1] = 3; a[2] = 9; a[3] = 1; - put_f(sqrt(2.0)); put_c(10); + put_f(sqrt(2.0)); rt.put_c(10); qsort(a, 4, 8, cfunc(cmp)); - put_n(a[0]); put_c(32); put_n(a[1]); put_c(32); put_n(a[2]); put_c(32); put_n(a[3]); put_c(10); - return 0; + rt.put_n(a[0]); rt.put_c(32); rt.put_n(a[1]); rt.put_c(32); rt.put_n(a[2]); rt.put_c(32); rt.put_n(a[3]); rt.put_c(10); } ``` @@ -523,55 +591,158 @@ output: ## types and containers +### inferred call types + +the compiler propagates type constraints through calls and returned values until they stop changing. a typed callee can determine a caller's parameter and return types, including through chains of untyped functions declared before that callee. recursive calls can use a typed callee as an anchor. when callers disagree on an argument type, the compiler leaves the shared parameter unresolved; an unknown reassignment also discards the previous initializer's type evidence. owned raw allocation results retain the `.u` return tag while their inferred storage type remains `word`. + +[example source](../examples/learn/32-inferred-calls.zc): + +```zc +namespace; +include rt; + +fn layer0(n) { return seed(n); } +fn layer1(n) { return layer0(n); } +fn layer2(n) { return layer1(n); } +int seed(int n) { return n; } +fn countdown(n) { + if (n == 0) return seed(7); + return countdown(n - 1); +} +.u make_list(cap) { + let list = rt.new(cap + 1); + list[0] = cap; + return list; +} +.u pass_list(n) { + let list = make_list(n); + return list; +} +fn main() { + let list = pass_list(layer2(41)); + rt.put_n(list[0]); rt.put_c(32); + rt.put_n(countdown(3)); rt.put_c(10); +} +``` + +output: + +```text +41 7 +``` + +### const fixed arrays + +`fixed const T[n]` and `const fixed T[n]` both declare read-only fixed arrays. the qualifier applies to the array cells and to aliases of the array. element assignment, increment, compound assignment, and rebinding a `const fixed T[?]` handle are rejected, even when `typecheck off;` is active. a read-only array reference cannot be converted to a mutable reference. a formal parameter declared `const fixed T[n]` accepts either a read-only or mutable array. copying an element into an ordinary numeric variable remains allowed. + +[example source](../examples/learn/33-const-fixed.zc): + +```zc +namespace; +include rt; +int first(const fixed int[2] values) { return values[0]; } +fn main() { + const fixed int[2] a = {7, 8}; + fixed const int[2] b = {2, 3}; + fixed int[2] c = {5, 6}; + int copy = a[0]; + copy++; + rt.put_n(first(a)); rt.put_c(32); + rt.put_n(first(b)); rt.put_c(32); + rt.put_n(first(c)); rt.put_c(32); + rt.put_n(copy); rt.put_c(10); +} +``` + +output: + +```text +7 2 5 8 +``` + ### arrays and methods -`[n]T` is an array stored in place. `[]T` is a growable handle that owns its items, created with `new([]T)` or `new([]T, n)`. `push`, `pop`, `resize`, `len`, and `cap` operate on it. `fixed []T` has a length that cannot change. +`T[n]` is an array stored in place. `T[]` infers a fixed length from its initializer. `int[] a = {1, [5] = 9, 7};` has seven elements: the unspecified elements are zero, index 5 is 9, and index 6 is 7. a trailing comma is allowed. `int[2][3]` has two rows of three elements. character and byte arrays also accept string initializers, with an inferred bound including the terminating zero. nested arrays accept nested initializer lists or flat lists with braces omitted. designators can chain indices, as in `[1][0] = 9`, and later entries resume after that element. repeated designators overwrite the earlier value. `(int[]){1, 2, 3}` is a compound literal that can be passed to a function taking `int*` or `int[]`; the storage belongs to the surrounding scope. bare `{1, 2, 3}` in an expression infers an integer array. `T[?]` is a growable handle that owns its items, created with `rt.new(T[?])` or `rt.new(T[?], n)`. `push`, `pop`, `resize`, `len`, and `cap` also have receiver forms such as `a.push(x)` and `a.len()`. `fixed T[n]` and `fixed T[?]` support `len()` and `cap()` but not the mutators `push()`, `pop()`, and `resize()`. + +`include std;` registers read-only receiver algorithms for growable arrays and mutable fixed arrays. support for const-qualified arrays has not been verified. `find(value)` and `find_if(predicate)` return the first matching index or -1; `contains(value)`, `any(predicate)`, and `all(predicate)` return 1 or 0; `count(predicate)` returns the number of matches. `any` is false and `all` is true for an empty input. `map(transform)` and `filter(predicate)` return new growable handles, and `reduce(initial, combine)` folds from left to right, returning `initial` for an empty input. callback parameter types follow the element type, `map` infers its result type from the callback, `filter` preserves the element type, and `reduce` uses the initial value's type for its accumulator. algorithms snapshot the input length and visit indices in order; callbacks should not change that input's length. map and filter produce independent handles but shallow-share reference elements with the collector. aggregate-by-value elements are not supported by `push`, `pop`, `map`, or `filter`; use pointers to records instead. struct methods receive a hidden `this` parameter: `p.get()` passes `p`. static functions belong to the struct and are called as `cnt.make()`. freeing a container also frees its owned children. [example source](../examples/learn/19-growable.zc): ```zc +namespace; +include rt; struct cnt { int n; static int make() { return 3; } int get() { return this.n; } }; fn main() { - []int xs = new([]int); - push(xs, 4); push(xs, 5); - put_n(len(xs)); put_c(10); - put_n(pop(xs)); put_c(10); - put_n(cnt.make()); put_c(10); - return 0; + int[?] xs = rt.new(int[?]); + xs.push(4); xs.push(5); + rt.put_n(xs.len()); rt.put_c(32); rt.put_n(xs.cap()); rt.put_c(10); + rt.put_n(xs.pop()); rt.put_c(10); + xs.resize(4); + rt.put_n(xs.len()); rt.put_c(10); + int[3] fixed = {1, 2, 3}; + rt.put_n(fixed.len()); rt.put_c(32); rt.put_n(fixed.cap()); rt.put_c(10); + rt.put_n(cnt.make()); rt.put_c(10); } ``` output: ```text -2 +2 4 5 +4 +3 3 3 ``` +the standard-library methods also accept mutable fixed arrays as read-only inputs. const-qualified array support has not been verified. `map` infers its callback parameter from the element type, `filter` preserves that type, and `reduce` combines elements in index order. + +[example source](../examples/learn/31-array-algorithms.zc): + +```zc +namespace; +include rt, std; + +fn main() { + int[?] values = {1, 2, 3, 4, 5}; + let doubled = values.map(fn(e) { return e * 2; }); + let large = doubled.filter(fn(e) { return e > 4; }); + rt.put_n(values.find(3)); rt.put_c(32); + rt.put_n(values.find_if(fn(e) { return e > 4; })); rt.put_c(32); + rt.put_n(large.reduce(0, fn(total, e) { return total + e; })); rt.put_c(10); +} +``` + +output: + +```text +2 4 24 +``` + ### generic functions and structs -type parameters follow the name: `record<K, V>` and `keyof<K, V>`. the compiler creates a distinct function instance for each set of concrete types. you can supply type arguments explicitly or infer them from the call. +type parameters follow the name: `record<K, V>` and `keyof<K, V>`. the compiler creates a distinct function instance for each set of concrete types. type arguments can be supplied explicitly or inferred from the call. `record<int, point>` stores its `point` field in place. inside an instantiated function, `sizeof` uses the concrete type. a generic body is checked when it is instantiated, so an unused template can contain a mistake that has not been reported yet. [example source](../examples/learn/15-generics.zc): ```zc +namespace; +include rt; struct record<K, V> { K key; V val; }; struct point { int x; int y; }; K keyof<K, V>(record<K, V>* r) { return r.key; } fn main() { - record<int, point>* r = new(record<int, point>); + record<int, point>* r = rt.new(record<int, point>); r.key = 9; - put_n(keyof(r)); put_c(10); - return 0; + rt.put_n(keyof(r)); rt.put_c(10); } ``` @@ -590,6 +761,8 @@ an enum member can carry fields written as a block. a constructor such as `shape [example source](../examples/learn/16-tagged-enums.zc): ```zc +namespace; +include rt; enum shape { circle { int r; }, rect { int w; int h; }, dot }; int area(shape* s) { switch (s) { @@ -601,12 +774,11 @@ int area(shape* s) { } fn main() { shape* s = shape.circle(3); - put_n(area(s)); put_c(10); - free(d*s); + rt.put_n(area(s)); rt.put_c(10); + rt.free(s); shape* t = shape.rect(2, 5); - put_n(area(t)); put_c(10); - free(d*t); - return 0; + rt.put_n(area(t)); rt.put_c(10); + rt.free(t); } ``` @@ -623,20 +795,21 @@ output: `o3` can mark a function, block, or variable. the compiler emits a region for the full core to optimize and compile into native code. the microcore and javascript runtime execute the ordinary bytecode fallback. -inside a region, locals and parameters must be typed. promoted variables have no address, so `&x` is not allowed for one. gate calls and closures are restricted. an untyped `new(n)` is zeroed without a type stamp. the fixed array below keeps its length, while the optimized block counts 143 matching integers. +inside a region, locals and parameters must be typed. promoted variables have no address, so `&x` is not allowed for one. gate calls and closures are restricted. an untyped `rt.new(n)` is zeroed without a type stamp. the fixed array below keeps its length, while the optimized block counts 143 matching integers. [example source](../examples/learn/18-o3.zc): ```zc -o3 int sum(fixed []int a) { int s = 0; for (int i = 0; i < len(a); i++) s += a[i]; return s; } +namespace; +include rt; +o3 int sum(fixed int[?] a) { int s = 0; for (int i = 0; i < len(a); i++) s += a[i]; return s; } fn main() { - fixed []int a = new(fixed []int, 5); + fixed int[?] a = rt.new(fixed int[?], 5); for (int i = 0; i < 5; i++) a[i] = i * i; - put_n(sum(a)); put_c(10); + rt.put_n(sum(a)); rt.put_c(10); o3 int hits = 0; o3 { for (int i = 0; i < 1000; i++) if (i % 7 == 3) hits++; } - put_n(hits); put_c(10); - return 0; + rt.put_n(hits); rt.put_c(10); } ``` @@ -649,22 +822,25 @@ output: ## closures -### capturing a value +### shared captures -`fn(x) { ... }` in an expression creates an owned closure. it copies the variables it uses from the surrounding function when it is created. calls use ordinary function syntax. +`fn(x) { ... }` in an expression creates an owned closure. it shares the variables it captures with the surrounding function and other closures. calls use ordinary function syntax. -bind a closure to a local before lending it to a plain parameter. to return an owned closure, use an owned return such as `return u*f;`; the result tag can be inferred or declared as `.u`. +write `fn:type(args)` to declare the result type, as in `fn:float(int x) { return x; }`; each `return` is converted and checked against it, and hovers show the declared result instead of an inferred one. + +captured locals and parameters use shared cells. a counter closure and its enclosing function see the same updates, and an escaping closure keeps those cells alive. a fresh closure can be passed to a checked borrowing parameter directly. returning an owned closure transfers it to the caller; its owned result can be inferred or declared as `.u`. [example source](../examples/learn/25-closures.zc): ```zc +namespace; +include rt; fn apply(f, x) { return f(x); } fn main() { let k = 10; let add = fn(x) { return x + k; }; - put_n(add(1)); put_c(10); - put_n(apply(add, 5)); put_c(10); - return 0; + rt.put_n(add(1)); rt.put_c(10); + rt.put_n(apply(add, 5)); rt.put_c(10); } ``` @@ -677,25 +853,26 @@ output: ### nested captures and mutable state -a nested closure can capture a variable from more than one enclosing function. each enclosing closure carries the values needed by the next one. +a nested closure can capture a variable from more than one enclosing function. each enclosing closure carries the shared cells needed by the next one. -captures are by value, but a closure can change its own copy. `tick` keeps its counter between calls; the `count` in `main` stays 0. changing closure state does not write back into the function that made it. +captured variables use shared storage. `tick` keeps its counter between calls, and `count` in `main` sees those changes. sibling and nested closures see the same updates; an escaping closure keeps its cells alive. [example source](../examples/learn/26-closure-state.zc): ```zc +namespace; +include rt; fn main() { let base = 2; let outer = fn(x) { let inner = fn(y) { return y + base + x; }; return inner(100); }; - put_n(outer(7)); put_c(10); + rt.put_n(outer(7)); rt.put_c(10); let count = 0; let tick = fn(d) { count = count + d; return count; }; - put_n(tick(1)); put_c(32); put_n(tick(4)); put_c(10); - put_n(count); put_c(10); - return 0; + rt.put_n(tick(1)); rt.put_c(32); rt.put_n(tick(4)); rt.put_c(10); + rt.put_n(count); rt.put_c(10); } ``` @@ -704,7 +881,62 @@ output: ```text 109 1 5 -0 +5 +``` + +### typed closures and returned ownership + +`fn:int(int n) { ... }` declares the closure's result and parameter types. the returned closure is an owned heap object; `word.u` marks the factory result as transferring that object to its caller. its captured `factor` remains available after the factory returns. + +[example source](../examples/learn/30-typed-closure.zc): + +```zc +namespace; +include rt; + +word.u multiplier(int factor) { + let scale = fn:int(int n) { return n * factor; }; + return u*scale; +} + +fn main() { + let scale = multiplier(3); + rt.put_n(scale(14)); rt.put_c(10); +} +``` + +output: + +```text +42 +``` + +## editor hints + +the language server reports informational hints for likely mistakes. by default, `hints on;` shows blue informational squiggles. `hints small;` keeps the same hints but displays compact dots; it does not reduce their number or kinds. `hints off;` suppresses hints after the directive. each setting applies from its source position. + +an inline block comment immediately before a numeric literal marks that literal as intentional and suppresses its magic-number hint. a comment before another operand does not suppress the hint. a block comment immediately above a declaration provides hover documentation; `//:` marks an implementation note instead. + +[example source](../examples/learn/29-block-comments.zc): + +```zc +namespace; +include rt; + +/* adds a memory offset to an address. */ +int offset_address(int address) { + return address + /* offset in memory */ 0x40ff; +} + +fn main() { + rt.put_n(offset_address(1)); rt.put_c(10); +} +``` + +output: + +```text +16640 ``` ## the optional typechecker @@ -716,3 +948,5 @@ output: [README.md](../README.md) is the full reference. `tests/` contains runnable feature checks, `tests/errors/` contains rejected lifetime examples, and `tests/typeerrors/` contains rejected type examples. `docs/design/` explains implementation decisions; `docs/history/` records earlier versions. use `make check` while editing and `make test` for the full available suite. enable `ZYTRACE=1` to inspect native translation, or `ZYHOT=1` on an llvm build to compile each block as early as possible. + +`//:` marks an implementation comment that never becomes hover documentation. use `/* ... */` or ordinary comments for declaration descriptions, and `//:` for section overviews and internal notes. diff --git a/docs/design/diagnostic-style.md b/docs/design/diagnostic-style.md new file mode 100644 index 0000000..7ac7da6 --- /dev/null +++ b/docs/design/diagnostic-style.md @@ -0,0 +1,3 @@ +# diagnostic wording + +compiler and language-server diagnostics should identify the relevant source context and state the problem in concise, actionable terms. use lowercase prose, avoid contractions, and separate any suggested fix from the diagnostic message. these conventions follow the [clang diagnostic wording guidance](https://clang.llvm.org/docs/InternalsManual.html#diagnostic-wording). diff --git a/docs/design/memory.md b/docs/design/memory.md index bcbdf35..c282df5 100644 --- a/docs/design/memory.md +++ b/docs/design/memory.md @@ -68,6 +68,8 @@ document slots, uri text, hover text, declaration types, parsed type spans, reso inference function records retain stable addresses while their index grows. inference edits, replacement text, and annotation verification buffers also grow; later declarations are not dropped at the original function-count or edit-count limits. protocol method names, request identifiers, json keys, nesting paths, and header lines also grow; request identifiers are echoed in full. +completed hover, navigation, token and completion replies have a bounded 64-entry cache. keys include the document generation, request position and include epoch. source updates, included-file updates, saves and close/reopen invalidate dependent replies; cached json values do not retain mutable compiler-table addresses. `zc/cacheStats` reports cache hits, misses and compiler checks for regression tests. + ## freestanding backing the stub and board ports reserve `MICRO_PORT_WORDS` cells at build time; it defaults to the initial image and heap size. the vm's active size and growth limit still change at runtime, within that physical backing. set the macro when linking a board with more reserved ram. a rejected physical allocation preserves the current memory. hosted maps keep their heap above the fixed regions; the historical `fixed` preset retains its bounded layout for compatibility. diff --git a/docs/design/storage-analysis.md b/docs/design/storage-analysis.md index 59fbcde..cff713a 100644 --- a/docs/design/storage-analysis.md +++ b/docs/design/storage-analysis.md @@ -18,9 +18,9 @@ unchecked fn example() { ## what stays unknown -this is basic analysis of local facts, not a proof of the whole program. branches, loops, short-circuit operators, raw assembly, and nested compilation discard facts instead of merging uncertain paths. unknown callees invalidate facts about allocations passed to them. array mutations invalidate recorded lengths. pointer arithmetic, explicit casts, pointer-valued fields, stored aliases, and function return contracts are not inferred by this pass. +this is analysis of compiler-recorded facts, not a proof of the whole program. branch and loop snapshots stay conservative where paths disagree. typed pointer arithmetic, casts, fields, element ranges, and checked function parameter effects carry provenance when the compiler can establish it. known array mutations advance a generation and invalidate older element borrows. an unknown call or opaque operation drops facts it may change; an inferred checked function summary can preserve facts for a read-only borrower. -an unknown fact does not produce a definite warning. runtime array checks still run. the pass may miss an error across one of these boundaries; silence is not a safety guarantee. the broader analysis in [BORROW.md](../BORROW.md) remains a design plan. +unknown facts do not produce definite bounds or lifetime findings. runtime array checks still run. unchecked code, raw assembly, integer-to-pointer casts, untyped word storage, and unknown native or indirect calls remain boundaries. the ownership and temporary-slot rules, including those limits, are described in [BORROW.md](../BORROW.md). ## editor previews diff --git a/docs/design/test-coverage.md b/docs/design/test-coverage.md new file mode 100644 index 0000000..fae9356 --- /dev/null +++ b/docs/design/test-coverage.md @@ -0,0 +1,121 @@ +# test coverage inventory + +this inventory covers runtime ownership boundaries, fuzzing and growth, variable hovers, unused-argument hints, the `rt.s_to_n` definition, string-literal types, and public standard-library documentation. it records existing tests and focused runs made during the documentation pass; it does not claim that the full suite ran. + +## runtime `unchecked` boundaries + +`src/runtime/rt.zc` still marks these functions `unchecked`: `rt.copy_w`, `rt.fill_w`, `rt.free`, `rt.recycle`, `rt.recycle_leaf`, `rt.cons`, `rt.triple`, `rt.dnew`, `rt.dgrow`, `rt.dpush`, `rt.reserve_mark`, `rt.gc`, and `rt.readfile`. + +existing runtime acceptance coverage exercises portions of those paths: `tests/programs/allocfast.zc`, `buddy.zc`, `gc.zc`, `quarantine.zc`, `recycle.zc`, `roots.zc`, `share.zc`, `sys.zc`, and `strings.zc` cover allocation, collection, ownership cells, file access, and string conversion. `tests/drivers/compiler_growth_test.py` stresses mark work and collection after table growth. `tests/errors/unchecked_touch.zc` rejects access to a checked owned value from an unchecked block. `tests/errors/temporary_unknown.zc` and `storage_analysis_test.js` exercise conservative behavior for unknown or unchecked effects. + +this is indirect coverage, not proof that each runtime function is now checked. `tests/drivers/core_lsp_test.js` checks maintained source files without severity-1 diagnostics, but an explicit `unchecked` function can pass that check while bypassing ownership analysis. there is no test that enumerates the runtime annotations, nor a per-function checked-call acceptance and rejection matrix for these boundaries. in particular, removal still needs regressions for the ownership and drop contracts of raw word copying, allocator fast paths, shared cells, growable arrays, collector work, and file results. + +## fuzzing, image growth, memory limits, and lsp longevity + +| area | existing tests and what they assert | remaining gap | +| --- | --- | --- | +| compiler, runtime, and stdlib fuzz | `fuzz_compiler.py` generates 128 arithmetic expressions, 300 scoped allocations plus collection, 16 runtime compiles, source-immutability checks, and 32 randomized `Text`/`Bytes` cases; it compares native, microcore, and optionally javascript output. `fuzz_malformed.py` runs 256 random token streams and 64 byte mutations on native and microcore, checking for crashes, timeouts, and host disagreement. `fuzz_wasm.js` covers 128 expressions, 300 allocations plus collection, and 32 `Text`/`Bytes` cases. | malformed inputs have no independent accept/reject oracle; generated ownership/analyzer properties are not fuzzed. the wasm fuzz does not include runtime compilation or source-immutability checks. | +| image and code growth | `image_growth_test.py` checks that image output grows past its initial buffer and genome slot while logical addresses and an older local remain valid. `code_emission_growth_test.py` checks stable functions, globals, strings, quotes, float payloads, and vm/micro recipes across code segments. `code_address_growth_test.py` checks new addresses, warm calls, bulk rewrites, and denied metadata growth. `seed_growth_test.py` checks stage-0 input over 5 mb and image output beyond its initial reservation. | there is no randomized compiler-image mutation or malformed-image oracle; these are focused growth and rewrite regressions. | +| memory limits | `memory_limits_test.py` checks invalid startup configuration, successful growth, denied growth, and changing limits on native, microcore, and javascript. `stack_growth_test.py` checks explicit reservations, failures, 40,000 operands, and 70,000 return addresses. `layout_config_test.py` checks map arithmetic rejection and generated controls. `compiler_growth_test.py` and `parser_growth_test.py` exercise growing tables and scratch storage. | coverage samples configured limits and selected capacities; it is not an exhaustive boundary matrix for every table or every host configuration. | +| long-running lsp | `fuzz_lsp.js` keeps one native server alive for 32 open/change sequences and 96 open/change/close/reopen cache cycles, requesting semantic tokens and checking diagnostic recovery. `lsp_growth_test.js` opens 32 documents with long uris and documentation, then checks diagnostics and hover text. `protocol_growth_test.js` checks long request ids, extra headers, nested json, and 24 document checks. `analysis_cache_test.js` checks reuse and invalidation after edits, includes, and reopen. | these check protocol/cache correctness, not process memory growth or bounded memory after prolonged use. the 96-cycle fuzz currently targets the native server, not wasm or vm hosts. | + +## variable hovers and value previews + +`hover_features_test.js` checks typed global and fixed-array hovers at uses, field and parameter hovers, inferred locals and function signatures, pointer aliases, closures, enum types, and embedded compiler/runtime/library documentation. it checks the type-first global format at both declarations and uses for a word-string global, a fixed global array, and a typed global. `hover_surface_test.js` checks embedded source, constants and backing arrays, asm descriptions, `include std;`, section comments, and trailing declaration comments. `storage_analysis_test.js` checks size and lifetime metadata, including bounds, ended lifetimes, stack storage, and unknown effects. `hover_values_test.js` checks position-specific local, global, constant, enum, integer, float, and string values, including invalidation after unknown calls, writes, and control flow. `vscode_hover_test.js` also checks a real editor hover for a value before and after an unknown call. + +the coverage is spread across drivers. `hover_values_test.js` starts a fresh server for each hover, so it does not verify several value requests against one open fixture in one server session. the other hover drivers cover multiple requests or documents, but no single test matrix checks every variable form at both declaration and use positions across native, javascript, and wasm hosts. the javascript and wasm hover surfaces are exercised by `tests/test.sh`; the value-preview and feature drivers exist but are not invoked by that script. + +## recursive type inference + +the `inference` group in `toolchain_lsp_contract_test.js` checks type propagation from typed callees through caller parameters and return aliases, including a twelve-function chain declared in reverse order, an uncalled wrapper, and recursion anchored by a typed callee. it also checks that conflicting caller argument types and unknown reassignment do not produce an invented type, that array indices and typed fields do not force a scalar parameter type, and that owned raw-list results preserve `word.u` while capacity and caller parameters infer as `int`. `inference_growth_test.py` checks large function/local sets, late parameters and nested closures, plus stable pointer annotations for array and struct aliases. the learning example covers a reverse-declared chain, typed recursive anchor and owned raw-list returns. + +## const fixed arrays + +the `qualifiers` group in `toolchain_lsp_contract_test.js` checks both qualifier orders, hover shape, element writes, increments, compound writes with typechecking disabled, rebinding, mutable-reference conversion, read-only aliases, and explicit read-only parameters. mutable arrays are accepted by read-only formals. the compiler contract fixtures also check accepted reads, rejected mutation, packed elements and global arrays. the 15 compiler contracts pass on native and micro, and the three qualifier hover cases pass on the language server. standard-library array algorithms on const-qualified arrays have not been verified. + +## public standard-library documentation + +`src/lib/std.zc` documents public operations for output capture, numeric helpers, packed byte strings, host access, arenas, growable byte buffers, and array algorithms, including parameter and return behavior. the array algorithms document their callback roles, result types, traversal order, and reference sharing. `src/tools/embed.sh all --check` verifies that the compiler's embedded source matches the module. `hover_surface_test.js` and `hover_features_test.js` check selected embedded library documentation; they do not assert that every public function has a description. edge-case behavior for the broader standard-library API still needs dedicated acceptance tests. + +## unused-argument hints + +`hints_test.js` checks informational severity and positional `hints on`, `hints small`, and `hints off` behavior for local-variable hints and style hints. `small` retains these hint kinds with compact markers; it does not reduce the set of hints. the `magic` group in `toolchain_lsp_contract_test.js` checks compact severity, suppression after `off`, and magic-number suppression by an inline block comment immediately before a literal; a comment before another operand does not suppress it. the `arguments` group covers unused, used, discard, write-only, and shadowed parameters, plus method, closure, and generic arguments and the positional directives. + +## `rt.s_to_n` definition navigation + +`tests/programs/strings.zc` exercises runtime conversion through `rt.s_to_n`. `nav_test.js` checks definition navigation to a runtime function, standard-library functions and types, and embedded source text. the `navigation` group in `toolchain_lsp_contract_test.js` specifically checks that `rt.s_to_n` opens at its runtime declaration. virtual runtime source checks do not currently cover javascript or wasm hosts. + +## string-literal types + +word-string inference is covered at a global's declaration and use by `hover_features_test.js`, and `toolchain_lsp_contract_test.js` checks a grouped local and inferred word-string return. the same contract driver checks `bstr` through a declaration, parameter, return, and use. `completion_test.js` checks local completion details for both `char*` and `bstr`. runtime behavior has coverage in `tests/programs/strings.zc`, `bytes.zc`, `fuzz_compiler.py`, and `fuzz_wasm.js`. + +the remaining string-type gaps include reassignment and consistent coverage across all supported hosts. the current focused cases cover representative inference and navigation paths, not every context or backend. + +## file namespaces and array receivers + +`toolchain_lsp_contract_test.js` checks that module-qualified use requires an explicit include and that completions expose only imported modules. `include_test.js` checks file-relative and nested includes, `include std;`, and diagnostics for missing or invalid modules; it passes with the current namespace fixture and diagnostic location. `learn.py` verifies a file-scoped namespace with multiple imports. + +`completion_test.js` checks receiver completion for growable arrays and confirms that fixed arrays offer only `len` and `cap`; it also checks the nine standard-library algorithms on fixed arrays and requires `include std;` for them. the learn fixtures exercise array receiver syntax. `array_methods_test.py` covers growable and fixed receivers, packed elements, callbacks, inferred map results, fixed-array filter, searching, predicates, reduction, empty inputs, shallow sharing for references, missing imports, ownership transfers, resize cleanup, and aggregate-value restrictions. all 39 cases pass on native and micro, including pop ownership transfer, cleanup of removed children after shrink, borrowed callbacks, packed strings, and coercion. + +## import quick fixes and module hovers + +the `quickfix` group in `toolchain_lsp_contract_test.js` checks missing runtime, compiler and core imports; qualification of bare exports; preservation of namespaces, comments and crlf; no duplicate import; and no unrelated edits. the editor's `codeactions_client_test.js` checks workspace-edit conversion, rejects stale document versions, and keeps protocol version epochs separate from editor versions. `ext_cache_test.js` exercises the real editor client with the bundled wasm server, including its code-action and completion providers. the `modulehover` group checks include-name hovers for `rt`, `run`, `core`, `lsp`, `std` and `icon`, including comments and newlines in the include list and source-library overview comments after namespace/include headers. `namespacecompletion` checks exported members, signatures and docs for those modules. `navigation` checks the embedded `core.zc`, the distinction between `core.syscall` and `rt.syscall`, and identity-preserving references. + +the formatter indents case labels one level inside their switch by default (`CaseIndent: 1`); `CaseIndent: 0` aligns the labels with the switch. `format_test.js` checks token preservation and idempotence for every repository `.zc` file. + +## command-line help and run-only images + +`cli_help_test.py` checks `--help` and `-h` with stdin left open and invalid memory environment settings for the native and micro vms, language server, bootstrap compiler, image converter, map generator, runner scripts, tool scripts, and javascript utilities. it checks unknown-option rejection and confirms that the language server's `--format`, `--annotate` and `--types` source options remain available. `runtime_image_test.py` checks self-hosted run-only images for enhancement startup, array algorithms, and owned `pop`; the runtime parses enhancement descriptors before initializing the heap. `make run` and `zcc` still compile through stage 0 and therefore accept its source subset, while `src/tools/build.zc` builds full-language run-only images. + +## focused commands + +run from the repository root. native binaries and the language server must already be built; the wasm command needs a built `src/web/zc.wasm`. + +```sh +python3 tests/drivers/fuzz_compiler.py bin/zy bin/zym --js +python3 tests/drivers/fuzz_malformed.py bin/zy bin/zym +node tests/drivers/fuzz_wasm.js src/web/zc.wasm +python3 tests/drivers/image_growth_test.py bin/zy +python3 tests/drivers/code_emission_growth_test.py bin/zy +python3 tests/drivers/code_address_growth_test.py bin/zy +python3 tests/drivers/seed_growth_test.py +python3 tests/drivers/memory_limits_test.py +python3 tests/drivers/stack_growth_test.py bin/zy +python3 tests/drivers/layout_config_test.py +python3 tests/drivers/compiler_growth_test.py bin/zy +python3 tests/drivers/parser_growth_test.py bin/zy +node tests/drivers/fuzz_lsp.js bin/zclsp +node tests/drivers/lsp_growth_test.js bin/zclsp +node tests/drivers/protocol_growth_test.js bin/zclsp +node tests/drivers/core_lsp_test.js bin/zclsp +node tests/drivers/hover_values_test.js bin/zclsp +node tests/drivers/hover_features_test.js bin/zclsp +node tests/drivers/hover_surface_test.js bin/zclsp +CONTRACT_GROUP=inference node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=qualifiers node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +python3 tests/drivers/compiler_contract_test.py bin/zy --group qualifiers +python3 tests/drivers/compiler_contract_test.py bin/zym --group qualifiers +python3 tests/drivers/inference_growth_test.py bin/zclsp +node tests/drivers/storage_analysis_test.js bin/zclsp +node tests/drivers/hints_test.js bin/zclsp +node tests/drivers/include_test.js bin/zclsp +CONTRACT_GROUP=quickfix node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=modulehover node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=namespacecompletion node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=navigation node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +node tests/drivers/codeactions_client_test.js +node tests/drivers/ext_cache_test.js +python3 tests/drivers/array_methods_test.py bin/zy +python3 tests/drivers/array_methods_test.py bin/zy --case map +python3 tests/drivers/array_methods_test.py bin/zy --case filter +python3 tests/drivers/array_methods_test.py bin/zy --case search +python3 tests/drivers/array_methods_test.py bin/zy --case reduce +python3 tests/drivers/cli_help_test.py +python3 tests/drivers/runtime_image_test.py +CONTRACT_GROUP=magic node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=strings node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=arguments node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=modules node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +node tests/drivers/nav_test.js bin/zclsp +tests/runerr.sh bin/zy tests/errors +``` diff --git a/docs/history/maintenance-notes.md b/docs/history/maintenance-notes.md index 2a2bed4..4a01cba 100644 --- a/docs/history/maintenance-notes.md +++ b/docs/history/maintenance-notes.md @@ -2,6 +2,20 @@ this material records earlier builds. current work and verification are tracked in TODO.md. +## current documentation checks (2026-10-14) + +the learning guide now has 33 runnable source examples, including receiver algorithms for arrays, recursive call inference, and const fixed arrays. `python3 tests/drivers/learn.py bin/zy` passed every runnable example on this host and confirmed that every fenced zc block matches its fixture; the x86-64-only raw-syscall example was skipped on this architecture. `node tests/drivers/include_test.js bin/zclsp`, `node tests/drivers/completion_test.js bin/zclsp`, the `magic`, `strings`, `arguments`, and `modules` groups in `toolchain_lsp_contract_test.js`, and `node tests/drivers/hover_features_test.js bin/zclsp` passed their focused checks. the full 39-case `array_methods_test.py` driver passed on both native and micro, including pop ownership transfer, cleanup of removed children after shrink, borrowed callbacks, packed strings, and coercion. no full suite was run for this documentation pass. + +the `inference` group in `toolchain_lsp_contract_test.js` passed nine focused cases: typed callee constraints, a reverse-declared twelve-function chain, recursion anchored by a typed callee, unresolved conflicts and reassignment, and owned raw-list results. `inference_growth_test.py bin/zclsp` passed its three large-inference and alias-annotation checks. the new learning example passed with output `41 7`. regenerated `tests/programs/annotate.expected` from the namespaced source with `bin/zclsp --annotate`; running the source and annotated result produced matching output. + +both qualifier orders and the read-only fixed-array behavior are documented in the new `33-const-fixed` learning example. all 15 qualifier compiler contracts passed on native and micro, the three qualifier hover cases passed on the language server, and the example passed on both vms. mutable fixed and growable arrays remain the verified inputs for standard-library algorithms; const-qualified inputs have not been verified. + +the cli help inventory was verified with `python3 tests/drivers/cli_help_test.py`: help exits with stdin open and invalid memory settings for the native and micro vms, language server, bootstrap and image tools, map generator, shell tools, and javascript utilities; unknown options are rejected, and the language server source options still work. `python3 tests/drivers/runtime_image_test.py` passed the enhancement, array-algorithm and owned-pop run-only image cases. the run-only runtime parses enhancement descriptors before heap initialization. `make run` and `zcc` still use the stage-0 source subset; `src/tools/build.zc` builds full-language run-only images. + +the `quickfix`, `modulehover`, `namespacecompletion` and `navigation` groups in `toolchain_lsp_contract_test.js` passed against `bin/zclsp`. they verify import fixes, module hovers and completions, source-library overview comments, `core.zc` navigation, and distinct `core.syscall`/`rt.syscall` references. `node tests/drivers/codeactions_client_test.js` passed edit conversion and version-epoch checks; `node tests/drivers/ext_cache_test.js` passed its 16 checks with the real editor provider and bundled wasm server. formatter output was checked for a nested switch: case labels indent one level within their switch by default, and `CaseIndent: 0` aligns them with the switch. + +the test counts and open ends below are preserved as a historical stage-48 snapshot. they describe that earlier build, not the current implementation. + ## test results the whole of `test.sh` is 614 checks on a machine with gcc, qemu-user, llvm, node and clang's wasm32 target (572 passed, 0 failed; the baremetal boards need cross compilers and are skipped without them). on the interpreter alone it starts with the fixpoint (236652 words), an up-to-date `genome.h`, and then every example and feature test, including a shift past bit 31 and equality of a value above 2^31. it also covers joins and fixup frees (with a leak count), `switch` and `?:`, the buddy allocator, aliases used while their owner is alive, quarantine (a freed block stays intact for a stale alias, a second free does nothing, and a collection keeps a block a word still points at), `do`/`while` with `enum` and a cast, `include` and the string helpers, syscalls, memdebug leaks, use-after-free (including through an interior pointer), double free, payload poison and interior pinning, several errors in one run, all three stack overflows, growable and fixed arrays, and `tests/o3.zc` (loops, recursion, calls inside loops, `switch`, `&& ||`, structs through pointers, arrays, division by constants, and a function with more variables than registers). it checks that every o3 region in that test lifts, that the o3 bounds check fires, and that all 26 programs in `tests/errors/` (including derived aliases used after their owner dies), all 10 in `tests/typeerrors/` and all 9 in `tests/langerrors/` get rejected with the expected message. the same script then runs that list on the microcore, checks that the microcore's self-compile matches the interpreter's image, and checks that `micro.c` and `port_stub.c` compile with `-ffreestanding`. that is 113 checks with the jit and llvm skipped. it then builds the 339968-word compiler and runs 22 programs on the small microcore and again on `core.c` at that size, builds the 32768-word run-only image for six programs, and checks that both reduced microcores compile freestanding. that is 165 checks with the jit, llvm and bare-metal boots skipped. when the cross gcc and `qemu-system-*` for a target exist, `test.sh` adds 7 boots on that target (run-only hello, fact, wide and gc, then the resident compiler fed hello, fact and wide). both virt boards were present here, which is 14 more. the jit needs `qemu-aarch64`, the linux-user emulator, which `qemu-system-aarch64` is not. the llvm backend needs `llvm-config`. those two were not executed in the environment that cut the word to 64 bits. they were executed later (stage 24), on this host, and the full script then passed 292 checks. stage 25 added three more (the default map matches the checked-in sources, and the flag form of `genmap` reproduces the small and run maps), 295 passed, 0 failed. stage 26 added eight more (echo until eof, the clock and RAM-file line, the exit text and the exit code, on each virt board), 303 passed, 0 failed. stage 27 added twelve more: the float line on the interpreter, the microcore, the arm64 jit and the llvm backend, the same line on the small microcore and on `core.c` at that size, the run-only image, both virt boards running the program and compiling it, and one llvm run with `ZYHOT=1` so every float block is compiled. 315 passed, 0 failed. stage 28 added twelve more, the same shape of check for byte loads and stores, including a one-byte patch of a live literal. 327 passed, 0 failed. stage 29 added twenty-three more for packed byte strings: the string line, the example, and the hosted byte-file check on the interpreter, the microcore, the arm64 jit and the llvm backend; the string line and the example on the small microcore and on `core.c` at that size; the run-only image of both; both virt boards running the program and compiling it; and one llvm run with `ZYHOT=1`. 350 passed, 0 failed. stage 30 added eighteen more for shared cells: the sharing line and the memdebug line on the interpreter, the microcore, the arm64 jit and the llvm backend; both of those on the small microcore and on `core.c` at that size; the run-only sharing line; both virt boards running it and compiling it; and one llvm run with `ZYHOT=1`. 368 passed, 0 failed. stage 31 added one more: a soft-float interpreter matches the float line. 369 passed, 0 failed. stage 32 added twenty-two more: negative zero is false, and a float stays a float through calls, globals and fields, on the interpreter, the microcore, the arm64 jit and the llvm backend; both of those on the small microcore and on `core.c` at that size; the run-only image of the negative-zero program; both virt boards running that program and compiling it and the float-type program; a soft-float check of negative zero; and llvm runs with `ZYHOT=1` for both. 391 passed, 0 failed. stage 33 added ten more for o3 floats: the lifted kernel matches the plain one, a constant expression folds, a NaN minus itself stays a NaN, and negative zero is false inside the region, on the interpreter, the microcore, the arm64 jit and the llvm backend; the same line on the small microcore and on `core.c` at that size; both virt boards compiling it; a soft-float interpreter; and one llvm run with `ZYHOT=1`. 401 passed, 0 failed. stage 34 added thirteen more for the float text: shortest, fixed and scientific forms, a 17-digit tie, and the full finite range, on the interpreter, the microcore, the arm64 jit and the llvm backend; the same line on the small microcore and on `core.c` at that size; the run-only image; both virt boards running it and compiling it; a soft-float interpreter; and one llvm run with `ZYHOT=1`. 414 passed, 0 failed. the jit binary is `aarch64-linux-gnu-gcc -O2 -w -DJTYPE=int -static core.c io_linux.c` (gcc 13.3.0, static, against the aarch64 sysroot) run by `qemu-aarch64` 8.2.2. the llvm binary is host gcc 13.3.0 with `-DZYLLVM` and LLVM 18.1.3 (`llvm-config --cflags --ldflags --libs`). both ran the same program list as the interpreter and self-compiled to the same image. `ZYTRACE=1` on `tests/o3.zc` printed 8 `bytes of arm64` lines and 8 `compiled by llvm` lines. `ZYHOT=1` on the llvm binary made `tests/types.zc` match the interpreter byte for byte. the first jit run failed one check: `tests/enhance.zc` printed `100 0 343 3,2` and then halted. opcode 21 calls a C function that returns a 32-bit `int`. a write to `w0` zeroes the high half of `x0`, so the failure code `-4` (the `noisy` enhancement, whose body does i/o and does not lift) was stored as a large positive cell. the compiler treated the install as a success, emitted the unwired opcode, and the jit halted. the translator now does `sxtw x0, w0` before storing that cell. the Darwin jit (`MAP_JIT` and the write-protect around each flush) was not run. this host is x86_64 Linux, and qemu-user is the arm64 cpu the jit actually executed on. the 64-bit image is not binary-compatible with a 32-bit genome. @@ -28,3 +42,5 @@ fixed in stage 13 (details in `history/13-gap-fixes`): branch and loop joins, fr --- + +`python3 tests/drivers/learn.py bin/zym` is not a full learning-guide pass: example 23 uses native C ABI `extern` calls, which require native FFI support and are unavailable in the microcore host. example 33 passes on both native and micro; the micro-only guide run must skip the C ABI example. |