From 1c53304d03414890af0da7432d1aea87e754a14a Mon Sep 17 00:00:00 2001 From: Moon Flower Date: Wed, 14 Oct 2026 08:29:46 +0330 Subject: feat: grow memory limits, add icon/actor libraries, overhaul LSP diagnostics Compiler and VM - make every memory map limit dynamic and adjustable at both build time and runtime, including compiler tables, stacks, and every execution region - reserve and grow tables, operand and return stacks, frame segments, and image-building output without relocating live code or globals - rename i/o and string helpers consistently (puts -> put_s, putn -> put_n, scat -> cat_s) across compiler, runtime, examples, tests, and docs - recast and consume values assigned to globals from inside functions - allow fixed and const together, add typed closure returns (fn:t(...)), multi-include syntax (include rt, run, core;), and recurse type inference - never infer string literals as word, anywhere, as argument, variable, or return type - draft docs/BORROW.md: a borrow checker less intrusive than Rust's but more intelligent LSP and editor integration - require explicit includes and give every toolchain area its own namespace (rt, lsp, run, core) - hover for built-in types, inferred types, values known at hover time, array and pointer lifetime/size, library names in include statements, and inferred lifetimes with explanations - honor /* */ comments as documentation for variables and arguments, and stop misattributing section headers or trailing comments as variable hints - stream diagnostics and cancel stale requests on change - jump to definition and references, including cross-file - quick fixes with automatic library import, and namespace completion Hints and diagnostics - render hints as blue squigglies by default, with hints on/small/off per file - suppress magic-number hints via a preceding /* */ comment, and say so - warn on unused arguments; allow multiple _ discards in parameters, pattern matching, and tagged enums, warning if a _ is actually used - rewrite hint and comment prose in professional compiler-doc register Libraries - add icon.zc: run-time self-modification, find/replace syntax, type and block edits, and step-through inspection of locals and globals - add actor.zc: async actors with Actor, implicit this, spawn/recv/send, injected into the compiler machinery at include time rather than built in Tooling and docs - reorganize sources, tests, and designs into subdirectories - unify windows entry points in tinyvm.bat and add a matching tinyvm.sh - add --help across the toolchain - polish the learning guide, stdlib docs, and design notes Tests - add accept, reject, and regression coverage across the compiler, LSP, - runtime, icon, actors, threads, formatting, and the toolchain contracts, plus fuzzers and property tests --- docs/design/diagnostic-style.md | 3 + docs/design/memory.md | 2 + docs/design/storage-analysis.md | 4 +- docs/design/test-coverage.md | 121 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 128 insertions(+), 2 deletions(-) create mode 100644 docs/design/diagnostic-style.md create mode 100644 docs/design/test-coverage.md (limited to 'docs/design') diff --git a/docs/design/diagnostic-style.md b/docs/design/diagnostic-style.md new file mode 100644 index 0000000..7ac7da6 --- /dev/null +++ b/docs/design/diagnostic-style.md @@ -0,0 +1,3 @@ +# diagnostic wording + +compiler and language-server diagnostics should identify the relevant source context and state the problem in concise, actionable terms. use lowercase prose, avoid contractions, and separate any suggested fix from the diagnostic message. these conventions follow the [clang diagnostic wording guidance](https://clang.llvm.org/docs/InternalsManual.html#diagnostic-wording). diff --git a/docs/design/memory.md b/docs/design/memory.md index bcbdf35..c282df5 100644 --- a/docs/design/memory.md +++ b/docs/design/memory.md @@ -68,6 +68,8 @@ document slots, uri text, hover text, declaration types, parsed type spans, reso inference function records retain stable addresses while their index grows. inference edits, replacement text, and annotation verification buffers also grow; later declarations are not dropped at the original function-count or edit-count limits. protocol method names, request identifiers, json keys, nesting paths, and header lines also grow; request identifiers are echoed in full. +completed hover, navigation, token and completion replies have a bounded 64-entry cache. keys include the document generation, request position and include epoch. source updates, included-file updates, saves and close/reopen invalidate dependent replies; cached json values do not retain mutable compiler-table addresses. `zc/cacheStats` reports cache hits, misses and compiler checks for regression tests. + ## freestanding backing the stub and board ports reserve `MICRO_PORT_WORDS` cells at build time; it defaults to the initial image and heap size. the vm's active size and growth limit still change at runtime, within that physical backing. set the macro when linking a board with more reserved ram. a rejected physical allocation preserves the current memory. hosted maps keep their heap above the fixed regions; the historical `fixed` preset retains its bounded layout for compatibility. diff --git a/docs/design/storage-analysis.md b/docs/design/storage-analysis.md index 59fbcde..cff713a 100644 --- a/docs/design/storage-analysis.md +++ b/docs/design/storage-analysis.md @@ -18,9 +18,9 @@ unchecked fn example() { ## what stays unknown -this is basic analysis of local facts, not a proof of the whole program. branches, loops, short-circuit operators, raw assembly, and nested compilation discard facts instead of merging uncertain paths. unknown callees invalidate facts about allocations passed to them. array mutations invalidate recorded lengths. pointer arithmetic, explicit casts, pointer-valued fields, stored aliases, and function return contracts are not inferred by this pass. +this is analysis of compiler-recorded facts, not a proof of the whole program. branch and loop snapshots stay conservative where paths disagree. typed pointer arithmetic, casts, fields, element ranges, and checked function parameter effects carry provenance when the compiler can establish it. known array mutations advance a generation and invalidate older element borrows. an unknown call or opaque operation drops facts it may change; an inferred checked function summary can preserve facts for a read-only borrower. -an unknown fact does not produce a definite warning. runtime array checks still run. the pass may miss an error across one of these boundaries; silence is not a safety guarantee. the broader analysis in [BORROW.md](../BORROW.md) remains a design plan. +unknown facts do not produce definite bounds or lifetime findings. runtime array checks still run. unchecked code, raw assembly, integer-to-pointer casts, untyped word storage, and unknown native or indirect calls remain boundaries. the ownership and temporary-slot rules, including those limits, are described in [BORROW.md](../BORROW.md). ## editor previews diff --git a/docs/design/test-coverage.md b/docs/design/test-coverage.md new file mode 100644 index 0000000..fae9356 --- /dev/null +++ b/docs/design/test-coverage.md @@ -0,0 +1,121 @@ +# test coverage inventory + +this inventory covers runtime ownership boundaries, fuzzing and growth, variable hovers, unused-argument hints, the `rt.s_to_n` definition, string-literal types, and public standard-library documentation. it records existing tests and focused runs made during the documentation pass; it does not claim that the full suite ran. + +## runtime `unchecked` boundaries + +`src/runtime/rt.zc` still marks these functions `unchecked`: `rt.copy_w`, `rt.fill_w`, `rt.free`, `rt.recycle`, `rt.recycle_leaf`, `rt.cons`, `rt.triple`, `rt.dnew`, `rt.dgrow`, `rt.dpush`, `rt.reserve_mark`, `rt.gc`, and `rt.readfile`. + +existing runtime acceptance coverage exercises portions of those paths: `tests/programs/allocfast.zc`, `buddy.zc`, `gc.zc`, `quarantine.zc`, `recycle.zc`, `roots.zc`, `share.zc`, `sys.zc`, and `strings.zc` cover allocation, collection, ownership cells, file access, and string conversion. `tests/drivers/compiler_growth_test.py` stresses mark work and collection after table growth. `tests/errors/unchecked_touch.zc` rejects access to a checked owned value from an unchecked block. `tests/errors/temporary_unknown.zc` and `storage_analysis_test.js` exercise conservative behavior for unknown or unchecked effects. + +this is indirect coverage, not proof that each runtime function is now checked. `tests/drivers/core_lsp_test.js` checks maintained source files without severity-1 diagnostics, but an explicit `unchecked` function can pass that check while bypassing ownership analysis. there is no test that enumerates the runtime annotations, nor a per-function checked-call acceptance and rejection matrix for these boundaries. in particular, removal still needs regressions for the ownership and drop contracts of raw word copying, allocator fast paths, shared cells, growable arrays, collector work, and file results. + +## fuzzing, image growth, memory limits, and lsp longevity + +| area | existing tests and what they assert | remaining gap | +| --- | --- | --- | +| compiler, runtime, and stdlib fuzz | `fuzz_compiler.py` generates 128 arithmetic expressions, 300 scoped allocations plus collection, 16 runtime compiles, source-immutability checks, and 32 randomized `Text`/`Bytes` cases; it compares native, microcore, and optionally javascript output. `fuzz_malformed.py` runs 256 random token streams and 64 byte mutations on native and microcore, checking for crashes, timeouts, and host disagreement. `fuzz_wasm.js` covers 128 expressions, 300 allocations plus collection, and 32 `Text`/`Bytes` cases. | malformed inputs have no independent accept/reject oracle; generated ownership/analyzer properties are not fuzzed. the wasm fuzz does not include runtime compilation or source-immutability checks. | +| image and code growth | `image_growth_test.py` checks that image output grows past its initial buffer and genome slot while logical addresses and an older local remain valid. `code_emission_growth_test.py` checks stable functions, globals, strings, quotes, float payloads, and vm/micro recipes across code segments. `code_address_growth_test.py` checks new addresses, warm calls, bulk rewrites, and denied metadata growth. `seed_growth_test.py` checks stage-0 input over 5 mb and image output beyond its initial reservation. | there is no randomized compiler-image mutation or malformed-image oracle; these are focused growth and rewrite regressions. | +| memory limits | `memory_limits_test.py` checks invalid startup configuration, successful growth, denied growth, and changing limits on native, microcore, and javascript. `stack_growth_test.py` checks explicit reservations, failures, 40,000 operands, and 70,000 return addresses. `layout_config_test.py` checks map arithmetic rejection and generated controls. `compiler_growth_test.py` and `parser_growth_test.py` exercise growing tables and scratch storage. | coverage samples configured limits and selected capacities; it is not an exhaustive boundary matrix for every table or every host configuration. | +| long-running lsp | `fuzz_lsp.js` keeps one native server alive for 32 open/change sequences and 96 open/change/close/reopen cache cycles, requesting semantic tokens and checking diagnostic recovery. `lsp_growth_test.js` opens 32 documents with long uris and documentation, then checks diagnostics and hover text. `protocol_growth_test.js` checks long request ids, extra headers, nested json, and 24 document checks. `analysis_cache_test.js` checks reuse and invalidation after edits, includes, and reopen. | these check protocol/cache correctness, not process memory growth or bounded memory after prolonged use. the 96-cycle fuzz currently targets the native server, not wasm or vm hosts. | + +## variable hovers and value previews + +`hover_features_test.js` checks typed global and fixed-array hovers at uses, field and parameter hovers, inferred locals and function signatures, pointer aliases, closures, enum types, and embedded compiler/runtime/library documentation. it checks the type-first global format at both declarations and uses for a word-string global, a fixed global array, and a typed global. `hover_surface_test.js` checks embedded source, constants and backing arrays, asm descriptions, `include std;`, section comments, and trailing declaration comments. `storage_analysis_test.js` checks size and lifetime metadata, including bounds, ended lifetimes, stack storage, and unknown effects. `hover_values_test.js` checks position-specific local, global, constant, enum, integer, float, and string values, including invalidation after unknown calls, writes, and control flow. `vscode_hover_test.js` also checks a real editor hover for a value before and after an unknown call. + +the coverage is spread across drivers. `hover_values_test.js` starts a fresh server for each hover, so it does not verify several value requests against one open fixture in one server session. the other hover drivers cover multiple requests or documents, but no single test matrix checks every variable form at both declaration and use positions across native, javascript, and wasm hosts. the javascript and wasm hover surfaces are exercised by `tests/test.sh`; the value-preview and feature drivers exist but are not invoked by that script. + +## recursive type inference + +the `inference` group in `toolchain_lsp_contract_test.js` checks type propagation from typed callees through caller parameters and return aliases, including a twelve-function chain declared in reverse order, an uncalled wrapper, and recursion anchored by a typed callee. it also checks that conflicting caller argument types and unknown reassignment do not produce an invented type, that array indices and typed fields do not force a scalar parameter type, and that owned raw-list results preserve `word.u` while capacity and caller parameters infer as `int`. `inference_growth_test.py` checks large function/local sets, late parameters and nested closures, plus stable pointer annotations for array and struct aliases. the learning example covers a reverse-declared chain, typed recursive anchor and owned raw-list returns. + +## const fixed arrays + +the `qualifiers` group in `toolchain_lsp_contract_test.js` checks both qualifier orders, hover shape, element writes, increments, compound writes with typechecking disabled, rebinding, mutable-reference conversion, read-only aliases, and explicit read-only parameters. mutable arrays are accepted by read-only formals. the compiler contract fixtures also check accepted reads, rejected mutation, packed elements and global arrays. the 15 compiler contracts pass on native and micro, and the three qualifier hover cases pass on the language server. standard-library array algorithms on const-qualified arrays have not been verified. + +## public standard-library documentation + +`src/lib/std.zc` documents public operations for output capture, numeric helpers, packed byte strings, host access, arenas, growable byte buffers, and array algorithms, including parameter and return behavior. the array algorithms document their callback roles, result types, traversal order, and reference sharing. `src/tools/embed.sh all --check` verifies that the compiler's embedded source matches the module. `hover_surface_test.js` and `hover_features_test.js` check selected embedded library documentation; they do not assert that every public function has a description. edge-case behavior for the broader standard-library API still needs dedicated acceptance tests. + +## unused-argument hints + +`hints_test.js` checks informational severity and positional `hints on`, `hints small`, and `hints off` behavior for local-variable hints and style hints. `small` retains these hint kinds with compact markers; it does not reduce the set of hints. the `magic` group in `toolchain_lsp_contract_test.js` checks compact severity, suppression after `off`, and magic-number suppression by an inline block comment immediately before a literal; a comment before another operand does not suppress it. the `arguments` group covers unused, used, discard, write-only, and shadowed parameters, plus method, closure, and generic arguments and the positional directives. + +## `rt.s_to_n` definition navigation + +`tests/programs/strings.zc` exercises runtime conversion through `rt.s_to_n`. `nav_test.js` checks definition navigation to a runtime function, standard-library functions and types, and embedded source text. the `navigation` group in `toolchain_lsp_contract_test.js` specifically checks that `rt.s_to_n` opens at its runtime declaration. virtual runtime source checks do not currently cover javascript or wasm hosts. + +## string-literal types + +word-string inference is covered at a global's declaration and use by `hover_features_test.js`, and `toolchain_lsp_contract_test.js` checks a grouped local and inferred word-string return. the same contract driver checks `bstr` through a declaration, parameter, return, and use. `completion_test.js` checks local completion details for both `char*` and `bstr`. runtime behavior has coverage in `tests/programs/strings.zc`, `bytes.zc`, `fuzz_compiler.py`, and `fuzz_wasm.js`. + +the remaining string-type gaps include reassignment and consistent coverage across all supported hosts. the current focused cases cover representative inference and navigation paths, not every context or backend. + +## file namespaces and array receivers + +`toolchain_lsp_contract_test.js` checks that module-qualified use requires an explicit include and that completions expose only imported modules. `include_test.js` checks file-relative and nested includes, `include std;`, and diagnostics for missing or invalid modules; it passes with the current namespace fixture and diagnostic location. `learn.py` verifies a file-scoped namespace with multiple imports. + +`completion_test.js` checks receiver completion for growable arrays and confirms that fixed arrays offer only `len` and `cap`; it also checks the nine standard-library algorithms on fixed arrays and requires `include std;` for them. the learn fixtures exercise array receiver syntax. `array_methods_test.py` covers growable and fixed receivers, packed elements, callbacks, inferred map results, fixed-array filter, searching, predicates, reduction, empty inputs, shallow sharing for references, missing imports, ownership transfers, resize cleanup, and aggregate-value restrictions. all 39 cases pass on native and micro, including pop ownership transfer, cleanup of removed children after shrink, borrowed callbacks, packed strings, and coercion. + +## import quick fixes and module hovers + +the `quickfix` group in `toolchain_lsp_contract_test.js` checks missing runtime, compiler and core imports; qualification of bare exports; preservation of namespaces, comments and crlf; no duplicate import; and no unrelated edits. the editor's `codeactions_client_test.js` checks workspace-edit conversion, rejects stale document versions, and keeps protocol version epochs separate from editor versions. `ext_cache_test.js` exercises the real editor client with the bundled wasm server, including its code-action and completion providers. the `modulehover` group checks include-name hovers for `rt`, `run`, `core`, `lsp`, `std` and `icon`, including comments and newlines in the include list and source-library overview comments after namespace/include headers. `namespacecompletion` checks exported members, signatures and docs for those modules. `navigation` checks the embedded `core.zc`, the distinction between `core.syscall` and `rt.syscall`, and identity-preserving references. + +the formatter indents case labels one level inside their switch by default (`CaseIndent: 1`); `CaseIndent: 0` aligns the labels with the switch. `format_test.js` checks token preservation and idempotence for every repository `.zc` file. + +## command-line help and run-only images + +`cli_help_test.py` checks `--help` and `-h` with stdin left open and invalid memory environment settings for the native and micro vms, language server, bootstrap compiler, image converter, map generator, runner scripts, tool scripts, and javascript utilities. it checks unknown-option rejection and confirms that the language server's `--format`, `--annotate` and `--types` source options remain available. `runtime_image_test.py` checks self-hosted run-only images for enhancement startup, array algorithms, and owned `pop`; the runtime parses enhancement descriptors before initializing the heap. `make run` and `zcc` still compile through stage 0 and therefore accept its source subset, while `src/tools/build.zc` builds full-language run-only images. + +## focused commands + +run from the repository root. native binaries and the language server must already be built; the wasm command needs a built `src/web/zc.wasm`. + +```sh +python3 tests/drivers/fuzz_compiler.py bin/zy bin/zym --js +python3 tests/drivers/fuzz_malformed.py bin/zy bin/zym +node tests/drivers/fuzz_wasm.js src/web/zc.wasm +python3 tests/drivers/image_growth_test.py bin/zy +python3 tests/drivers/code_emission_growth_test.py bin/zy +python3 tests/drivers/code_address_growth_test.py bin/zy +python3 tests/drivers/seed_growth_test.py +python3 tests/drivers/memory_limits_test.py +python3 tests/drivers/stack_growth_test.py bin/zy +python3 tests/drivers/layout_config_test.py +python3 tests/drivers/compiler_growth_test.py bin/zy +python3 tests/drivers/parser_growth_test.py bin/zy +node tests/drivers/fuzz_lsp.js bin/zclsp +node tests/drivers/lsp_growth_test.js bin/zclsp +node tests/drivers/protocol_growth_test.js bin/zclsp +node tests/drivers/core_lsp_test.js bin/zclsp +node tests/drivers/hover_values_test.js bin/zclsp +node tests/drivers/hover_features_test.js bin/zclsp +node tests/drivers/hover_surface_test.js bin/zclsp +CONTRACT_GROUP=inference node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=qualifiers node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +python3 tests/drivers/compiler_contract_test.py bin/zy --group qualifiers +python3 tests/drivers/compiler_contract_test.py bin/zym --group qualifiers +python3 tests/drivers/inference_growth_test.py bin/zclsp +node tests/drivers/storage_analysis_test.js bin/zclsp +node tests/drivers/hints_test.js bin/zclsp +node tests/drivers/include_test.js bin/zclsp +CONTRACT_GROUP=quickfix node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=modulehover node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=namespacecompletion node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=navigation node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +node tests/drivers/codeactions_client_test.js +node tests/drivers/ext_cache_test.js +python3 tests/drivers/array_methods_test.py bin/zy +python3 tests/drivers/array_methods_test.py bin/zy --case map +python3 tests/drivers/array_methods_test.py bin/zy --case filter +python3 tests/drivers/array_methods_test.py bin/zy --case search +python3 tests/drivers/array_methods_test.py bin/zy --case reduce +python3 tests/drivers/cli_help_test.py +python3 tests/drivers/runtime_image_test.py +CONTRACT_GROUP=magic node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=strings node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=arguments node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +CONTRACT_GROUP=modules node tests/drivers/toolchain_lsp_contract_test.js bin/zclsp +node tests/drivers/nav_test.js bin/zclsp +tests/runerr.sh bin/zy tests/errors +``` -- cgit v1.2.3