1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
|
# memory configuration
## one image and one layout
the compiler, runtime, and host must agree on their addresses. `genmap` writes a matching `memmap.h`, runtime source, and compiler source; `src/tools/automap.sh` builds an image against those files. rebuild the host against the generated headers. changing a header alone does not relocate an existing genome. compact compiler and run-only maps also put their heap above the fixed regions, so compiler storage can grow after startup.
```sh
bin/genmap --preset default --check
bash src/tools/automap.sh small build/small
bin/genmap --preset default --code 65536 --symbols 256 --locals 64 --fwd 512 --pool 8192 --hash 1024 -o build/custom
```
run `bin/genmap --help` for the table, heap, genome, and stack options. sizes are in vm words unless an option says otherwise; one word is eight bytes. `--snap-cap` selects the initial snapshot reservation and `--quote-cap` selects an initial token count, with three words per token. both regions grow after startup.
## layout inventory
the generated addresses reserve initial storage. native code-address tables start at `PW` and grow together when code needs new addresses; the microcore and javascript execute within their mapped memory. every host must use the same layout as its image.
| region | build control | runtime behavior |
| --- | --- | --- |
| initial emitted program, through `CODE_END` | `--code` | links stable heap segments when full |
| type arena below `ZFP` | `--types` | stable type chunks and a growing index |
| compiler names and columns | `--hash`, `--pool`, `--symbols`, `--locals`, `--fwd` | grow independently |
| compiler scratch regions | `--snap`, `--quote`, `--src`, `--seq`, `--ctx`, `--loop` | grow independently |
| enum members, tagged-enum index, struct methods, generated source | `--enum-members`, `--tagged-enums`, `--methods`, `--generated-text` | grow independently |
| operator columns | `--binary`, `--unary`, `--assignments`, `--keywords` | grow independently |
| assembly labels and patches at `R_ASM` | `--asm-labels`, `--asm-refs` | columns grow |
| exit storage at `R_EXIT` | `--exit` | grows |
| genome reservation, from `GORG` to `FBOT` | `--genome` | existing image addresses stay fixed |
| frame region, from `FBOT` to `FTOP` | `--frame` | adds stable heap segments |
| return region, from `RS0` to `MARK0` | `--return` | copies live return addresses to larger storage |
| collector work at `MARK0` | `--mark` | grows during collection |
| data region, from `DSLO` to `DSHI` | `--data` | copies live operands to larger storage |
| initial heap | `--heap`, `--heap-lo` on compact maps | grows when the host supports it |
| total host memory | `--total`, host startup options | growth limit can change during execution |
`--words` selects the fixed-map size; unused words extend the initial data region. `--total` selects the host reservation for a growing map. inside `genmap`, the fixed-map size is called `MW` and the host reservation `TW`; generated headers name them `PW` and `MW`. `HEAP_HI` is the initial heap end, not its eventual limit. `SYSGATE` stays at `ZFP + 2`, and the linked image supplies `GORG`.
parameter tags, native calling conventions, packed token positions, and opcode formats also have limits. these describe the language or machine interface; they are separate from the storage capacities above. their parser buffers grow independently; for example, an enum index can grow while one tagged enum keeps its 63-member exhaustiveness mask. function parameter tags and generic arity retain their current encoding. packed token positions retain a 24-bit line, 16-bit token length, and 21-bit byte offset; this is an encoding constraint, not a growing buffer.
## compiler tables during execution
the generated layout supplies initial storage. the following tables grow when they fill:
| storage | growth behavior | explicit reservation |
| --- | --- | --- |
| symbols and their types | copy columns; symbol indices stay valid | `reserve_symbols(n)` |
| locals and analyzer facts | copy every local column | `reserve_locals(n)` |
| forward references | copy patch columns; links retain indices | `reserve_fixes(n)` |
| intern hash | rehash pointers; interned spellings stay at their old addresses | `reserve_names(n)` |
| string pool | add larger chunks; old handles and symbol-chain heads stay valid | `poolchunk` sets the next chunk's minimum size |
| type records and type index | add stable chunks and grow the index | automatic |
| loop-range facts, joined paths, queued o3 regions | copy records before appending; build controls are `--facts`, `--paths`, `--regions` | automatic |
| analyzer snapshots and exits | copy storage; saved offsets stay valid | `reserve_R_SNAP(n)`, `reserve_R_EXIT(n)` |
| source, loop, and compiler-context stacks | copy complete frames | `reserve_R_SRC(n)`, `reserve_R_LOOP(n)`, `reserve_R_CTX(n)` |
| quote tokens and operator sequences | copy storage before appending | `reserve_R_QBUF(n)`, `reserve_R_SEQ(n)` |
| emitted code | link stable heap segments; code addresses remain valid | `reserve_code(n)`, `code_chunk` |
| operator columns | copy columns before adding an operator | `reserve_binary(n)`, `reserve_unary(n)`, `reserve_assignment(n)`, `reserve_operators(n)` |
| execution frames | add segments without moving older local addresses | `reserve_frames(n)` |
reservation functions take minimum capacities and round upward. they can be called by running programs or immediate functions. scratch reservations count words; source frames use 32 words, loop frames use eight, and compiler contexts use 34. local and symbol capacities count entries. the intern hash stays a power of two and grows before it is half full.
new chunks and columns are collector roots. garbage collection can reclaim replaced heap columns, while chunk chains retain interned text and type records for the current compiler state. fixed map storage remains reserved; growing a table does not reuse another map region.
## language server
document slots, uri text, hover text, declaration types, parsed type spans, resolved names, comment ranges, and diagnostics grow independently. hint deduplication uses a growing set of source positions. local name lookup uses growing name heads and shadow links; leaving a scope restores its previous bindings. a large source file must not lose its later diagnostics because earlier hints filled a buffer. the editor bundle must be rebuilt when the server changes; packaging always rebuilds its image.
inference function records retain stable addresses while their index grows. inference edits, replacement text, and annotation verification buffers also grow; later declarations are not dropped at the original function-count or edit-count limits. protocol method names, request identifiers, json keys, nesting paths, and header lines also grow; request identifiers are echoed in full.
## freestanding backing
the stub and board ports reserve `MICRO_PORT_WORDS` cells at build time; it defaults to the initial image and heap size. the vm's active size and growth limit still change at runtime, within that physical backing. set the macro when linking a board with more reserved ram. a rejected physical allocation preserves the current memory. hosted maps keep their heap above the fixed regions; the historical `fixed` preset retains its bounded layout for compatibility.
## host memory and image storage
the default native map places a growing heap above the fixed regions. hosted runners accept `TINYVM_MEMORY_WORDS` for initial memory and `TINYVM_MAX_MEMORY_WORDS` for its growth limit. javascript also accepts `memoryWords` and `maxMemoryWords` when constructing a vm. invalid configurations are rejected before execution.
```sh
TINYVM_MAX_MEMORY_WORDS=16777216 bin/zy < examples/hello.zc
TINYVM_MEMORY_WORDS=4194304 bin/zym < examples/hello.zc
```
the native core reserves a stable backing allocation up to its startup maximum; increasing the available size keeps addresses and native code pointers valid. the microcore and javascript vm replace backing allocations when they grow, preserving existing contents. native bounds use the reservation; portable bounds use the current allocation. memory syscall `9` with argument `4` requests growth, and argument `5` changes the growth limit during execution. a limit cannot fall below the current size. the native limit cannot exceed its startup reservation.
data and return stacks grow automatically before their guard space is exhausted. `reserve_data(n)` and `reserve_returns(n)` request minimum capacities during execution; failure returns `-1` and preserves the old stack. live operands and return addresses are copied before old heap storage is returned to its free list. locals remain in frame storage, so pointers into locals keep their addresses. raw indices returned by `dsp()` describe the current operand stack and must be read again after a reservation.
the host takes stack blocks directly from the runtime's buddy free lists. it does not call vm code or collect while servicing an overflow. when it needs another heap extent, the host's configured memory limit still applies. newly allocated stack blocks are excluded from user-object counts and are protected if collection is already running. operand-stack roots use the current stack's live range.
frame segments grow on the heap, preserve pointers into older frames, and return empty segments to the allocator. collection scans only the live ranges of each segment. `frame_chunk` controls the minimum size of the next segment; `reserve_frames(n)` raises it. the growth helper uses a guard region to allocate without recursively growing its own frame. program emission links new segments through `reserve_code(n)`, with `code_chunk` controlling their minimum size. earlier functions and globals retain their addresses. strings, quotes, global arrays, and enhancement recipes reserve contiguous storage.
optimizer regions that cross a segment boundary run as ordinary bytecode. image output grows independently through `build_image_grow(org, initial)`; its length is returned, and `image_words` and `image_size` expose its output. `build_image(org, dst)` retains the explicit-destination interface. a finished artifact still needs a host layout whose genome reservation fits it; `src/tools/automap.sh` sizes that reservation before linking.
assembly columns, enum members, tagged-enum indices, struct methods, and generated source grow. native program-address metadata grows through memory syscall `9`, argument `9`; a denied request preserves the previous tables. annotation records and edit buffers also grow, within the host’s configured memory limit. generated initial addresses remain part of the image format; language and machine-interface limits do not change when storage grows.
mark work grows through `reserve_mark(n)`, including during a collection. generic templates and instances grow through `reserve_generics(n)` and `reserve_instances(n)`. closure frames and capture rows grow through `reserve_closures(n)` and `reserve_captures(n)`. lexer text, documentation text, and stored signatures also grow. documentation objects keep stable addresses, and compiler storage is excluded from `live_objects()`.
## verification
`tests/drivers/compiler_growth_test.py` checks locals, distinct array types, long names, generic records, captures, and collector work, then reserves larger tables and scratch stacks and runs collection. `tests/drivers/frame_growth_test.py` checks deep recursion, oversized frames, and collection with a pointer into an older segment. `tests/drivers/inference_growth_test.py` checks 600 functions and 5000 local declarations. `tests/drivers/core_lsp_test.js` opens every maintained core zc source through the editor protocol and checks diagnostics and highlighting. `tests/drivers/memory_limits_test.py` checks startup validation and runtime growth on the native, microcore, and javascript hosts. a bootstrap fixpoint alone does not verify runtime growth. collection is conservative: code literals can resemble heap addresses, particularly on compact maps, so reclamation counts can differ between layouts. checks must verify collection progress and retained roots without assuming identical counts.
## bootstrap storage
stage 0 reads source and grows its temporary image buffer on the host. `MAXW`, when supplied at build time, selects an initial image reservation; it is not a final size limit. the resident compiler builds images with `build_image_grow(org, initial)` and returns their current buffer through `image_words`. image output is temporary storage, so it can move while the logical addresses recorded in the image stay unchanged. a matching generated layout must still fit the finished artifact before it is loaded.
vm literals carry 63 signed bits in a single odd word. the compiler emits a short sequence for other 64-bit values; float payloads use their own full word. `setlit` patches one encoded word and requires a value that fits that single-word format.
|
