path: root/ONBOARD.md

# onboarding

a reading and verification order for getting from a fresh checkout to working
knowledge of tinyvm and zc. each step names what to read, what to run, and what
you should be able to answer afterwards. the steps build on each other, so follow
them in order; the whole pass takes a few hours, most of it reading.

the order was recorded while learning the code from scratch, starting with the
markdown files (the fastest way in) and only then the sources.

## 0. the shape of the repository

read `README.md` from the top through "build and run", then skim its section
headings. list the tree:

```sh
ls; ls src src/*; ls tests docs docs/design
```

- the root holds entry points only: `Makefile`, `tinyvm.sh`, `tinyvm.bat`, the
  markdown files.
- `src/` holds every source: `core/` (the c vm), `micro/` (the portable vm),
  `runtime/` and `compiler/` (zc sources that become the genome), `lib/` (std,
  icon, actor), `lsp/` (the language server, in zc), `tools/` (bootstrap and build
  tools), `js/`, `web/`, `baremetal/`.
- `build/` holds everything generated and is not in git. nothing outside it is
  ever written by a build or a test.

you should be able to say: what the genome is (the compiler and runtime, compiled
to vm words, embedded in every core), and which files make it
(`src/runtime/rt.zc`, `src/core/core.zc`, and the compiler: the files of
`src/compiler/`, listed in order in `src/compiler/parts`).

## 1. build it from nothing

the cores embed their genome with c23 `#embed`, so you need clang 19 or gcc 15
or later; `sh src/tools/findcc.sh` prints the compiler `make` will use.

```sh
make clean
make
build/bin/zy < examples/hello.zc
```

watch the output of `make`: the default memory map is generated
(`build/core/map/`), zc0 compiles the genome sources (stage 0), a core built
around that image compiles the compiler again (stage 1) and again (stage 2), the
two self-compiled images are compared, and the first becomes
`build/core/genome.bin`. then `zy`, `zym` and `zclsp` are built around it.

you should be able to say: why the bootstrap needs three stages, what `mkgenome`
protects against (a compiler's error report taken for an image), and how a core
finds its image (`src/core/genome.h`, `--embed-dir`).

## 2. the language, by example

read `docs/LEARN.md` while running its examples:

```sh
for f in examples/learn/*.zc; do echo "== $f"; build/bin/zy < $f; done
python3 tests/drivers/learn.py build/bin/zy
```

keep `docs/SPEC.md`, the language reference, open beside it for the exact rules (precedence, types, ownership errors); then read the language parts of `README.md`: "zc, the language", "memory model",
"types", "o3", "code as data". run a few of the larger examples
(`examples/lang.zc`, `examples/bf.zc`, `examples/dump.zc`, `examples/tree.zc`).

you should be able to write a small program that includes `rt`, allocates with
`rt.new(T)`, passes an owner to a `.d` parameter, and returns one with `.u`; and
explain why `include rt;` is required before `rt.put_n`.

## 3. ownership, the analyzer's view

read `docs/BORROW.md`, then `docs/history/BOOTSTRAP-OWNERSHIP.md` (how two missing rules
once stopped the compiler compiling itself). run the rejection suites and read a
few of the fixtures, each of which starts with the message it expects:

```sh
bash tests/runerr.sh build/bin/zy tests/errors
bash tests/runerr.sh build/bin/zy tests/typeerrors
bash tests/runerr.sh build/bin/zy tests/langerrors
```

in `src/compiler/analyzer.zc`, find the block that starts at
`// the rules below are the heart of the analyzer` and read `recast`,
`argcheck`, `assigncheck` and `retcheck`, then `exit_vars` and the `lend_*`
helpers.

you should be able to say what states a local can be in (plain, m, u, d, child,
borrow, dead) and what each of `rt.free(r); r = replacement;` and `g = m*x;`
does to them.

## 4. the compiler's structure

the compiler is about ten thousand lines in twelve files, one per phase, which
the build joins in the order `src/compiler/parts` lists (one file name per line;
every script and the makefile read that list, so a new file is added there and
nowhere else). the files are one source cut at section boundaries, not modules:
they share one namespace and global state, and each only makes sense after the
ones above it. `base.zc` begins with an overview. start there, then list the
section markers of every part:

```sh
cat src/compiler/parts
grep -n "^// ----" $(sed 's|^|src/compiler/|' src/compiler/parts)
```

read in this order:

- `base.zc`: the overview, the data region, state, errors, interning, symbols,
  code emission and the o3 bookkeeping
- `lex.zc`: input sources, the lexer (`lex1`), float literals and float code
- `ops.zc`: the operator tables a program can extend
- `analyzer.zc`: the ownership pass (function effects, locals, joins, the rules
  at the places values flow)
- `types.zc`: types, parsing types, methods
- `builtins.zc`: statements that configure the compiler, `include`, built-ins
  that look like functions, the c abi (`extern`, `cfunc`), reflection, fattening
  (`kw_embed` is here)
- `expr.zc`: expressions (`primary`, `unary`, `bin`, `expr`), type facts, hints,
  closures, array initializers, constant expressions
- `stmt.zc`: statements, asm blocks, `quote`
- `decl.zc`: reading a unit, functions (`fndef`, `fn_begin`, `prologue`),
  globals (`gdecl2`), enhancements, generics
- `image.zc`: the symbol blob and `build_image` (image mode for
  `src/tools/build.zc`)
- `setup.zc`: the keyword table (`add_kw` calls), built-in names, the entry point
- `check.zc`: `chk_unit`, check mode for the language server

to see the compiler as one text, as the bootstrap does, run
`cat $(sed 's|^|src/compiler/|' src/compiler/parts)`.

the state is mostly global arrays and scalars that distant functions read and write: run `make xref` before changing compiler state, and read `build/xref/report.txt` for who writes the global and who only reads it (`make xref XREF_ARGS=--check` compares with `tests/xref.baseline`).

you should be able to say where a new keyword is registered, how a keyword
works at the top level and inside a function, and how the compiler emits a
byte-string literal into the code stream.

## 5. the vm cores

read "the instruction set", "memory map and abi", "how the core works" and "the
microcore" in `README.md`, then `docs/design/memory.md`. in `src/core/core.c`,
find `main` and follow it to `run(GORG)`; in `src/micro/micro.c`, `micro_run`.

```sh
build/bin/genmap --preset default --check | head
ZYTRACE=1 build/bin/zy < tests/programs/o3.zc 2>&1 | head -20
```

you should be able to say where the genome is loaded, where the syscall gate is,
and which backends (interpreter, arm64 jit, llvm, microcore, javascript, wasm)
run the same image.

## 6. the toolchain around it

read "language server", "vscode extension", "javascript & typescript runtime"
and "web" in `README.md`, then `editors/vscode/README.md`. build the rest:

```sh
make js web lspimage
node build/js/tinyvm.js examples/hello.zc
build/bin/zclsp < examples/opt_test.zc
```

skim `src/lsp/server.zc` from its overview: the check (`check_doc`), hovers
(`do_hover`), navigation, the formatter (`fm_*`).

you should be able to say how the editor gets diagnostics (the real compiler in
check mode), and why the javascript and wasm servers need no native build.

## 7. the libraries

read `docs/ICON.md` and `docs/ACTOR.md`, then the headers of `src/lib/std.zc`,
`src/lib/icon.zc` and `src/lib/actor.zc`. run their fixtures:

```sh
python3 tests/drivers/icon_test.py
python3 tests/drivers/actor_test.py build/bin/zy
```

you should be able to say how `include actor;` adds the `actor` keyword without
any actor code in the compiler (`run.syntax`, `run.statement_hook`).

## 8. the tests

read the header of `tests/test.sh` and `docs/design/test-coverage.md`. run the
quick suite, then the whole one when there is time:

```sh
make check
make test JOBS=4
```

`make parity` compiles the programs in `tests/programs/` and `examples/` with zc0 and
with the self-hosted compiler and diffs their output; `tests/programs/parity_ops.zc`
pins operator precedence. it uses `build/`, and skips what zc0 rejects.

the suite builds its own map, bootstrap and cores in a scratch directory, so it
does not depend on what is in `build/`. on a host without the arm64 jit the
language server is interpreted and slow; `ZC_TEST_TIMEOUT_SCALE` stretches the
drivers' deadlines (the suite sets it). `LLVM_CONFIG` selects an llvm with
headers for the llvm section.

### benchmarks

`make bench` runs `tests/bench/bench.sh` (it needs `bc`), which times the kernels of
`tests/programs/bench.zc` (collatz, sieve, digits, fib), plain and o3, on every backend
it can build here (interpreter, the host's jit, llvm, arm64 under qemu), and adds the
pointer sweep of `tests/bench/sweep.zc` (checked and unbounded) on `build/bin/zy`. the
table goes to `build/bench.txt`, one row per backend and kernel, ms per repetition in a
plain and an o3 column; start-up and compile time cancel out (slopes between one
repetition and many). the checks column waits for `--explain-checks` and the peak heap
and gc mark columns for runtime counters that do not exist yet; they show `-`. a full
run takes about ten minutes.
`make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check`
(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a
row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't
flake), or a row keeps more checks once both sides have a count. rows missing from a run
don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times
the interpreter today, so this doesn't make the run shorter yet). before timing, the suite
runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and
it refuses when jit o3 collatz isn't under a fifth of interp o3; that is how a baseline of
interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on
doctored tables in a second, and `suite.py check --rows FILE` compares a saved table
without measuring.
when something looks slower the whole run is repeated once and each cell keeps its best
figure. the baseline belongs to the machine it
was saved on: save a new one after changing machines, and expect noise on a busy host.

## 9. history and open work

`docs/history/README.md` is the scrapbook of earlier stages and `TODO.md` the
maintenance checklist. read the
open items in `TODO.md` last: by then each one should point at code you have
seen.

## checklist

after the pass you should be able to do each of these without looking anything up:

- rebuild from `make clean`, and say which file each step writes under `build/`
- add a keyword to the compiler and rebuild the genome
- write a program that the analyzer rejects, and predict the message
- build a run-only image (`make run PROG=...`) and say why its source must fit zc0
- find a built-in's declaration from a hover in the editor or with the server