merge work/bench: make bench, bench-check, bench-selftest; baseline at 89f35bd

1 files changed, 26 insertions, 0 deletions

diff --git a/ONBOARD.md b/ONBOARD.md
index 35d5e35..0f1c95c 100644
--- a/ONBOARD.md
+++ b/ONBOARD.md
@@ -201,6 +201,32 @@ language server is interpreted and slow; `ZC_TEST_TIMEOUT_SCALE` stretches the
 drivers' deadlines (the suite sets it). `LLVM_CONFIG` selects an llvm with
 headers for the llvm section.
 
+### benchmarks
+
+`make bench` runs `tests/bench/bench.sh` (it needs `bc`), which times the kernels of
+`tests/programs/bench.zc` (collatz, sieve, digits, fib), plain and o3, on every backend
+it can build here (interpreter, the host's jit, llvm, arm64 under qemu), and adds the
+pointer sweep of `tests/bench/sweep.zc` (checked and unbounded) on `build/bin/zy`. the
+table goes to `build/bench.txt`, one row per backend and kernel, ms per repetition in a
+plain and an o3 column; start-up and compile time cancel out (slopes between one
+repetition and many). the checks column waits for `--explain-checks` and the peak heap
+and gc mark columns for runtime counters that do not exist yet; they show `-`. a full
+run takes about ten minutes.
+`make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check`
+(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a
+row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't
+flake), or a row keeps more checks once both sides have a count. rows missing from a run
+don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times
+the interpreter today, so this doesn't make the run shorter yet). before timing, the suite
+runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and
+it refuses when jit o3 collatz isn't under half of jit plain collatz; that is how a baseline of
+interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on
+doctored tables in a second, and `suite.py check --rows FILE` compares a saved table
+without measuring.
+when something looks slower the whole run is repeated once and each cell keeps its best
+figure. the baseline belongs to the machine it
+was saved on: save a new one after changing machines, and expect noise on a busy host.
+
 ## 9. history and open work
 
 `docs/history/README.md` is the scrapbook of earlier stages and `TODO.md` the