make bench, bench-save, bench-check: wrap tests/bench/bench.sh plus a pointer sweep; baseline after jit step 2

1 files changed, 13 insertions, 10 deletions

diff --git a/ONBOARD.md b/ONBOARD.md
index 7a7d46d..f60063a 100644
--- a/ONBOARD.md
+++ b/ONBOARD.md
@@ -203,17 +203,20 @@ headers for the llvm section.
 
 ### benchmarks
 
-`make bench` times the benchmark programs on `build/bin/zy` (the host's jit where
-there is one) and writes `build/bench.txt`: the kernels of `tests/programs/bench.zc`
-(collatz, sieve, digits, fib, each plain and o3) and the pointer sweep of
-`tests/bench/sweep.zc` (checked and unbounded). each figure is ms per repetition, the
-median of five slopes between one repetition and many, so start-up and compile time
-cancel out. the checks column waits for `--explain-checks` and the peak heap and gc
-mark columns for runtime counters that do not exist yet; they show `-`.
+`make bench` runs `tests/bench/bench.sh` (it needs `bc`), which times the kernels of
+`tests/programs/bench.zc` (collatz, sieve, digits, fib), plain and o3, on every backend
+it can build here (interpreter, the host's jit, llvm, arm64 under qemu), and adds the
+pointer sweep of `tests/bench/sweep.zc` (checked and unbounded) on `build/bin/zy`. the
+table goes to `build/bench.txt`, one row per backend and kernel, ms per repetition in a
+plain and an o3 column; start-up and compile time cancel out (slopes between one
+repetition and many). the checks column waits for `--explain-checks` and the peak heap
+and gc mark columns for runtime counters that do not exist yet; they show `-`. a full
+run takes about ten minutes.
 `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check`
-(or `make bench CHECK=1`) measures again and fails when a program is more than 10%
-slower, or keeps more checks once both sides have a count. a program that looks slower
-is measured up to twice more before it counts. the baseline belongs to the machine it
+(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a
+row is more than 10% slower, or a row keeps more checks once both sides have a count.
+when something looks slower the whole run is repeated once and each cell keeps its best
+figure. the baseline belongs to the machine it
 was saved on: save a new one after changing machines, and expect noise on a busy host.
 
 ## 9. history and open work