bench: no-region guard, selftest and --rows, interp rows behind INTERP=1, fail only over 10% and 0.1 ms

1 files changed, 9 insertions, 1 deletions

diff --git a/ONBOARD.md b/ONBOARD.md
index f60063a..2ccb987 100644
--- a/ONBOARD.md
+++ b/ONBOARD.md
@@ -214,7 +214,15 @@ and gc mark columns for runtime counters that do not exist yet; they show `-`. a
 run takes about ten minutes.
 `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check`
 (or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a
-row is more than 10% slower, or a row keeps more checks once both sides have a count.
+row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't
+flake), or a row keeps more checks once both sides have a count. rows missing from a run
+don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times
+the interpreter today, so this doesn't make the run shorter yet). before timing, the suite
+runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and
+it refuses when jit o3 collatz isn't under a fifth of interp o3; that is how a baseline of
+interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on
+doctored tables in a second, and `suite.py check --rows FILE` compares a saved table
+without measuring.
 when something looks slower the whole run is repeated once and each cell keeps its best
 figure. the baseline belongs to the machine it
 was saved on: save a new one after changing machines, and expect noise on a busy host.