aboutsummaryrefslogtreecommitdiff
path: root/ONBOARD.md
diff options
context:
space:
mode:
Diffstat (limited to 'ONBOARD.md')
-rw-r--r--ONBOARD.md10
1 files changed, 9 insertions, 1 deletions
diff --git a/ONBOARD.md b/ONBOARD.md
index f60063a..2ccb987 100644
--- a/ONBOARD.md
+++ b/ONBOARD.md
@@ -214,7 +214,15 @@ and gc mark columns for runtime counters that do not exist yet; they show `-`. a
run takes about ten minutes.
`make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check`
(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a
-row is more than 10% slower, or a row keeps more checks once both sides have a count.
+row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't
+flake), or a row keeps more checks once both sides have a count. rows missing from a run
+don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times
+the interpreter today, so this doesn't make the run shorter yet). before timing, the suite
+runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and
+it refuses when jit o3 collatz isn't under a fifth of interp o3; that is how a baseline of
+interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on
+doctored tables in a second, and `suite.py check --rows FILE` compares a saved table
+without measuring.
when something looks slower the whole run is repeated once and each cell keeps its best
figure. the baseline belongs to the machine it
was saved on: save a new one after changing machines, and expect noise on a busy host.