From b3273f04b876228d7b84a9d5a0fe053eb95cdb95 Mon Sep 17 00:00:00 2001 From: zinc Date: Wed, 14 Oct 2026 17:54:07 +0330 Subject: bench: no-region guard, selftest and --rows, interp rows behind INTERP=1, fail only over 10% and 0.1 ms --- ONBOARD.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) (limited to 'ONBOARD.md') diff --git a/ONBOARD.md b/ONBOARD.md index f60063a..2ccb987 100644 --- a/ONBOARD.md +++ b/ONBOARD.md @@ -214,7 +214,15 @@ and gc mark columns for runtime counters that do not exist yet; they show `-`. a run takes about ten minutes. `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check` (or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a -row is more than 10% slower, or a row keeps more checks once both sides have a count. +row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't +flake), or a row keeps more checks once both sides have a count. rows missing from a run +don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times +the interpreter today, so this doesn't make the run shorter yet). before timing, the suite +runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and +it refuses when jit o3 collatz isn't under a fifth of interp o3; that is how a baseline of +interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on +doctored tables in a second, and `suite.py check --rows FILE` compares a saved table +without measuring. when something looks slower the whole run is repeated once and each cell keeps its best figure. the baseline belongs to the machine it was saved on: save a new one after changing machines, and expect noise on a busy host. -- cgit v1.2.3