diff options
Diffstat (limited to 'ONBOARD.md')
| -rw-r--r-- | ONBOARD.md | 10 |
1 files changed, 9 insertions, 1 deletions
@@ -214,7 +214,15 @@ and gc mark columns for runtime counters that do not exist yet; they show `-`. a run takes about ten minutes. `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check` (or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a -row is more than 10% slower, or a row keeps more checks once both sides have a count. +row is more than 10% slower and also more than 0.1 ms slower (so sub-ms cells don't +flake), or a row keeps more checks once both sides have a count. rows missing from a run +don't fail. interp rows are left out of the table unless `INTERP=1` (bench.sh still times +the interpreter today, so this doesn't make the run shorter yet). before timing, the suite +runs `ZYTRACE=1` on `tests/programs/o3.zc` and refuses when zy compiles no o3 region, and +it refuses when jit o3 collatz isn't under a fifth of interp o3; that is how a baseline of +interpreter numbers got saved once. `make bench-selftest` checks the pass/fail verdicts on +doctored tables in a second, and `suite.py check --rows FILE` compares a saved table +without measuring. when something looks slower the whole run is repeated once and each cell keeps its best figure. the baseline belongs to the machine it was saved on: save a new one after changing machines, and expect noise on a busy host. |
