diff options
Diffstat (limited to 'ONBOARD.md')
| -rw-r--r-- | ONBOARD.md | 23 |
1 files changed, 13 insertions, 10 deletions
@@ -203,17 +203,20 @@ headers for the llvm section. ### benchmarks -`make bench` times the benchmark programs on `build/bin/zy` (the host's jit where -there is one) and writes `build/bench.txt`: the kernels of `tests/programs/bench.zc` -(collatz, sieve, digits, fib, each plain and o3) and the pointer sweep of -`tests/bench/sweep.zc` (checked and unbounded). each figure is ms per repetition, the -median of five slopes between one repetition and many, so start-up and compile time -cancel out. the checks column waits for `--explain-checks` and the peak heap and gc -mark columns for runtime counters that do not exist yet; they show `-`. +`make bench` runs `tests/bench/bench.sh` (it needs `bc`), which times the kernels of +`tests/programs/bench.zc` (collatz, sieve, digits, fib), plain and o3, on every backend +it can build here (interpreter, the host's jit, llvm, arm64 under qemu), and adds the +pointer sweep of `tests/bench/sweep.zc` (checked and unbounded) on `build/bin/zy`. the +table goes to `build/bench.txt`, one row per backend and kernel, ms per repetition in a +plain and an o3 column; start-up and compile time cancel out (slopes between one +repetition and many). the checks column waits for `--explain-checks` and the peak heap +and gc mark columns for runtime counters that do not exist yet; they show `-`. a full +run takes about ten minutes. `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check` -(or `make bench CHECK=1`) measures again and fails when a program is more than 10% -slower, or keeps more checks once both sides have a count. a program that looks slower -is measured up to twice more before it counts. the baseline belongs to the machine it +(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a +row is more than 10% slower, or a row keeps more checks once both sides have a count. +when something looks slower the whole run is repeated once and each cell keeps its best +figure. the baseline belongs to the machine it was saved on: save a new one after changing machines, and expect noise on a busy host. ## 9. history and open work |