diff options
| -rw-r--r-- | ONBOARD.md | 23 | ||||
| -rw-r--r-- | tests/bench.baseline | 28 |
2 files changed, 29 insertions, 22 deletions
@@ -203,17 +203,20 @@ headers for the llvm section. ### benchmarks -`make bench` times the benchmark programs on `build/bin/zy` (the host's jit where -there is one) and writes `build/bench.txt`: the kernels of `tests/programs/bench.zc` -(collatz, sieve, digits, fib, each plain and o3) and the pointer sweep of -`tests/bench/sweep.zc` (checked and unbounded). each figure is ms per repetition, the -median of five slopes between one repetition and many, so start-up and compile time -cancel out. the checks column waits for `--explain-checks` and the peak heap and gc -mark columns for runtime counters that do not exist yet; they show `-`. +`make bench` runs `tests/bench/bench.sh` (it needs `bc`), which times the kernels of +`tests/programs/bench.zc` (collatz, sieve, digits, fib), plain and o3, on every backend +it can build here (interpreter, the host's jit, llvm, arm64 under qemu), and adds the +pointer sweep of `tests/bench/sweep.zc` (checked and unbounded) on `build/bin/zy`. the +table goes to `build/bench.txt`, one row per backend and kernel, ms per repetition in a +plain and an o3 column; start-up and compile time cancel out (slopes between one +repetition and many). the checks column waits for `--explain-checks` and the peak heap +and gc mark columns for runtime counters that do not exist yet; they show `-`. a full +run takes about ten minutes. `make bench-save` keeps the result as `tests/bench.baseline`, and `make bench-check` -(or `make bench CHECK=1`) measures again and fails when a program is more than 10% -slower, or keeps more checks once both sides have a count. a program that looks slower -is measured up to twice more before it counts. the baseline belongs to the machine it +(or `make bench CHECK=1`) measures again and fails when the plain or the o3 column of a +row is more than 10% slower, or a row keeps more checks once both sides have a count. +when something looks slower the whole run is repeated once and each cell keeps its best +figure. the baseline belongs to the machine it was saved on: save a new one after changing machines, and expect noise on a busy host. ## 9. history and open work diff --git a/tests/bench.baseline b/tests/bench.baseline index b3ac5d3..18fba1c 100644 --- a/tests/bench.baseline +++ b/tests/bench.baseline @@ -1,12 +1,16 @@ -# tinyvm bench: ms per repetition (median of 5 slopes); '-' where a figure is not available -program ms checks peak_heap_words gc_mark_ms -collatz 32.547 - - - -collatz.o3 255.490 - - - -sieve 4.411 - - - -sieve.o3 8.302 - - - -digits 15.497 - - - -digits.o3 81.287 - - - -fib 0.822 - - - -fib.o3 3.267 - - - -sweep.checked 3.371 - - - -sweep.unbounded 2.382 - - - +# tinyvm bench: ms per repetition, from tests/bench/bench.sh and tests/bench/suite.py; '-' where a figure is not available +program plain_ms o3_ms checks peak_heap_words gc_mark_ms +interp:collatz 594.240 259.680 - - - +interp:sieve 82.310 8.400 - - - +interp:digits 328.180 84.460 - - - +interp:fib 10.200 2.820 - - - +x86-64-jit:collatz 31.410 10.750 - - - +x86-64-jit:sieve 4.320 0.410 - - - +x86-64-jit:digits 14.750 3.070 - - - +x86-64-jit:fib 0.780 0.860 - - - +llvm:collatz 54.080 3.420 - - - +llvm:sieve 13.560 0.190 - - - +llvm:digits 26.590 1.600 - - - +llvm:fib 2.970 0.150 - - - +zy:sweep.checked 3.535 - - - - +zy:sweep.unbounded 2.625 - - - - |