Performance monitoring #14
Description
We should keep track of performance over time and for PRs.
Can be either manually when PRing, on pre-commit, or in CI.
The first two are more reliable for timings, but the latter can automatically use both x86 and arm CPUs.
I'm thinking we probably want to count the number of cycles rather than instructions, since more or less instruction-level-parallellism is very relevant for us.
(Counting wall-clock time is also fine as a starting point, but extremely brittle on github runners.)
Rust uses measureme https://github.com/rust-lang/measureme for self profiling, but it doesn't support cycle counters currently. I'd probably just extract the relevant bits out of that.
There's also Bencher: https://bencher.dev/, but it feels too commercial (even when free).
Maybe having a look at iai https://github.com/bheisler/iai is better actually; a one-shot runner based on criterion.
As for how to track regressions: for one of my previous projects I had something that ran in CI and pushed to a perf branch which contained a single json file with data, and then a simple html page with embedded js that render a nice plot.
Simpler could be to just append values to a .tsv file in master by ourselves.
For A*PA I used benchmark-action/github-action-benchmark, but I dropped it (commit) because the timings were too unstable. But maybe with measuring cycles it's better.
See also here: https://nnethercote.github.io/perf-book/profiling.html
and here: https://bsky.app/profile/permutans.bsky.social/post/3m2mnv6jidc2d