Subject: bug#75356: impossible to benchmark byte-compiled code in a native-comp build

From: Pip Cet <[email-masked]>
Date: Sat, 04 Jan 2025 16:35:01 +0000

elisp-benchmarks.el unconditionally forces use of native compilation
if run on an Emacs which is compiled with native-comp enabled.  This
is a minor issue, but it's still important to benchmark this
configuration once in a while.

For example, running

./src/emacs -Q --batch --load elisp-benchmarks/elisp-benchmarks.el --eval '(progn (setq no-native-compile t) (elisp-benchmarks-run))'

produces error messages and an "empty" results table:

  | test  | non-gc (s) | gc (s) | gcs | total (s) | err (s) |
  |-------+------------+--------+-----+-----------+---------|
  |-------+------------+--------+-----+-----------+---------|
  | total |       0.00 |   0.00 |   0 |      0.00 |    0.00 |

In particular, I believe it is legitimate to use a native-comp build
on a system which no longer supports compiling additional code.

In general, the compilation logic of elisp-benchmarks.el is fragile
and will lead to unreliabe test results, which is the worst possible
outcome for a benchmark (in some cases, we simply fail to produce
data, which is better but still not good).