GNU bug report logs - #75359
[scratch/elisp-benchmarks] test dependencies

Previous Next

Full log

View this message in rfc822 format

From: Andrea Corallo <acorallo <at> gnu.org>
To: 75359 <at> debbugs.gnu.org
Cc: pipcet <at> protonmail.com, monnier <at> iro.umontreal.ca
Subject: bug#75359: [scratch/elisp-benchmarks] test dependencies
Date: Mon, 06 Jan 2025 04:53:45 -0500

Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs <at> gnu.org> writes:

> There are dependencies between emacs-lisp-benchmarks tests, meaning we
> can't always modify one test without affecting others.  The ones I'm
> aware of are:
>
> - elb-bytecomp.el uses elb-smie.el as compilation input (in this
>   context, elb-smie.el is a "resource", not source code).  If we
>   modify elb-smie.el, elb-bytecomp.el results will change and become
>   uncomparable.

Shouldn't we just copy of elb-smie.el to resources?

> - elb-scroll.el uses elb-smie.el as a prerequisite.  If we modify
>   elb-smie.el, elb-scroll.el results will change and become
>   uncomparable.  Furthermore, elb-scroll.el relies on elb-smie.el
>   being compiled in the appropriate fashion by some external
>   mechanism.  (I think the best solution here is to merge elb-smie.el
>   and elb-scroll.el into a single file declaring several tests, using
>   the established ERT test framework)

I'm not sure these are real/serious bugs, different versions of
elisp-benchmarks don't guarantee to give comparable results (like every
benchmark).  Anyway adding Stefan who added this code.

> - bubble.el and bubble-no-cons.el both "provide" the same feature.
>   I'm not sure whether this causes any acute problems, but it
>   demonstrates that the inconsistent use of (provide) in the
>   benchmarks is problematic.  If we ever load tests using Frequire, we
>   will load the wrong files.  If we don't, provide shouldn't be
>   necessary.

AFAIU the provide is not used by elisp-benchmarks, anyway the typo is
now fixed.

> - xmenu.c is used as a resource; this is fine.  But we must make sure
>   we never copy the "current" xmenu.c to the benchmarks directory
>   without clearly indicating that the benchmark results will become
>   uncomparable.

Again, changes to the benchmark pertubating results should be tagged as
a new version, so will be fine to update xmenu.c if necessary as long as
we update elisp-benchmarks version.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.