GNU bug report logs - #71988
31.0.50; ert-test-run-tests-batch-expensive runs out of memory if previous tests failed

Previous Next

To reply to this bug, email your comments to 71988 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: 31.0.50;
 ert-test-run-tests-batch-expensive runs out of memory if previous
 tests failed
Date: Mon, 08 Jul 2024 04:49:54 +0000

[Message part 1 (text/plain, inline)]

This bug report is mostly to get a bug number to put in the patch :-).

The test `ert-test-run-tests-batch-expensive' overrides print settings
and prints a full backtrace to a temporary buffer repeatedly. If a
previous test, such as `ert-test-run-tests-batch', failed, this
eventually runs out of memory even on a 64 GB system.

For example, one can add "(should nil)" to ert-test-run-tests-batch and
run:

$ make -C test lisp/emacs-lisp/ert-tests

The problem is the ert--stats structure's tests vector is being printed,
which recursively prints all previous tests' data.

A possible fix is to temporarily override cl-print-object for either
vectors or the ert--stats object to print nothing.

Unfortunately, we don't appear to have an official mechanism for
temporarily calling cl-defmethod, such as a generalized variable one can
cl-letf to. The patch does that by running cl-defmethod inside a
(cl-letf* (((symbol-function 'cl-print-object) (symbol-function
'cl-print-object))) ...) form. As future changes to cl-generic might
break that, an additional test is added to make sure our binding is
undone and die loudly if it hasn't been.

This also speeds up the test significantly and it might be possible to
remove the :unstable tag.

[0001-ert-tests-reproducer.patch (text/x-patch, attachment)]

[0001-ert-tests.patch (text/x-patch, attachment)]

Message #8 received at 71988 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Pip Cet <pipcet <at> protonmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 71988 <at> debbugs.gnu.org
Subject: Re: bug#71988: 31.0.50;
 ert-test-run-tests-batch-expensive runs out of memory if previous
 tests failed
Date: Mon, 08 Jul 2024 14:17:26 +0300

> Date: Mon, 08 Jul 2024 04:49:54 +0000
> From:  Pip Cet via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> This bug report is mostly to get a bug number to put in the patch :-).
> 
> The test `ert-test-run-tests-batch-expensive' overrides print settings
> and prints a full backtrace to a temporary buffer repeatedly. If a
> previous test, such as `ert-test-run-tests-batch', failed, this
> eventually runs out of memory even on a 64 GB system.
> 
> For example, one can add "(should nil)" to ert-test-run-tests-batch and
> run:
> 
> $ make -C test lisp/emacs-lisp/ert-tests
> 
> The problem is the ert--stats structure's tests vector is being printed,
> which recursively prints all previous tests' data.
> 
> A possible fix is to temporarily override cl-print-object for either
> vectors or the ert--stats object to print nothing.
> 
> Unfortunately, we don't appear to have an official mechanism for
> temporarily calling cl-defmethod, such as a generalized variable one can
> cl-letf to. The patch does that by running cl-defmethod inside a
> (cl-letf* (((symbol-function 'cl-print-object) (symbol-function
> 'cl-print-object))) ...) form. As future changes to cl-generic might
> break that, an additional test is added to make sure our binding is
> undone and die loudly if it hasn't been.
> 
> This also speeds up the test significantly and it might be possible to
> remove the :unstable tag.
> 
> 
> diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
> index 1aff73d66f6..2f8e7382bea 100644
> --- a/test/lisp/emacs-lisp/ert-tests.el
> +++ b/test/lisp/emacs-lisp/ert-tests.el
> @@ -583,6 +583,7 @@ ert-test-run-tests-batch
>  		   (setq found-long (string-match long-text msg)))
>  		 (unless found-complex
>  		   (setq found-complex (string-match complex-text msg))))
> +        (should nil)
>  	(should found-long)
>  	(should found-complex)))))

Stefan, any comments or suggestions?

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: Re: 31.0.50;
 ert-test-run-tests-batch-expensive runs out of memory if previous
 tests failed
Date: Mon, 08 Jul 2024 19:54:25 +0000

[Message part 1 (text/plain, inline)]

On Monday, July 8th, 2024 at 04:49, Pip Cet <pipcet <at> protonmail.com> wrote:
> This bug report is mostly to get a bug number to put in the patch :-).

Which I'm now attaching. Sorry about that.

Pip

[0001-Bug-71988-Avoid-excessively-large-backtraces-in-ert-.patch (text/x-patch, attachment)]

Message #14 received at 71988 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: 71988 <at> debbugs.gnu.org
Subject: Re: bug#71988: 31.0.50; ert-test-run-tests-batch-expensive runs out
 of memory if previous tests failed
Date: Fri, 28 Feb 2025 17:43:53 -0800

tags 71988 + patch
thanks

Pip Cet <pipcet <at> protonmail.com> writes:

> On Monday, July 8th, 2024 at 04:49, Pip Cet <pipcet <at> protonmail.com> wrote:
>> This bug report is mostly to get a bug number to put in the patch :-).
>
> Which I'm now attaching. Sorry about that.

Makes sense to me.  How about installing it?

>
> Pip
>
> From 8f7e9b50f0edd33f601fab0842c5b4a7e4398490 Mon Sep 17 00:00:00 2001
> From: Pip Cet <pipcet <at> protonmail.com>
> Date: Mon, 8 Jul 2024 19:50:16 +0000
> Subject: [PATCH] (Bug#71988) Avoid excessively large backtraces in ert-tests
>
> * test/lisp/emacs-lisp/ert-tests.el (ert-test-run-tests-batch-expensive):
> Temporarily make `cl-print-object' not print out vectors while running
> the test.
> ---
>  test/lisp/emacs-lisp/ert-tests.el | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
> index 1aff73d66f6..4237d354f38 100644
> --- a/test/lisp/emacs-lisp/ert-tests.el
> +++ b/test/lisp/emacs-lisp/ert-tests.el
> @@ -594,9 +594,13 @@ ert-test-run-tests-batch-expensive
>  			 :body (lambda () (should (equal complex-list 1))))))
>      (let ((ert-debug-on-error nil)
>            messages)
> -      (cl-letf* (((symbol-function 'message)
> +      (cl-letf* (((symbol-function 'cl-print-object) (symbol-function 'cl-print-object))
> +                 ((symbol-function 'message)
>                    (lambda (format-string &rest args)
>                      (push (apply #'format format-string args) messages))))
> +        ;; don't print the ert--stats-tests vector, which would run out of
> +        ;; memory if previous tests have failed.
> +        (cl-defmethod cl-print-object ((object vector) stream))
>          (save-window-excursion
>            (let ((case-fold-search nil)
>                  (ert-batch-backtrace-right-margin nil)
> @@ -611,7 +615,9 @@ ert-test-run-tests-batch-expensive
>  		 do
>  		 (unless found-frame
>  		   (setq found-frame (cl-search frame msg :test 'equal))))
> -        (should found-frame)))))
> +        (should found-frame))))
> +  ;; ensure our temporary binding was undone.
> +  (should (equal (cl-prin1-to-string [a]) "[a]")))
>
>  (ert-deftest ert-test-special-operator-p ()
>    (should (ert--special-operator-p 'if))

Message #19 received at 71988 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 71988 <at> debbugs.gnu.org
Subject: Re: bug#71988: 31.0.50;
 ert-test-run-tests-batch-expensive runs out of memory if previous
 tests failed
Date: Sat, 01 Mar 2025 09:47:18 +0000

"Stefan Kangas" <stefankangas <at> gmail.com> writes:

> tags 71988 + patch
> thanks
>
> Pip Cet <pipcet <at> protonmail.com> writes:
>
>> On Monday, July 8th, 2024 at 04:49, Pip Cet <pipcet <at> protonmail.com> wrote:
>>> This bug report is mostly to get a bug number to put in the patch :-).
>>
>> Which I'm now attaching. Sorry about that.
>
> Makes sense to me.  How about installing it?

I'm not sure whether this needs to be applied to
ert-test-run-tests-batch and ert-test-run-tests-interactively, too (and
at that point we'd probably need to make it a macro capturing messages).

However, I can't reproduce the problem with the patch applied, so I'll
push it assuming Stefan Monnier doesn't speak up against the local
overriding of cl-print-object.  Is that something we can rely on working
in the future, too?

Pip

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.