GNU bug report logs - #48113
Self-test timeout functionality

Previous Next

To reply to this bug, email your comments to 48113 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Bruno Haible <bruno <at> clisp.org>
Cc: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 bug-automake <at> gnu.org, bug-gnulib <at> gnu.org
Subject: Re: Module suggestion: timeout
Date: Fri, 30 Apr 2021 11:24:04 +0200

[Message part 1 (text/plain, inline)]

Bruno Haible <bruno <at> clisp.org> writes:

> So, I don't think the "let's treat timeout like valgrind" approach is going
> to work. Instead, you need to design a way to deal with timeouts, independently.

Hi!  I think Marc's request for functionality to introduce timeouts for
self-tests is a good one.  However I reach the same conclusion as Bruno,
that having a module like valgrind-tests is probably not the best way to
solve it.  To me, having a timeout seems like an essential feature of a
self-test framework.  I know automake isn't primarily a self-test
framework, but it has concepts for it and the test framework has been
improved significantly over the years, so I think adding a timeout
functionality to automake makes sense.  What do bug-automake people
think?

The functionality could be conditioned on the coreutils 'timeout' tool,
and if that tool exists, and appears to work, running all self-tests
under that tool could be done automatically.  The default self-test
timeout be quite generous (say 17 hours?) but it should be easy to
modify both by end-user and project developer.  If we want to be
conservative, the functionality could be opt-in initially, and then
after a few years become the default behaviour.

Thoughts?

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: "bug-gnulib <at> gnu.org List" <bug-gnulib <at> gnu.org>, bug-automake <at> gnu.org,
 Bruno Haible <bruno <at> clisp.org>,
 Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
Subject: Re: Module suggestion: timeout
Date: Fri, 30 Apr 2021 11:37:45 +0200

[Message part 1 (text/plain, inline)]

Dear Bruno, dear Simon,

thank you for your replies.

I understand that valgrind-tests and the proposed "timeout-tests" solution
are not completely equivalent. Nevertheless, I still think that some
timeout functionality provided by Gnulib would be useful.

Bruno's solution

**
#if HAVE_DECL_ALARM
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);
#endif
**

works for unit tests that have been written specifically for the project.
It doesn't help, though, if I want to test (production) binaries that are
to be installed because I generally don't want testing code inside them.

Moreover, use cases for a baked-in timeout are not restricted to tests. For
example, I may want to restrict the build time of certain components in
situations where a logical error may lead to infinite build times (a simple
example is that of a Scheme compiler used as a build tool; thanks to
Turing-completeness of Scheme macros, such a build may not terminate).

Marc

Am Fr., 30. Apr. 2021 um 11:24 Uhr schrieb Simon Josefsson <
simon <at> josefsson.org>:

> Bruno Haible <bruno <at> clisp.org> writes:
>
> > So, I don't think the "let's treat timeout like valgrind" approach is
> going
> > to work. Instead, you need to design a way to deal with timeouts,
> independently.
>
> Hi!  I think Marc's request for functionality to introduce timeouts for
> self-tests is a good one.  However I reach the same conclusion as Bruno,
> that having a module like valgrind-tests is probably not the best way to
> solve it.  To me, having a timeout seems like an essential feature of a
> self-test framework.  I know automake isn't primarily a self-test
> framework, but it has concepts for it and the test framework has been
> improved significantly over the years, so I think adding a timeout
> functionality to automake makes sense.  What do bug-automake people
> think?
>
> The functionality could be conditioned on the coreutils 'timeout' tool,
> and if that tool exists, and appears to work, running all self-tests
> under that tool could be done automatically.  The default self-test
> timeout be quite generous (say 17 hours?) but it should be easy to
> modify both by end-user and project developer.  If we want to be
> conservative, the functionality could be opt-in initially, and then
> after a few years become the default behaviour.
>
> Thoughts?
>
> /Simon
>

[Message part 2 (text/html, inline)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
Cc: "bug-gnulib <at> gnu.org List" <bug-gnulib <at> gnu.org>, bug-automake <at> gnu.org,
 Bruno Haible <bruno <at> clisp.org>
Subject: Re: Module suggestion: timeout
Date: Fri, 30 Apr 2021 11:43:16 +0200

[Message part 1 (text/plain, inline)]

Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com> writes:

> Moreover, use cases for a baked-in timeout are not restricted to tests. For
> example, I may want to restrict the build time of certain components in
> situations where a logical error may lead to infinite build times (a simple
> example is that of a Scheme compiler used as a build tool; thanks to
> Turing-completeness of Scheme macros, such a build may not terminate).

This makes me believe even stronger that the functionality ought to be
provided by automake natively -- it seems the desired functionality is
not only timeouts for self-tests but timeouts for all operations.

Implementing this for self-tests in automake would probably be quite
simple, but implementing it for all operations is probably more
complicated.  Maybe it should be two separate features.  I have wanted
timeouts for self-tests but rarely for building.  If the second is more
complicated to implement, maybe starting with the first will be
sufficient and useful.

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: bug-automake <at> gnu.org,
 Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 "bug-gnulib <at> gnu.org List" <bug-gnulib <at> gnu.org>, Bruno Haible <bruno <at> clisp.org>
Subject: Re: Module suggestion: timeout
Date: Fri, 30 Apr 2021 11:47:14 +0200

[Message part 1 (text/plain, inline)]

Adding timeout to all build operations automatically is probably too much
and too intrusive.

But in any case, it should be easy to add such a timeout to certain build
operations (e.g. downloading, running a compiler for which the halting
problem cannot be solved, testing a production binary, ...).

Am Fr., 30. Apr. 2021 um 11:43 Uhr schrieb Simon Josefsson <
simon <at> josefsson.org>:

> Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com> writes:
>
> > Moreover, use cases for a baked-in timeout are not restricted to tests.
> For
> > example, I may want to restrict the build time of certain components in
> > situations where a logical error may lead to infinite build times (a
> simple
> > example is that of a Scheme compiler used as a build tool; thanks to
> > Turing-completeness of Scheme macros, such a build may not terminate).
>
> This makes me believe even stronger that the functionality ought to be
> provided by automake natively -- it seems the desired functionality is
> not only timeouts for self-tests but timeouts for all operations.
>
> Implementing this for self-tests in automake would probably be quite
> simple, but implementing it for all operations is probably more
> complicated.  Maybe it should be two separate features.  I have wanted
> timeouts for self-tests but rarely for building.  If the second is more
> complicated to implement, maybe starting with the first will be
> sufficient and useful.
>
> /Simon
>

[Message part 2 (text/html, inline)]

Message #17 received at 48113 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: simon <at> josefsson.org
Cc: marc.nieper+gnu <at> gmail.com, bug-gnulib <at> gnu.org, bruno <at> clisp.org,
 48113 <at> debbugs.gnu.org
Subject: Re: bug#48113: Module suggestion: timeout
Date: Sat, 1 May 2021 15:13:00 -0600

    What do bug-automake people think?

For myself, I have no objection to sprinkling timeout commands through
the Automake test infrastructure wherever appropriate. It's not ever
going to rise to the top of my own list of things to do, though, so if
it's going to happen, you or someone will have to write the patch.

Of course I don't speak for Jim, but from what he's said in the past,
I suspect he is in a similar situation.

No one else has come forward to work on Automake, despite my plea
(https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html),
so I guess that's where we are.
    
    the functionality could be opt-in initially, 

Certainly.

    and then after a few years become the default behaviour.

Personally, I think it should be opt-in forever. People could easily
have test suites that need to run for days. I prefer not to
unnecessarily break compatibility.

Thanks,
Karl

Message #20 received at 48113 <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: marc.nieper+gnu <at> gmail.com, bug-gnulib <at> gnu.org, bruno <at> clisp.org,
 48113 <at> debbugs.gnu.org
Subject: Re: bug#48113: Module suggestion: timeout
Date: Sun, 02 May 2021 09:30:29 +0200

[Message part 1 (text/plain, inline)]

severity 48113 wishlist
retitle 48113 Self-test timeout functionality
thanks

Karl Berry <karl <at> freefriends.org> writes:

>     What do bug-automake people think?
>
> For myself, I have no objection to sprinkling timeout commands through
> the Automake test infrastructure wherever appropriate. It's not ever
> going to rise to the top of my own list of things to do, though, so if
> it's going to happen, you or someone will have to write the patch.
>
> Of course I don't speak for Jim, but from what he's said in the past,
> I suspect he is in a similar situation.

Thanks for confirming that it isn't an obviously bad idea.  Tagging this
as a wishlist bug, and improving the bug title a bit.

> No one else has come forward to work on Automake, despite my plea
> (https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html),
> so I guess that's where we are.

Ah right -- I'll see if I can help in any way, but will respond
separately.

>     the functionality could be opt-in initially, 
>
> Certainly.
>
>     and then after a few years become the default behaviour.
>
> Personally, I think it should be opt-in forever. People could easily
> have test suites that need to run for days. I prefer not to
> unnecessarily break compatibility.

Yes, I probably agree with this -- chosing the default timeout setting
is difficult as it will most likely just cause problems for some people
with long-running tests, and not solve the initial problem for some
other people (that have small projects and a timeout of 5 minutes is
what you want).  Better leave this up the each project to decide.

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #27 received at 48113 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: 48113 <at> debbugs.gnu.org
Subject: Re: bug#48113: Module suggestion: timeout
Date: Mon, 21 Jun 2021 15:04:37 -0600

Just to mention further evidence supporting Simon's final idea of
"choosing the default timeout setting is difficult". This report
recently came into bug-dejagnu precisely about different timeouts being
needed on different systems:
  https://lists.gnu.org/archive/html/bug-dejagnu/2021-05/msg00011.html

And the replies by the maintainer reject the idea of a single timeout.

No action required, just thought it was interesting that the same
general issue came up elsewhere. -k

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.