Subject: Re: Module suggestion: timeout

From: Marc Nieper-Wißkirchen <[email-masked]>
Date: Fri, 30 Apr 2021 11:37:45 +0200

Dear Bruno, dear Simon,

thank you for your replies.

I understand that valgrind-tests and the proposed "timeout-tests" solution
are not completely equivalent. Nevertheless, I still think that some
timeout functionality provided by Gnulib would be useful.

Bruno's solution

**
#if HAVE_DECL_ALARM
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);
#endif
**

works for unit tests that have been written specifically for the project.
It doesn't help, though, if I want to test (production) binaries that are
to be installed because I generally don't want testing code inside them.

Moreover, use cases for a baked-in timeout are not restricted to tests. For
example, I may want to restrict the build time of certain components in
situations where a logical error may lead to infinite build times (a simple
example is that of a Scheme compiler used as a build tool; thanks to
Turing-completeness of Scheme macros, such a build may not terminate).

Marc

Am Fr., 30. Apr. 2021 um 11:24 Uhr schrieb Simon Josefsson <
[email-masked]>:

> Bruno Haible <[email-masked]> writes:
>
> > So, I don't think the "let's treat timeout like valgrind" approach is
> going
> > to work. Instead, you need to design a way to deal with timeouts,
> independently.
>
> Hi!  I think Marc's request for functionality to introduce timeouts for
> self-tests is a good one.  However I reach the same conclusion as Bruno,
> that having a module like valgrind-tests is probably not the best way to
> solve it.  To me, having a timeout seems like an essential feature of a
> self-test framework.  I know automake isn't primarily a self-test
> framework, but it has concepts for it and the test framework has been
> improved significantly over the years, so I think adding a timeout
> functionality to automake makes sense.  What do bug-automake people
> think?
>
> The functionality could be conditioned on the coreutils 'timeout' tool,
> and if that tool exists, and appears to work, running all self-tests
> under that tool could be done automatically.  The default self-test
> timeout be quite generous (say 17 hours?) but it should be easy to
> modify both by end-user and project developer.  If we want to be
> conservative, the functionality could be opt-in initially, and then
> after a few years become the default behaviour.
>
> Thoughts?
>
> /Simon
>