Re: Module suggestion: timeout

[Marc Nieper-Wißkirchen]

Dear Bruno, dear Simon,

thank you for your replies.

I understand that valgrind-tests and the proposed "timeout-tests" solution are not completely equivalent. Nevertheless, I still think that some timeout functionality provided by Gnulib would be useful.

Bruno's solution

**

#if HAVE_DECL_ALARM
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);
#endif

**

works for unit tests that have been written specifically for the project. It doesn't help, though, if I want to test (production) binaries that are to be installed because I generally don't want testing code inside them.

Moreover, use cases for a baked-in timeout are not restricted to tests. For example, I may want to restrict the build time of certain components in situations where a logical error may lead to infinite build times (a simple example is that of a Scheme compiler used as a build tool; thanks to Turing-completeness of Scheme macros, such a build may not terminate).

Marc

Am Fr., 30. Apr. 2021 um 11:24 Uhr schrieb Simon Josefsson <simon@josefsson.org>:

Bruno Haible <bruno@clisp.org> writes:

> So, I don't think the "let's treat timeout like valgrind" approach is going
> to work. Instead, you need to design a way to deal with timeouts, independently.

Hi!  I think Marc's request for functionality to introduce timeouts for
self-tests is a good one.  However I reach the same conclusion as Bruno,
that having a module like valgrind-tests is probably not the best way to
solve it.  To me, having a timeout seems like an essential feature of a
self-test framework.  I know automake isn't primarily a self-test
framework, but it has concepts for it and the test framework has been
improved significantly over the years, so I think adding a timeout
functionality to automake makes sense.  What do bug-automake people
think?

The functionality could be conditioned on the coreutils 'timeout' tool,
and if that tool exists, and appears to work, running all self-tests
under that tool could be done automatically.  The default self-test
timeout be quite generous (say 17 hours?) but it should be easy to
modify both by end-user and project developer.  If we want to be
conservative, the functionality could be opt-in initially, and then
after a few years become the default behaviour.

Thoughts?

/Simon