Re: GNU make 4.3.91 on Debian 9.1

[Bruno Haible]

Hi Paul,

> For example, for the failures that appear to be
> timeouts sometimes the test suite will generate some information about
> that, to stdout/stderr.
> 
> Maybe there's some make target or script that, if it exists, the
> platform testing scripts will run on failure to generate the tar file?

Some testers (like Nelson Beebe) have more evolved "platform testing scripts"
than others. I use some "platform testing routine", that involves looking at
the "make check" output and guess what info would be useful to send back.

> With an error I'd like to get various other files such as the
> src/config.h file, the config.status file if it exists, plus the
> test/work directory and also the output of the configure/build/test
> run.

It would be useful to formalize this, so that the platform testers' feedback
becomes more useful to you. For example, Nelson's summary today [1] shows
that he thought it would be useful to report the part of the "make check"
output that contains the failures. I thought — seeing the line
  3 Tests in 3 Categories Failed (See .diff* files in work dir for details)
— that it would be good to do
    cd tests; tar cvfz $platform-failures.tar.gz `find work -name '*.diff*'`
and send that. It turns out that both are insufficient for your analysis.
If you tell us what to return as part of the feedback, we can accommodate it.

Many GNU packages create a 'test-suite.log' file, that contains the list of
failed or skipped tests, plus the log file (stdout+stderr) for each. This
makes it simple: As a tester, I simply attach this test-suite.log (or the
parts I judge relevant).

> the config.status file if it exists

Yes, the config.status file is useful to attach, especially when the
compilation failed, but probably also when the compilation succeeded and
some tests failed.

> the src/config.h file

The src/config.h is redundant IMO, since it is derived from config.status.

> plus the test/work directory

All of it? Or a selected part of it?

> and also the output of the configure/build/test run.

Nelson and I do keep these outputs, as part of our build scripts. If you
want it, we can send it. We only need to know what to send.

> I'm happy to add something to GNU make to collect "useful stuff" if
> there's a standard way that testers would like it.
> 
> Maybe just augmenting the check target to do more?

Yes. Instead of printing
  See .diff* files in work dir for details
it could
  - generate a summary file with details (like test-suite.log, described
    above), and/or
  - generate tarball with useful info to send, and/or
  - (if you prefer receiving mails with several attachments instead of a
    tarball attachment) print a list of files to send.
And ideally, also print the email address for the reports. I sent my reports
to bug-make@; Nelson sent his to platform-testers@.

Assuming what you want is a tarball with
  config.status
  `find tests/work -name '*.diff*'`
  `find tests/work -name '*.diff*' | sed -e 's/\.diff/.log/'`
that's what I'm sending now.

Bruno

[1] https://lists.gnu.org/archive/html/platform-testers/2022-10/msg00005.html

debian91-failures.tar.gz
Description: application/compressed-tar