Re: [PATCH + QUESTION] Improve and extend tests `man*.test'.

[Stefano Lattarini]

At Tuesday 13 July 2010, Ralf Wildenhues wrote:
> Hi Stefano,
> 
> * Stefano Lattarini wrote on Mon, Jul 12, 2010 at 06:04:17PM CEST:
> > * tests/man7.test: New test, extracted from old man4.test, which
> > checks for a bug in maintainer-clean w.r.t. generated manpages.
> > Passes with GNU make and Solaris make, fails with BSD make.
> 
> it passes for me with freebsd-make on GNU/Linux, but fails with
> native make on FreeBSD and OpenBSD
> Which make and system do you see the failure on, is it the same?
No, for me it fails also with freebsd-make (from package
freebsd-buildutils 8.0-1) on Debian.  It works OK with GNU
and Heirloom make.

Note that I have amended the patch to separate tests on
maintainer-clean from those on distcheck, and I have modified
the new test `man8.test' a bit to expose ...
> old logs of man4.test
> 
>   + make distcheck
>   [...]
>   /usr/local/bin/bash
> /tmp/build/tests/man4.dir/build/man4-1.0/missing --run help2man
> --output=../bar.1 ../bar help2man: can't unlink ../bar.1
> (Permission denied)
>   *** Error code 1
>   Stop in /tmp/build/tests/man4.dir/build/man4-1.0/_build.
>   *** Error code 1 
>   Stop in /tmp/build/tests/man4.dir/build.
... this bug, which as you said cropped up in old `man4.test'.  The 
amended patch is attached.
 
> Maybe it's time we analyse and fix it?
Well, it seems to me that the failure is not spurious, but it's due to 
a limitation/bug in BSD make.  Now we have a testcase exposing it 
clearly, so the fix (if there's a fix) can be done in another patch 
IMHO.

Thanks,
   Stefano

From 64199ec8aa3c4ed75b7066a3333c10915b47205d Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Mon, 12 Jul 2010 17:37:33 +0200
Subject: [PATCH] Improve and extend tests `man*.test'.

* tests/man.test: Enable `errexit' shell flag, and related changes.
Make grepping of generated Makefile.in slighty stricter.
* tests/man3.test: Add trailing `:' command.
* tests/man5.test: Prefer cat + here-doc over echo to append to
configure.in.
* tests/man2.test: Likewise, and add trailing `:' command.
* tests/man4.test: More thorough and consistent checking of make
error messages.  Move the checks that everything works correctly
when the `help2man' program is installed to ...
* tests/man6.test: ... this new test, and extend them.
* tests/man7.test: New test, extracted from old man4.test, which
checks for a bug in maintainer-clean w.r.t. generated manpages.
* tests/man8.test: New test, extracted from old man4.test, which
checks for a bug in distcheck w.r.t. generated manpages.
Passes with GNU make and Solaris make, fails with BSD make.
---
 ChangeLog       |   19 ++++++++++
 tests/man.test  |   13 +++++--
 tests/man2.test |    8 +++-
 tests/man3.test |    4 ++-
 tests/man4.test |   40 +++++++++++-----------
 tests/man5.test |    6 ++-
 tests/man6.test |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/man7.test |   61 ++++++++++++++++++++++++++++++++++
 tests/man8.test |   58 ++++++++++++++++++++++++++++++++
 9 files changed, 279 insertions(+), 29 deletions(-)
 create mode 100755 tests/man6.test
 create mode 100755 tests/man7.test
 create mode 100755 tests/man8.test

diff --git a/ChangeLog b/ChangeLog
index 4366381..e0630a5 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@
+2010-07-13  Stefano Lattarini  <address@hidden>
+
+       Improve and extend tests `man*.test'.
+       * tests/man.test: Enable `errexit' shell flag, and related changes.
+       Make grepping of generated Makefile.in slighty stricter.
+       * tests/man3.test: Add trailing `:' command.
+       * tests/man5.test: Prefer cat + here-doc over echo to append to
+       configure.in.
+       * tests/man2.test: Likewise, and add trailing `:' command.
+       * tests/man4.test: More thorough and consistent checking of make
+       error messages.  Move the checks that everything works correctly
+       when the `help2man' program is installed to ...
+       * tests/man6.test: ... this new test, and extend them.
+       * tests/man7.test: New test, extracted from old man4.test, which
+       checks for a bug in maintainer-clean w.r.t. generated manpages.
+       * tests/man8.test: New test, extracted from old man4.test, which
+       checks for a bug in distcheck w.r.t. generated manpages.
+       Passes with GNU make and Solaris make, fails with BSD make.
+
 2010-06-26  Ralf Wildenhues  <address@hidden>
 
        Update program --help output to match current GCS.
diff --git a/tests/man.test b/tests/man.test
index 93b3bb2..7037330 100755
--- a/tests/man.test
+++ b/tests/man.test
@@ -1,5 +1,6 @@
 #! /bin/sh
-# Copyright (C) 1996, 1997, 1998, 2001, 2002  Free Software Foundation, Inc.
+# Copyright (C) 1996, 1997, 1998, 2001, 2002, 2010 Free Software
+# Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -18,6 +19,8 @@
 
 . ./defs || Exit 1
 
+set -e
+
 cat > Makefile.am << 'END'
 dist_man_MANS = foo.1
 nodist_man1_MANS = bar.man
@@ -26,6 +29,8 @@ END
 : > foo.1
 : > bar.man
 
-$ACLOCAL || Exit 1
-$AUTOMAKE || Exit 1
-grep '^MANS ' Makefile.in
+$ACLOCAL
+$AUTOMAKE
+grep '^MANS = ' Makefile.in
+
+:
diff --git a/tests/man2.test b/tests/man2.test
index 0027645..cafef02 100755
--- a/tests/man2.test
+++ b/tests/man2.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2002, 2003, 2004  Free Software Foundation, Inc.
+# Copyright (C) 2002, 2003, 2004, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,7 +21,9 @@
 
 set -e
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 man_MANS = foo.2
@@ -49,3 +51,5 @@ DESTDIR="`pwd`/_inst" $MAKE -e uninstall
 test ! -f ./_inst/man/man2/foo.2
 test ! -f ./_inst/man/man4/foo.4
 test ! -f ./_inst/man/man4/bar.4
+
+:
diff --git a/tests/man3.test b/tests/man3.test
index b5afe0e..8b9e3ee 100755
--- a/tests/man3.test
+++ b/tests/man3.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2008  Free Software Foundation, Inc.
+# Copyright (C) 2008, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -42,3 +42,5 @@ $AUTOMAKE
 $AUTOCONF
 ./configure
 DISTCHECK_CONFIGURE_FLAGS=foo=bar $MAKE -e distcheck
+
+:
diff --git a/tests/man4.test b/tests/man4.test
index aa86a9e..8be3135 100755
--- a/tests/man4.test
+++ b/tests/man4.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2008  Free Software Foundation, Inc.
+# Copyright (C) 2008, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -42,9 +42,6 @@ $(srcdir)/foo.1:
        $(HELP2MAN) --output=$@ $(srcdir)/foo
 bar.1:
        $(HELP2MAN) --output=$(srcdir)/bar.1 $(srcdir)/bar
-
-## It is a bug that we need to list $(srcdir)/bar.1 explicitly here.
-MAINTAINERCLEANFILES = $(dist_man_MANS) $(srcdir)/bar.1
 END
 
 cat >>configure.in <<'END'
@@ -72,16 +69,30 @@ exit 127
 END
 
 chmod +x foo bar help2man
-save_PATH=$PATH
+
+# FIXME: not portable
 PATH=`pwd`:$PATH
 
+grep_error_messages()
+{
+  grep ' man pages contain.*missing help2man.* replacement text' stderr \
+   && grep 'install help2man' stderr \
+   && grep 'regenerate the man pages' stderr \
+   || Exit 1
+}
+
 $ACLOCAL
 $AUTOMAKE
 $AUTOCONF
+
 ./configure
 $MAKE
-$MAKE dist && Exit 1
-$MAKE distcheck && Exit 1
+$MAKE dist 2>stderr && { cat stderr >&2; Exit 1; }
+cat stderr >&2
+grep_error_messages
+$MAKE distcheck 2>stderr && { cat stderr >&2; Exit 1; }
+cat stderr >&2
+grep_error_messages
 $MAKE distclean
 
 mkdir build
@@ -90,20 +101,9 @@ cd build
 $MAKE
 $MAKE dist 2>stderr && { cat stderr >&2; Exit 1; }
 cat stderr >&2
-grep 'install help2man' stderr
+grep_error_messages
 $MAKE distcheck 2>stderr && { cat stderr >&2; Exit 1; }
 cat stderr >&2
-grep 'install help2man' stderr
-
-rm -f ../help2man
-PATH=$save_PATH
-export PATH
+grep_error_messages
 
-# If help2man is installed, then ensure that the recommendation works.
-if (help2man --version) >/dev/null 2>&1; then
-  $MAKE maintainer-clean
-  ../configure
-  $MAKE
-  $MAKE distcheck
-fi
 :
diff --git a/tests/man5.test b/tests/man5.test
index 437dc33..f21a099 100755
--- a/tests/man5.test
+++ b/tests/man5.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2009  Free Software Foundation, Inc.
+# Copyright (C) 2009, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,7 +20,9 @@
 
 set -e
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 man_MANS = foo-1.4.5/foo.2 foo-1.4.5/bar.3 baz-1.4.2
diff --git a/tests/man6.test b/tests/man6.test
new file mode 100755
index 0000000..94f59ad
--- /dev/null
+++ b/tests/man6.test
@@ -0,0 +1,99 @@
+#! /bin/sh
+# Copyright (C) 2010 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Ensure `make distcheck' passes when help2man generates man pages,
+# even if the `missing' script is involved.
+
+required=help2man
+. ./defs || Exit 1
+
+set -e
+
+cat > Makefile.am << 'END'
+dist_man_MANS = $(srcdir)/foobar.1 bazquux.1 zardoz.1
+dist_bin_SCRIPTS = foobar bazquux zardoz
+$(srcdir)/foobar.1:
+       $(HELP2MAN) --output=$@ $(srcdir)/foobar
+bazquux.1:
+       $(HELP2MAN) --output=$@ $(srcdir)/bazquux
+zardoz.1:
+       $(HELP2MAN) --output=$(srcdir)/zardoz.1 $(srcdir)/zardoz
+END
+
+cat >> configure.in <<'END'
+AM_MISSING_PROG([HELP2MAN], [help2man])
+AC_OUTPUT
+END
+
+cat > foobar <<'END'
+#! /bin/sh
+while test $# -gt 0; do
+  case $1 in
+    -h | --help) echo "usage: $0 [OPTIONS]..."; exit 0;;
+    -v | --version) echo "$0 1.0"; exit 0;;
+  esac
+  shift
+done
+END
+
+cp foobar bazquux
+cp foobar zardoz
+
+chmod +x foobar bazquux zardoz
+
+$ACLOCAL
+$AUTOMAKE
+$AUTOCONF
+
+mkdir build
+cd build
+
+../configure
+# sanity check
+grep '^HELP2MAN *=.*/missing --run help2man' Makefile
+
+$MAKE
+$FGREP foobar ../foobar.1
+$FGREP bazquux ./bazquux.1
+$FGREP zardoz ../zardoz.1
+
+$MAKE distdir
+$FGREP foobar  $me-1.0/foobar.1
+$FGREP bazquux $me-1.0/bazquux.1
+$FGREP zardoz  $me-1.0/zardoz.1
+
+$MAKE distcheck
+
+cd ..
+rm -f *.1 # remove leftover generated manpages
+
+./configure
+# sanity check
+grep '^HELP2MAN *=.*/missing --run help2man' Makefile
+
+$MAKE
+$FGREP foobar  foobar.1
+$FGREP bazquux bazquux.1
+$FGREP zardoz  zardoz.1
+
+$MAKE distdir
+$FGREP foobar  $me-1.0/foobar.1
+$FGREP bazquux $me-1.0/bazquux.1
+$FGREP zardoz  $me-1.0/zardoz.1
+
+$MAKE distcheck
+
+:
diff --git a/tests/man7.test b/tests/man7.test
new file mode 100755
index 0000000..748b8f8
--- /dev/null
+++ b/tests/man7.test
@@ -0,0 +1,61 @@
+#! /bin/sh
+# Copyright (C) 2010 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check for a bug in maintainer-clean w.r.t. generated manpages.
+
+. ./defs || Exit 1
+
+set -e
+
+cat > Makefile.am << 'END'
+dist_man_MANS = $(srcdir)/foo.1 bar.1
+$(srcdir)/foo.1 bar.1:
+       : > $@
+MAINTAINERCLEANFILES = $(dist_man_MANS)
+END
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+$ACLOCAL
+$AUTOMAKE
+$AUTOCONF
+
+mkdir build
+cd build
+../configure
+
+$MAKE
+test -f bar.1
+test -f ../foo.1
+
+$MAKE maintainer-clean
+test ! -f bar.1
+test ! -f ../foo.1
+
+cd ..
+./configure
+
+$MAKE
+test -f bar.1
+test -f foo.1
+
+$MAKE maintainer-clean
+test ! -f bar.1
+test ! -f foo.1
+
+:
diff --git a/tests/man8.test b/tests/man8.test
new file mode 100755
index 0000000..2ef52b4
--- /dev/null
+++ b/tests/man8.test
@@ -0,0 +1,58 @@
+#! /bin/sh
+# Copyright (C) 2010 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check for a bug in distcheck w.r.t. generated manpages.
+
+. ./defs || Exit 1
+
+set -e
+
+cat > Makefile.am << 'END'
+dist_man_MANS = foo.1
+foo.1:
+       rm -f $(srcdir)/$@  # fails if $(srcdir) is unwritable
+       : > $(srcdir)/$@
+END
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+$ACLOCAL
+$AUTOMAKE
+$AUTOCONF
+
+mkdir build
+cd build
+../configure
+
+$MAKE
+test -f ../foo.1
+$MAKE distdir
+test -f $me-1.0/foo.1
+$MAKE distcheck
+
+cd ..
+rm -f foo.1
+./configure
+
+$MAKE
+test -f foo.1
+$MAKE distdir
+test -f $me-1.0/foo.1
+$MAKE distcheck
+
+:
-- 
1.7.1