Re: [Bug-tar] Re: AMTAR brokenness

[Alexandre Duret-Lutz]

>>> "Paul" == Paul Eggert <address@hidden> writes:

[...]
 Paul> Given all the hassles that will accompany any change, perhaps we
 Paul> should give the maintainer a more gradual upgrade path.

 Paul> For example, we could add an automake macro AM_TAR_FORMAT.
 Paul> AM_TAR_FORMAT([v7]) would use the current behavior (flawed as it is --
 Paul> at least it's something people are familiar with).
 Paul> AM_TAR_FORMAT([ustar]) would attempt to generate ustar format, using
 Paul> the commands "tar --format=ustar", "tar" (if not GNU tar), "pax -x
 Paul> ustar", and "cpio -H ustar", in that order.  Likewise for other
 Paul> formats as we find the need to support them.  
[...]

Sounds great.  Here is a preliminary patch for this.  I have
still to test it more extensively, but that should be enough for
comments.

(BTW the real purpose of tar2.test is to get report from
people on platforms where the pax format is not supported.)

2004-04-18  Alexandre Duret-Lutz  <address@hidden>

        For PR automake/414:
        Introduce options tar-v7, tar-ustar, and tar-pax to select
        tar format.
        * doc/automake.texi (Options): Document them.
        * lib/Automake/Options.pm (_process_option_list): Process
        these new options.
        * lib/am/distdir.am (dist-gzip, dist-bzip2, dist-tarZ, dist,
        distcheck): Adjust to use am__tar and am__untar.
        * m4/tar.m4: New file.
        * m4/Makefile.am (dist_m4data_DATA): Add tar.m4.
        * m4/init.m4 (AM_INIT_AUTOMAKE): Support the new options
        and call _AM_PROG_TAR.
        * tests/tar.test, tests/tar2.test, tests/tar3.test: New files.
        * tests/Makefile.am (TESTS): Add them.

Index: NEWS
===================================================================
RCS file: /cvs/automake/automake/NEWS,v
retrieving revision 1.267
diff -u -r1.267 NEWS
--- NEWS        17 Apr 2004 18:46:05 -0000      1.267
+++ NEWS        18 Apr 2004 21:12:28 -0000
@@ -44,6 +44,9 @@
 
 * Diagnose AC_CONFIG_AUX_DIR calls following AM_INIT_AUTOMAKE. (PR/49)
 
+* Tar format can be chosen with the new options tar-v7, tar-ustar, and
+  tar-pax.
+
 
 New in 1.8:
 
Index: doc/automake.texi
===================================================================
RCS file: /cvs/automake/automake/doc/automake.texi,v
retrieving revision 1.31
diff -u -r1.31 automake.texi
--- doc/automake.texi   2 Apr 2004 07:14:26 -0000       1.31
+++ doc/automake.texi   18 Apr 2004 21:12:34 -0000
@@ -5916,6 +5916,40 @@
 @file{subdir/file.cxx}, then the output file would be
 @file{subdir/file.o}.
 
address@hidden @code{tar-v7}
address@hidden @code{tar-ustar}
address@hidden @code{tar-pax}
+These three mutually exclusive options select the tar format to use
+when generating tarballs with @code{make dist}.  (The tar file created
+is then compressed according to the set of @code{no-dist-gzip},
address@hidden and @code{dist-tarZ} options in use.)
+
+These options must be passed as argument to @code{AM_INIT_AUTOMAKE}
+(@xref{Macros}) because they can causes new configure check to be
+performed.  Automake will complain if it sees such option in a
address@hidden variable.
+
address@hidden selects the old V7 tar format.  This is the historical
+default.  This antiquated format is understood by all tar
+implementations and supports filenames with up to 99 characters. When
+given longer filenames some tar implementations will diagnose the
+problem while other will generate broken tarballs or use non-portable
+extensions.  Furthermore, the V7 format cannot store empty
+directories.
+
address@hidden selects the ustar format defined by POSIX
+1003.1-1988.  This format is believed to be old enough to be portable.
+It fully supports directories, and stores filenames with up to 255
+characters.  However you may run against broken tar implementations
+that incorrectly handle filenames longer than 99 characters (please
+report them to @email{bug-automake@@gnu.org} so we can document this
+accurately).
+
address@hidden selects the new pax format defined by POSIX
+1003.1-2001.  It supports filenames with up to 65535 characters.
+However this format is very young and should probably be avoided in
+all packages that do not target only modern systems.
+
 @item @var{version}
 @cindex Option, version
 A version number (e.g. @samp{0.30}) can be specified.  If Automake is not
Index: lib/Automake/Options.pm
===================================================================
RCS file: /cvs/automake/automake/lib/Automake/Options.pm,v
retrieving revision 1.2
diff -u -r1.2 Options.pm
--- lib/Automake/Options.pm     30 Sep 2003 19:05:54 -0000      1.2
+++ lib/Automake/Options.pm     18 Apr 2004 21:12:35 -0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2004  Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -270,6 +270,22 @@
             || $_ eq 'cygnus' || $_ eq 'no-dependencies')
        {
          # Explicitly recognize these.
+       }
+      elsif ($_ eq 'tar-v7' || $_ eq 'tar-ustar' || $_ eq 'tar-pax')
+       {
+         error ($where,
+                "option `$_' must be an argument of AM_INIT_AUTOMAKE")
+           if $where->get !~ /^configure\./;
+         for my $opt ('tar-v7', 'tar-ustar', 'tar-pax')
+           {
+             next if $opt eq $_;
+             if (exists $options->{$opt})
+               {
+                 error ($where,
+                        "options `$_' and `$opt' are mutually exclusive");
+                 last;
+               }
+           }
        }
       elsif (/^\d+\.\d+(?:\.\d+)?[a-z]?(?:-[A-Za-z0-9]+)?$/)
        {
Index: lib/am/distdir.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/distdir.am,v
retrieving revision 1.56
diff -u -r1.56 distdir.am
--- lib/am/distdir.am   18 Apr 2004 18:36:40 -0000      1.56
+++ lib/am/distdir.am   18 Apr 2004 21:12:35 -0000
@@ -238,19 +238,19 @@
 GZIP_ENV = --best
 .PHONY: dist-gzip
 dist-gzip: distdir
-       $(AMTAR) chof - $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
+       tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c 
>$(distdir).tar.gz
        $(am__remove_distdir)
 
 ?BZIP2?DIST_ARCHIVES += $(distdir).tar.bz2
 .PHONY: dist-bzip2
 dist-bzip2: distdir
-       $(AMTAR) chof - $(distdir) | bzip2 -9 -c >$(distdir).tar.bz2
+       tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
        $(am__remove_distdir)
 
 ?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z
 .PHONY: dist-tarZ
 dist-tarZ: distdir
-       $(AMTAR) chof - $(distdir) | compress -c >$(distdir).tar.Z
+       tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
        $(am__remove_distdir)
 
 ?SHAR?DIST_ARCHIVES += $(distdir).shar.gz
@@ -281,9 +281,9 @@
 
 .PHONY: dist dist-all
 dist dist-all: distdir
-?GZIP? $(AMTAR) chof - $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
-?BZIP2?        $(AMTAR) chof - $(distdir) | bzip2 -9 -c >$(distdir).tar.bz2
-?COMPRESS?     $(AMTAR) chof - $(distdir) | compress -c >$(distdir).tar.Z
+?GZIP? tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c 
>$(distdir).tar.gz
+?BZIP2?        tardir=$(distdir) && $(am__tar) | bzip2 -9 -c 
>$(distdir).tar.bz2
+?COMPRESS?     tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
 ?SHAR? shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz
 ?ZIP?  -rm -f $(distdir).zip
 ?ZIP?  zip -rq $(distdir).zip $(distdir)
@@ -305,11 +305,11 @@
 distcheck: dist
        case '$(DIST_ARCHIVES)' in \
        *.tar.gz*) \
-         GZIP=$(GZIP_ENV) gunzip -c $(distdir).tar.gz | $(AMTAR) xf - ;;\
+         GZIP=$(GZIP_ENV) gunzip -c $(distdir).tar.gz | $(am__untar) ;;\
        *.tar.bz2*) \
-         bunzip2 -c $(distdir).tar.bz2 | $(AMTAR) xf - ;;\
+         bunzip2 -c $(distdir).tar.bz2 | $(am__untar) ;;\
        *.tar.Z*) \
-         uncompress -c $(distdir).tar.Z | $(AMTAR) xf - ;;\
+         uncompress -c $(distdir).tar.Z | $(am__untar) ;;\
        *.shar.gz*) \
          GZIP=$(GZIP_ENV) gunzip -c $(distdir).shar.gz | unshar ;;\
        *.zip*) \
Index: m4/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/m4/Makefile.am,v
retrieving revision 1.52
diff -u -r1.52 Makefile.am
--- m4/Makefile.am      5 Jan 2004 22:21:31 -0000       1.52
+++ m4/Makefile.am      18 Apr 2004 21:12:35 -0000
@@ -53,7 +53,8 @@
 regex.m4 \
 runlog.m4 \
 sanity.m4 \
-strip.m4
+strip.m4 \
+tar.m4
 
 EXTRA_DIST = dirlist amversion.in
 
Index: m4/init.m4
===================================================================
RCS file: /cvs/automake/automake/m4/init.m4,v
retrieving revision 1.58
diff -u -r1.58 init.m4
--- m4/init.m4  10 Nov 2003 20:55:33 -0000      1.58
+++ m4/init.m4  18 Apr 2004 21:12:35 -0000
@@ -3,7 +3,7 @@
 # This macro actually does too much some checks are only needed if
 # your package does certain things.  But this isn't really a big deal.
 
-# Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003
+# Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004
 # Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
@@ -79,7 +79,6 @@
 AM_MISSING_PROG(AUTOMAKE, automake-${am__api_version})
 AM_MISSING_PROG(AUTOHEADER, autoheader)
 AM_MISSING_PROG(MAKEINFO, makeinfo)
-AM_MISSING_PROG(AMTAR, tar)
 AM_PROG_INSTALL_SH
 AM_PROG_INSTALL_STRIP
 AC_REQUIRE([AM_PROG_MKDIR_P])dnl
@@ -88,7 +87,9 @@
 AC_REQUIRE([AC_PROG_AWK])dnl
 AC_REQUIRE([AC_PROG_MAKE_SET])dnl
 AC_REQUIRE([AM_SET_LEADING_DOT])dnl
-
+_AM_IF_OPTION([tar-ustar], [_AM_PROG_TAR([ustar])],
+              [_AM_IF_OPTION([tar-pax], [_AM_PROG_TAR([pax])],
+                            [_AM_PROG_TAR([v7])])])
 _AM_IF_OPTION([no-dependencies],,
 [AC_PROVIDE_IFELSE([AC_PROG_CC],
                   [_AM_DEPENDENCIES(CC)],
Index: m4/tar.m4
===================================================================
RCS file: m4/tar.m4
diff -N m4/tar.m4
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ m4/tar.m4   18 Apr 2004 21:12:35 -0000
@@ -0,0 +1,103 @@
+# Check how to create a tarball.                            -*- Autoconf -*-
+
+# Copyright (C) 2004  Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+
+# serial 1
+
+
+# _AM_PROG_TAR(FORMAT)
+# --------------------
+# Check how to create a tarball in format FORMAT.
+# FORMAT should be one of `v7', `ustar', or `pax'.
+#
+# Substitute a variable $(am__tar) that is a command
+# writing to stdout a FORMAT-tarball containing the directory
+# $tardir.
+#     tardir=directory && $(am__tar) > result.tar
+#
+# Substitute a variable $(am__untar) that exctract such
+# a tarball read from stdin.
+#     $(am__untar) < result.tar
+AC_DEFUN([_AM_PROG_TAR],
+[# Always define AMTAR for backward compatibility.
+AM_MISSING_PROG([AMTAR], [tar])
+m4_if([$1], [v7],
+     [am__tar='${AMTAR} chof - "$$tardir"'; am__untar='${AMTAR} xf -'],
+     [m4_case([$1], [ustar],, [pax],,
+              [m4_fatal([Unknown tar format])])
+AC_MSG_CHECKING([how to create a $1 tar archive])
+# Loop over all known methods to create a tar archive until one works.
+for _am_tool in ${am_cv_prog_tar-gnutar m4_if([$1], [ustar], [plaintar]) pax 
cpio none}
+do
+  case $_am_tool in
+  gnutar)
+    for _am_tar in tar gnutar gtar;
+    do
+      AM_RUN_LOG([$_am_tar --version]) && break
+    done
+    am__tar="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - 
"'"$$tardir"'
+    am__tar_="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - 
"'"$tardir"'
+    am__untar="$_am_tar -xf -"
+    ;;
+  plaintar)
+    # Must skip GNU tar: if it does not support --format= it doesn't create
+    # ustar tarball either.
+    (tar --version) >/dev/null 2>&1 && continue
+    am__tar='tar chf - "$$tardir"'
+    am__tar_='tar chf - "$tardir"'
+    am__untar='tar xf -'
+    ;;
+  pax)
+    am__tar='pax -L -x $1 -w "$$tardir"'
+    am__tar_='pax -L -x $1 -w "$tardir"'
+    am__untar='pax -r'
+    ;;
+  cpio)
+    am__tar='find "$$tardir" -print | cpio -H $1 -L -o'
+    am__tar_='find "$tardir" -print | cpio -H $1 -L -o'
+    am__untar='cpio -i'
+    ;;
+  none)
+    am__tar=false
+    am__tar_=false
+    am__untar=false
+    ;;
+  esac
+
+  # If the value was cached, stop now.  We just wanted to have am__tar
+  # and am__untar set.
+  test -n "${am_cv_prog_tar}" && break
+
+  # tar/untar a dummy directory, and stop if the command works
+  rm -rf conftest.dir
+  mkdir conftest.dir
+  echo GrepMe > conftest.dir/file
+  AM_RUN_LOG([tardir=conftest.dir && eval $am__tar_ >conftest.tar])
+  rm -rf conftest.dir
+  if test -s conftest.tar; then
+    AM_RUN_LOG([$am__untar <conftest.tar])
+    grep GrepMe conftest.dir/file >/dev/null 2>&1 && break
+  fi
+done
+rm -rf conftest.dir
+
+AC_CACHE_VAL([am_cv_prog_tar], [am_cv_prog_tar=$_am_tool])
+AC_MSG_RESULT([$am_cv_prog_tar])])
+AC_SUBST([am__tar])
+AC_SUBST([am__untar])
+]) # _AM_PROG_TAR
Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.556
diff -u -r1.556 Makefile.am
--- tests/Makefile.am   17 Apr 2004 18:46:06 -0000      1.556
+++ tests/Makefile.am   18 Apr 2004 21:12:35 -0000
@@ -484,6 +484,9 @@
 syntax.test \
 tags.test \
 tagsub.test \
+tar.test \
+tar2.test \
+tar3.test \
 target-cflags.test \
 targetclash.test \
 txinfo.test \
Index: tests/Makefile.in
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.in,v
retrieving revision 1.717
diff -u -r1.717 Makefile.in
--- tests/Makefile.in   17 Apr 2004 18:46:06 -0000      1.717
+++ tests/Makefile.in   18 Apr 2004 21:12:35 -0000
@@ -43,7 +43,7 @@
        $(top_srcdir)/m4/missing.m4 $(top_srcdir)/m4/mkdirp.m4 \
        $(top_srcdir)/m4/options.m4 $(top_srcdir)/m4/runlog.m4 \
        $(top_srcdir)/m4/sanity.m4 $(top_srcdir)/m4/strip.m4 \
-       $(top_srcdir)/configure.ac
+       $(top_srcdir)/m4/tar.m4 $(top_srcdir)/configure.ac
 am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \
        $(ACLOCAL_M4)
 mkinstalldirs = $(SHELL) $(top_srcdir)/lib/mkinstalldirs
@@ -91,6 +91,8 @@
 VERSION = @VERSION@
 ac_ct_STRIP = @ac_ct_STRIP@
 am__leading_dot = @am__leading_dot@
+am__tar = @am__tar@
+am__untar = @am__untar@
 bindir = @bindir@
 build = @build@
 build_alias = @build_alias@
@@ -599,6 +601,9 @@
 syntax.test \
 tags.test \
 tagsub.test \
+tar.test \
+tar2.test \
+tar3.test \
 target-cflags.test \
 targetclash.test \
 txinfo.test \
Index: tests/tar.test
===================================================================
RCS file: tests/tar.test
diff -N tests/tar.test
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ tests/tar.test      18 Apr 2004 21:12:36 -0000
@@ -0,0 +1,41 @@
+#! /bin/sh
+# Copyright (C) 2004  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check the tar-ustar option.
+
+. ./defs || exit 1
+
+set -e
+
+cat > configure.in << 'END'
+AC_INIT([tar], [1.0])
+AM_INIT_AUTOMAKE([tar-ustar])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+: > Makefile.am
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+am_cv_prog_tar=cpio ./configure
+$MAKE distcheck
+test -f tar-1.0.tar.gz
Index: tests/tar2.test
===================================================================
RCS file: tests/tar2.test
diff -N tests/tar2.test
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ tests/tar2.test     18 Apr 2004 21:12:36 -0000
@@ -0,0 +1,41 @@
+#! /bin/sh
+# Copyright (C) 2004  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check the tar-pax option.
+
+. ./defs || exit 1
+
+set -e
+
+cat > configure.in << 'END'
+AC_INIT([tar2], [1.0])
+AM_INIT_AUTOMAKE([tar-pax])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+: > Makefile.am
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+./configure
+$MAKE distcheck
+test -f tar2-1.0.tar.gz
Index: tests/tar3.test
===================================================================
RCS file: tests/tar3.test
diff -N tests/tar3.test
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ tests/tar3.test     18 Apr 2004 21:12:36 -0000
@@ -0,0 +1,52 @@
+#! /bin/sh
+# Copyright (C) 2004  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check the tar options diagnostics.
+
+. ./defs || exit 1
+
+set -e
+
+cat > configure.in << 'END'
+AC_INIT([tar2], [1.0])
+AM_INIT_AUTOMAKE([tar-pax tar-v7])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+: > Makefile.am
+
+$ACLOCAL
+AUTOMAKE_fails
+grep 'configure.in:2:.*mutually exclusive' stderr
+
+rm -rf autom4te.cache
+
+cat > configure.in << 'END'
+AC_INIT([tar2], [1.0])
+AM_INIT_AUTOMAKE
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+echo 'AUTOMAKE_OPTIONS = tar-pax' > Makefile.am
+
+AUTOMAKE_fails
+grep 'Makefile.am:1:.*tar-pax.*AM_INIT_AUTOMAKE' stderr


-- 
Alexandre Duret-Lutz