parallel compression in 'make dist' (was: [PATCH] Automake support for ObjC++)

[Ralf Wildenhues]

* Peter Breitenlohner wrote on Wed, Sep 16, 2009 at 10:11:58AM CEST:
> Another completely unrelated item:
> 
> it would be nice if Automake could run gzip and bzip2 (and others) in
> parallel when creating the distribution tarballs.  In a parallel make that
> could save quite some time when running dist(check) for a largish project.
> 
> Feasible?

I think so.  Can you try this patch and check whether it improves `make
dist' time for you significantly?  Thanks.

I did some quick best of three tests on the GraphicsMagick (uses gzip,
bzip2, and lzma) and the Automake packages on a 2-way GNU/Linux system:
               GM       Automake
  before, -j1: 34.28    3.56
  before, -j : 34.03    3.57
  after,  -j1: 34.54    3.65
  after,  -j : 27.11    3.13

indicating that the extra make recursion seems to be acceptable, and the
optimization worthwhile for some packages.  Notably, -j2 isn't helpful
for GM, as it starts the slowest encoder late only.

Further notes:

With the patch below, the compressor pipes can run in parallel.
I do not see how we can avoid the multiple 'tar' invocations without
an intermediate on-disk file.  Leaving it as it is currently, since I
don't know a good way to measure which way the balance tilts on average.

We need to ensure that the individual compression rules don't remove the
distdir after they are done.  We could just set
  am__remove_distdir=:

but that would also cause a small semantic change: currently, the
distdir rule may engage in some sanity checks, then cleans up the
distdir (from eventual old remains), then populates it.  I'd like
the sanity checks to remain happening before the removal of old stuff:
that way, the checks happen early, and if one of them fails, we haven't
otherwise touched the distdir tree at all yet.

Nice side-effect: it lowers Makefile.in size in common cases. :-)

BTW, this is definitely a 1.12-only change: while I'm fairly certain
that it won't cause big backward compatibility issues in practice, the
introduction of another make recursion might expose some variable
override-related oddities.

Thanks,
Ralf

    dist: allow running several compressors in parallel.
    
    * lib/am/distdir.am (am__post_remove_distdir): New internal
    variable.
    (DIST_TARGETS): New variable, set to list of chosen distribution
    formats.
    (dist-gzip, dist-bzip2, dist-lzma, dist-xz, dist-tarZ)
    (dist-shar, dist-zip): Use $(am__post_remove_distdir).
    (dist, dist-all): Do not depend on distdir.  Instead of
    replicating each compression command, use a recursive invocation
    to allow running all $(DIST_TARGETS) in parallel.
    * NEWS: Update.
    Report by Peter Breitenlohner.

diff --git a/NEWS b/NEWS
index 01aa492..97f4aca 100644
--- a/NEWS
+++ b/NEWS
@@ -26,6 +26,8 @@ New in 1.11a:
   - Automake is licensed under GPLv3+ now.  As before, the generated aclocal.m4
     and Makefile.in files carry all-permissive licenses.
 
+  - The `dist' and `dist-all' targets now can run compressors in parallel.
+
 Bugs fixed in 1.11a:
 
 * Bugs introduced by 1.11:
diff --git a/lib/am/distdir.am b/lib/am/distdir.am
index 9664587..e698ec3 100644
--- a/lib/am/distdir.am
+++ b/lib/am/distdir.am
@@ -26,7 +26,7 @@ am__remove_distdir = \
   { test ! -d "$(distdir)" \
     || { find "$(distdir)" -type d ! -perm -200 -exec chmod u+w {} ';' \
          && rm -fr "$(distdir)"; }; }
-
+am__post_remove_distdir = $(am__remove_distdir)
 endif %?TOPDIR_P%
 
 if %?SUBDIRS%
@@ -335,48 +335,55 @@ endif %?TOPDIR_P%
 if %?TOPDIR_P%
 
 ?GZIP?DIST_ARCHIVES += $(distdir).tar.gz
+?GZIP?DIST_TARGETS += dist-gzip
 GZIP_ENV = --best
 .PHONY: dist-gzip
 dist-gzip: distdir
        tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c 
>$(distdir).tar.gz
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?BZIP2?DIST_ARCHIVES += $(distdir).tar.bz2
+?BZIP2?DIST_TARGETS += dist-bzip2
 .PHONY: dist-bzip2
 dist-bzip2: distdir
        tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?LZMA?DIST_ARCHIVES += $(distdir).tar.lzma
+?LZMA?DIST_TARGETS += dist-lzma
 .PHONY: dist-lzma
 dist-lzma: distdir
        tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?XZ?DIST_ARCHIVES += $(distdir).tar.xz
+?XZ?DIST_TARGETS += dist-xz
 .PHONY: dist-xz
 dist-xz: distdir
        tardir=$(distdir) && $(am__tar) | xz -c >$(distdir).tar.xz
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z
+?COMPRESS?DIST_TARGETS += dist-tarZ
 .PHONY: dist-tarZ
 dist-tarZ: distdir
        tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?SHAR?DIST_ARCHIVES += $(distdir).shar.gz
+?SHAR?DIST_TARGETS += dist-shar
 .PHONY: dist-shar
 dist-shar: distdir
        shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 ?ZIP?DIST_ARCHIVES += $(distdir).zip
+?ZIP?DIST_TARGETS += dist-zip
 .PHONY: dist-zip
 dist-zip: distdir
        -rm -f $(distdir).zip
        zip -rq $(distdir).zip $(distdir)
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
 
 endif %?TOPDIR_P%
 
@@ -396,16 +403,9 @@ if %?SUBDIRS%
 AM_RECURSIVE_TARGETS += dist dist-all
 endif %?SUBDIRS%
 
-dist dist-all: distdir
-?GZIP? tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c 
>$(distdir).tar.gz
-?BZIP2?        tardir=$(distdir) && $(am__tar) | bzip2 -9 -c 
>$(distdir).tar.bz2
-?LZMA? tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma
-?XZ?   tardir=$(distdir) && $(am__tar) | xz -c >$(distdir).tar.xz
-?COMPRESS?     tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
-?SHAR? shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz
-?ZIP?  -rm -f $(distdir).zip
-?ZIP?  zip -rq $(distdir).zip $(distdir)
-       $(am__remove_distdir)
+dist dist-all:
+       $(MAKE) $(AM_MAKEFLAGS) $(DIST_TARGETS) am__post_remove_distdir=:
+       $(am__post_remove_distdir)
 
 endif %?TOPDIR_P%
 
@@ -508,7 +508,7 @@ distcheck: dist
 ## Cater to parallel BSD make (see above).
          && cd "$$am__cwd" \
          || exit 1
-       $(am__remove_distdir)
+       $(am__post_remove_distdir)
        @(echo "$(distdir) archives ready for distribution: "; \
          list='$(DIST_ARCHIVES)'; for i in $$list; do echo $$i; done) | \
          sed -e 1h -e 1s/./=/g -e 1p -e 1x -e '$$p' -e '$$x'