Re: argument list too long in project with many files

[Bob Friesenhahn]

On Fri, 30 May 2014, Marco Maggi wrote:
>  I  have read  the manual  section "Length  Limitations" and  an oldish
> mailing list thread[2].  By inspection of the output of "make dist -n" I
> see  only a  single big  list of  files, so  I do  not think  that using
> convenience libraries can solve anything.   Is there a way to circumvent
> this problem other  than splitting the single  Makefile.am into multiple
> Makefile.ams?

It looks like DISTFILES is very large and the rule for 'distdir' uses 
shell code which passes this list to an external utility.  Whether 
this script fails may depend on the exact SHELL used such as which 
commands are shell built-ins (e.g. 'echo') or if it has any arbitrary 
limits.

It seems that the design of the 'distdir' target is fragile since it
tries to accomplish so much in one target.

Possible solutions are to investigate how to make DISTFILES smaller, 
check if there is a different shell which works better, or use the 
dist-hook facility to copy the files while intentionally erasing most 
of the content of DISTFILES.

This is how DISTFILES is defined in Makefile.in:

DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)

You should be able to offer a new definition for this in your 
Makefile.in which removes content (e.g. removes $(DIST_SOURCES)) while 
arranging to copy that content via a suitably designed dist-hook.  A 
suitably designed dist-hook could copy the files in multiple smaller 
steps, or could use an approach which does not consider a list of 
files at all (more risky).

Here is an example of a working dist-hook target which is insensitive 
to the number of files:

# Non-Automake subdirectories to distribute
DISTDIRS = locale scripts www PerlMagick TclMagick
dist-hook:
        ( \
          builddir=`pwd` ; \
          cd $(srcdir) && \
          ( \
            for dir in $(DISTDIRS) ; do \
              find $$dir -depth -print | egrep -v 
'(~$$)|(/\.hg)|(/\.#)|(/\.deps)|(\.pyc)' \
                | cpio -pdum $$builddir/$(distdir) 2> /dev/null ; \
            done \
          ) \
        )

Bob
--
Bob Friesenhahn
address@hidden, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/