[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 'Argument list too long' in non-recursive setup

From: Xan Lopez
Subject: Re: 'Argument list too long' in non-recursive setup
Date: Wed, 29 Jul 2009 08:55:18 +0300

On Wed, Jul 29, 2009 at 8:03 AM, Ralf Wildenhues<address@hidden> wrote:
> Hello Xan,
> thanks for the report.  This is an Automake issue more than an Autoconf
> one.

Agh, silly mistake. I googled "autotools mailing list" and got carried
away, didn't stop a minute to look for an automake specific one,

> * Xan Lopez wrote on Tue, Jul 28, 2009 at 09:29:29PM CEST:
>> we are using a non-recursive automake setup at the WebKitGTK+ project
>> (http://www.webkitgtk.org). Today, when adding two new files to the
>> sources list, I started getting this error when doing a 'make
>> distcheck':
>> niraikanai:~/git/WebKit/build/normal%make distcheck
>> { test ! -d "webkit-1.1.12" || { find "webkit-1.1.12" -type d ! -perm
>> -200 -exec chmod u+w {} ';' && rm -fr "webkit-1.1.12"; }; }
>> test -d "webkit-1.1.12" || mkdir "webkit-1.1.12"
>> make: execvp: /bin/sh: Argument list too long
>> make: *** [distdir] Error 127
>> The problem is somewhere in the 'distdir' rule, as the output
>> suggests, where we are passing the whole source files list of WebKit
>> to a command, which overflows the reserved space in Linux for such
>> issues.
> On Linux before 2.6.23(?), the default limit for command line plus
> environment when executing a process is something like 128KB, but this
> can be modified in the kernel (or /proc, I don't remember).  On newer
> Linux kernels, this limit has been lifted to not exist any more.  So one
> workaround would be to build your distribution tarball under a recent
> kernel only.

That's right, this limit can be modified recompiling the kernel (it's
a #define IIRC), although that workaround is not very practical. It's
good to know this is going away in recent kernels though.

>> I'd be happy to hear of any workaround we could apply in our setup to
>> avoid this, but I suspect this needs fixing in the distdir.am file.
> The Automake manual has a section on avoiding long lists for this
> reason,
> <http://www.gnu.org/software/automake/manual/html_node/Length-Limitations.html>
> but in your case that would mean using recursive makefiles (at least
> without changes to distdir.am it would).  I'd still need to think about
> how practical an distdir.am adjustment would be.  At least it would lead
> to some code duplication.

Right. I don't think it's practical for us to change to a recursive
setup as long as there's some kind of workaround the maintainers can
apply to be able to run make distcheck, as that would involve a lot of
work (not to mention we like non-recursive automake better).

>> I've tried to look at it, but couldn't really figure out where it's
>> failing exactly (my lack of shell skills is not helping here...).
> This failure is generaated by the kernel, it fails to start the shell
> command that is that part of the rule.  You can debug such make rules
> with
>  make SHELL=/bin/sh\ -x

That indeed seems to give more output, although in this case it does
not really help much (you still can't see the command that's failing).
I also tried make -d, with similar results. In the end I just went
ahead and looked directly in my Makefile; luckily the distdir rule is
fairly short.

>> There's an odd thing, in that we iterate over the DISTFILES (of the
>> form 'file1 file2 file3...') with for file in list-of-files; do...
>> construct;
> That's likely the culprit.
>> AFAIK that will iterate through the lines in the input,
> No; it does something like
>  for file in $$long_list; do ... done
>  for file in $$long_list; do ... done

Sorry, what I meant with 'input' is that the for will iterate over
each line in the contents of $$long_list, but the file list that's
there does not seem to have lines at all, only space separated
filenames. I'm pretty sure I'm missing something anyway, because this
does indeed work in general, so I'm glad I found the place where it's
failing even if I'm not sure why it's failing.

> for two reasons:
> - this way, the command passed from make contains the long list string
>  only once, in the variable assignment; this helps keep below the limit
>  if possible.
> - if the list happens to be empty, it avoids a shell syntax error that
>  would happen with
>    for file in $(LONG_LIST); ...
> Hope that helps.

I think what would help me most is if you sketched a possible solution
in distdir.am, and I'll try to get that done so we can 'fix' automake
for people with older kernels (2.6.32 isn't likely to be widely
distributed for at least a year or two I'd say). And even if you don't
want to fix it upstream, I think probably asking anyone interested in
running distcheck to apply a small patch to distdir.am is easier than
asking them to rebuild their kernel.

> Man, your package *is* big.  I started the 'svn co' before writing this
> email, and it's still not done.  Speaks for svn slowness, too, but your
> git repo seems unavailable ATM.

Yeah, it has *lots* of tests, that does not help. Fortunately we do
not distribute or do anything with those, so we don't have to add them
in our automake files (otherwise we'd have discovered this problem
from day zero I suspect).


> Cheers,
> Ralf

reply via email to

[Prev in Thread] Current Thread [Next in Thread]