Re: Evading the command line length limit (on Linux)

[Ralf Wildenhues]

Hi Paul,

* Paul Smith wrote on Thu, Jul 30, 2009 at 04:19:53PM CEST:
> On Wed, 2009-07-29 at 22:51 +0200, Ralf Wildenhues wrote:
> > 
> > 2) Another possibility would be a hack like the following: split the
> > long line into N substrings, each below the argument limit, and invoke
> > 
> >   sh -c 'eval "${1}${2}...${10}...${N}"' sh quoted-substring1 ...
[...]
> > - ${10} and up are not portable to pre-Posix shells like Solaris sh and
> >   the heirloom shell under Linux

> > I don't think these cause big problems in practice, more so if we go
> > this way only with command line lengths that would otherwise have failed
> > anyway, and only on Linux systems.
> 
> Hrm.  I have to say I don't think I like option #2.  It has "special
> hack" written all over it.

Yep.

> Not only will it only work on Linux but the ${10} thing ties it to bash,
> which means even for Linux-based for systems like Ubuntu where /bin/sh !
> = bash it's not so great.

That is not true.  All remotely Posixy shells that were written after
1990 accept ${10}.  Heirloom sh is special in that, after the SYSV
sources were released under a liberal license, somebody dug out the code
that Steve Bourne wrote and made it compilable under GNU/Linux.  That's
the only Linux shell that doesn't grok ${10}.  All of dash, pdksh,
ksh93, zsh, bash, posh, have no problem with ${10}.  And nobody right in
their mind would use heirloom sh as /bin/sh on GNU/Linux.

Of course, other (proprietary) unices is a different matter altogether,
but that's not what I'm attacking here at all.

> Remember that we actually have this problem in spades on other systems,
> like Windows, where the command line length if far SHORTER than Linux.

I know.  I think what made me try this patch was the little kid in me
that wants to have one developer system without arbitrary limits, but
not as esoteric as plan 9 or as experimental as the Hurd.

Let's face it, there are packages out there that only ever care about
GNU/Linux any more.  GNU Automake is all about portable makefiles and
working around all kinds of limitations (and we'll probably add
some measure for this issue, too), but GNU make is, at least somewhat,
also about unleashing all the power of your system to you.

> I'd prefer to find some solution which solves the problem for everyone,
> rather than adding some pretty obscure special-case situations.

Except of course that the code already effectively has at least two
special-case situations for EMX, w32.

> I'd prefer to use something closer to #1, which is more portable (I
> think).  However, it does break POSIX and could break some makefiles
> that rely on stdin being available.  Maybe the thing to do is require
> the user to specify which rules would be treated in this special way.
> That makes it less automatic, which is too bad, but the length of the
> command line is pretty static so the person writing the makefile should
> know when it's needed.

Well, I guess my point is that, if you require special treatment from
your users anyway, then they can easily use a technique to avoid hitting
this limit.  GNU make provides the means to do so, see for example
<http://thread.gmane.org/gmane.comp.sysutils.autoconf.bugs/6807/focus=6819>.
However, this of course makes your complete makefile unportable to other
makes (as opposed to the original bug report, where with my patch, only
'make dist' would not work with other makes).

> Alternatively if we can find a way to do it automatically in a portable
> way that would be OK too, since the alternative would be to fail anyway.

Yes.  I too would prefer such a solution.

> For Windows and other systems we can probably just choose a hardcoded
> number; for UNIX systems it's more complicated.

For Windows you can likely use @-file support from command.com(?).

> There have been various ideas floating around about allowing make to
> create/use @-files, which a lot of utilities are supporting now.  Maybe
> that is a solution here as well?

Well if the command you're invoking needs this support, then I'd say
this is even less of an automatic solution than anything else.  I guess
we should add @-file support to bash, but even then your argument above
(regarding other shells) applies.

Oh well.  Even if I sound a bit defensive, I won't mind you rejecting
this patch, seeing that you have good reasons, but I do think that it
has some merit.  I'm open to exploring other solutions.

Cheers, and thanks,
Ralf