[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

type errors, command length limits, and Awk (was: portability of xargs)

From: Jacob Bachmeyer
Subject: type errors, command length limits, and Awk (was: portability of xargs)
Date: Tue, 15 Feb 2022 21:17:17 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20090807 MultiZilla/ SeaMonkey/1.1.17 Mnenhy/

Mike Frysinger wrote:
Looking at the highlighted line in the context:
>   echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
It seems that the problem is that am__base_list expects ListOf/File (and produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob. This works in the usual case because the shell implicitly converts Glob -> ListOf/File and implicitly flattens argument lists, but results in the overall command line being longer than expected if the globs expand to more filenames than expected, as described there.

It seems that the proper solution to the problem at hand is to have am__pep3147_tweak expand globs itself somehow and thus provide ListOf/File as am__base_list expects.

Do I misunderstand?  Is there some other use for xargs?

I note that the current version of standards.texi also allows configure and make rules to use awk(1); could that be useful here instead? (see below)


automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.

First, I thought that GNU-ish systems were not supposed to have such arbitrary limits, and this issue (the context) originated from Gentoo GNU/Linux. Is this a more fundamental bug in Gentoo or still an issue because Automake build scripts are supposed to be portable to foreign system that do have those limits?

Second, counting files in the list, as you note, does not necessarily actually conform to the system limits, while Awk can track both number of elements in the list and the length of the list as a string, allowing to break the list to meet both command tail length limits (on Windows or total size of block to transfer with execve on POSIX) and argument count limits (length of argv acceptable to execve on POSIX).

POSIX Awk should be fairly widely available, although at least Solaris 10 has a non-POSIX awk in /usr/bin and a POSIX awk in /usr/xpg4/bin; I found this while working on DejaGnu. I ended up using this test to ensure that "awk" is suitable:

# The non-POSIX awk in /usr/bin on Solaris 10 fails this test
if echo | "$awkbin" '1 && 1 {exit 0}' > /dev/null 2>&1 ; then

Another "gotcha" with Solaris 10 /usr/bin/awk is that it will accept "--version" as a valid Awk program, so if you use that to test whether "awk" is GNU Awk, you must redirect input from /dev/null or it will hang.

Automake may want to do more extensive testing to find a suitable Awk; the above went into a script that remains generic when installed and so must run its tests every time the user invokes it, so "quick" was a high priority.

-- Jacob

reply via email to

[Prev in Thread] Current Thread [Next in Thread]