[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
type errors, command length limits, and Awk (was: portability of xargs)
From: |
Jacob Bachmeyer |
Subject: |
type errors, command length limits, and Awk (was: portability of xargs) |
Date: |
Tue, 15 Feb 2022 21:17:17 -0600 |
User-agent: |
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 |
Mike Frysinger wrote:
context: https://bugs.gnu.org/53340
Looking at the highlighted line in the context:
> echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
It seems that the problem is that am__base_list expects ListOf/File (and
produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob.
This works in the usual case because the shell implicitly converts Glob
-> ListOf/File and implicitly flattens argument lists, but results in
the overall command line being longer than expected if the globs expand
to more filenames than expected, as described there.
It seems that the proper solution to the problem at hand is to have
am__pep3147_tweak expand globs itself somehow and thus provide
ListOf/File as am__base_list expects.
Do I misunderstand? Is there some other use for xargs?
I note that the current version of standards.texi also allows configure
and make rules to use awk(1); could that be useful here instead? (see below)
[...]
automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build. it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length. it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.
First, I thought that GNU-ish systems were not supposed to have such
arbitrary limits, and this issue (the context) originated from Gentoo
GNU/Linux. Is this a more fundamental bug in Gentoo or still an issue
because Automake build scripts are supposed to be portable to foreign
system that do have those limits?
Second, counting files in the list, as you note, does not necessarily
actually conform to the system limits, while Awk can track both number
of elements in the list and the length of the list as a string, allowing
to break the list to meet both command tail length limits (on Windows or
total size of block to transfer with execve on POSIX) and argument count
limits (length of argv acceptable to execve on POSIX).
POSIX Awk should be fairly widely available, although at least Solaris
10 has a non-POSIX awk in /usr/bin and a POSIX awk in /usr/xpg4/bin; I
found this while working on DejaGnu. I ended up using this test to
ensure that "awk" is suitable:
8<------
# The non-POSIX awk in /usr/bin on Solaris 10 fails this test
if echo | "$awkbin" '1 && 1 {exit 0}' > /dev/null 2>&1 ; then
have_awk=true
else
have_awk=false
fi
8<------
Another "gotcha" with Solaris 10 /usr/bin/awk is that it will accept
"--version" as a valid Awk program, so if you use that to test whether
"awk" is GNU Awk, you must redirect input from /dev/null or it will hang.
Automake may want to do more extensive testing to find a suitable Awk;
the above went into a script that remains generic when installed and so
must run its tests every time the user invokes it, so "quick" was a high
priority.
-- Jacob
- Re: portability of xargs, (continued)
Re: portability of xargs, Jan Engelhardt, 2022/02/15
Re: portability of xargs, Nick Bowler, 2022/02/15
type errors, command length limits, and Awk (was: portability of xargs),
Jacob Bachmeyer <=