Re: Pathname expansion not performed in Here Documents

[Dan Douglas]

On Monday, February 27, 2012 02:07:25 PM Davide Baldini wrote:
> FROM Davide Baldini
> 
> On 02/27/12 04:11, Dan Douglas wrote:
> > "If word is unquoted, all lines of the here-document are subjected to
> > parameter expansion, command substitution, and arithmetic expansion."
> > 
> > No pathname expansion.
> 
> That section of manual doesn't specifically include word splitting nor
> pathname expansion into the list of performed expansions, but the word
> 
> splitting does include itself unconditionally:
> >      Word Splitting
> > 
> > The  shell  scans the results of parameter expansion, command substitu-
> > tion, and arithmetic expansion that did not occur within double  quotes
> > for word splitting.
> 
> and pathname expansion ties itself to word splitting:
> >      Pathname Expansion
> > 
> > After  word  splitting, [...]

Pathname expansion and word splitting are separate, unrelated steps that 
normally occur in an ordinary command evaluation context and other places 
where "words", "arguments", or "parameters" are relevant concepts. 

Word splitting only applies to unquoted expansions resulting from steps higher 
in the expansion hierarchy. Since the body of a here doc ument is basically 
just a string, and in shell land "words" are more structured data, word 
splitting in this context is nonsensical. According to POSIX: "The here-
document shall be treated as a single word".

Pathname expansion is a step that generates words by applying constraints 
using the language of shell pattern matching over the domain of files under a 
given path. Because pathname expansion can only occur in unquoted contexts, It 
happens after word splitting so that files containing characters in IFS are not 
split into separate words. Again, like word splitting, they are only 
applicable to contexts where arguments/parameters/words are relevant.
 
There are a number of genuine issues with this section of the manual that I 
haven't gotten around to addressing on this list yet. This isn't one of them. 

In situations such as this where a feature is specified by POSIX, you should 
also refer to it. In modern times, it's the lowest common denominator upon 
which many shells like Bash attempt to comply. The manpage just briefly 
reiterates what's in the perfectly good spec.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04

Additionally, you can test on other shells that try to emulate the POSIX shell 
or some superset such as Dash, many of the Korn shells, zsh, and (maybe) 
busybox. They should all mostly agree upon fundamental features like this and 
have their own documentation you can cross-reference.

> If intended behaviour is to exclude some expansions from performing word
> splitting or pathname expansion, they should be specifically pointed out

Enumerating which expansions apply to all possible contexts would be a 
monumental task. In general most expansions apply most places, otherwise the 
manual lists those that are applicable.

In this case the dead giveaway should be that quote removal doesn't occur in a 
heredoc which means quoting expansions to disable pathname expansion would be 
impossible. Even if literal quotes required escaping, there's no way it could 
be useful for anything other than breaking things people forgot to quote by 
dumping the meaningless result of a glob into your heredoc.

The more places word splitting and pathname expansion are implicitly disabled, 
the better. There are only a handful of places where pathname expansion is 
apropos, and in Bash, unless writing in a restricted subset for portability, 
word splitting is virtually NEVER desirable. This is why (good) shell 
programming resources so heavily emphesize proper quoting, because in no case 
do you want to allow the possibility of dangerous inadvertant word splitting 
or pathname expansion.  When in doubt, quote, because almost everyplace else 
redundant quotes are removed with few exceptions where they cause side-effects.

If you do for some reason want a glob result in a heredoc, put it into an 
array or the positional parameters and expand that.
-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.