Re: Multi-line bash strings that end in ! improperly treated as event designator

[Lane Schwartz]

On Mon, Aug 24, 2015 at 8:19 AM, Chet Ramey <chet.ramey@case.edu> wrote:

On 8/21/15 5:17 PM, Lane Schwartz wrote:

> Bash Version: 4.3
> Patch Level: 11
> Release Status: release
>
> Description:
>         Per the Bash Reference Manual, section 9.3.1 "Event Designators", a
> bare exclamation point should be treated as "Start[ing] a history
> substitution, except when followed by a space, tab, the end of line, '=',
> or '('". Bash fails to respect this behavior when a multi-line string ends
> in a bare exclamation point. In such cases, the exclamation point is in
> fact followed by the end of line. Despite this fact, Bash treats the
> exclamation point as the start of a history substitution. In contrast, this
> errant behavior is not observed when a bare exclamation point terminates a
> single-line string.
>
> Repeat-By:
>         $ echo "He didn't fall? Inconceivable!"
>
>         $ echo "He didn't fall?
>> Inconceivable!"

There are a couple of points here.  First, the history expansion mechanism
works a line at a time.  Second, history expansion does not know much of
anything about shell state or syntax, and is performed very early in the
parsing process.

History expansion knows about vaguely Unix-like quoting with double quotes,
single quotes, and backslashes.  It doesn't know about whether or not the
shell is in the middle of reading a multi-line string, and there is no way
to tell it.  It can inhibit history expansion within double-quoted strings
when the opening and closing quotes are on the same line (a relatively
recent addition), but not when they appear on different lines.

The last point is that double quote is not listed as one of the characters
that inhibit history expansion in the manual section you quoted.  Your
example doesn't show the line ending in a `bare exclamation point'; the
exclamation point is followed by a double quote, not a newline.

This is one place where the intersection of history expansion and shell
syntax is not very clean.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/

Chet,

Thanks for taking the time to reply.

You are correct that in my examples the exclamation point is followed by a double quote. However, the fact that the single-line example succeeds without escaping the exclamation point led me to the following interpretation of the manual:

* Section 3.5 "Shell Expansions" specifies seven types of expansions that take place prior to quote removal. History expansion is not one of the listed types of expansions. 

* Section 3.5 states: "After all expansions, quote removal is performed." 

* Section 9.3.1 states that exclamation point is not treated as the start of a history substitution when followed by the end of line.

* History substitution is not performed in the single-line example: echo "He didn't fall? Inconceivable!"

* The only way I could find to reconcile this behavior with the manual was to conclude that Section 3.5 should be interpreted to mean that quote removal takes place after the 7 types of expansions listed there, but before history expansion. This interpretation would mean that at the time history expansion takes place, the exclamation point is in immediately followed by the end of line, because the closing double quote would have already been removed.

In light of your comments, might I suggest two additions to the manual?

* History expansion takes place before quote removal. (This could be added to the bullet list in Section 3.5)

* Beginning with Bash version X.Y.Z, history expansion is suppressed within double-quoted strings, but only when the opening and closing quotes are on the same line. (This could be added to Section 3.1.2.3 and/or Section 9.3)

Thanks,

Lane