Re: syntax error while parsing a case command within `$(...)'

[Robert Elz]

    Date:        Wed, 17 Feb 2021 23:39:07 +0800
    From:        Koichi Murase <myoga.murase@gmail.com>
    Message-ID:  
<CAFLRLk8a75zHnATwZ+8JuEVrMkcE2H3PB=GxTcHqHaqUmZ_6jg@mail.gmail.com>

Andreas answered the technical part of your message, and I agree
with that.

  | Anyway, even if the POSIX specifies some inconsistent grammar, I feel
  | that cannot be the reason that Bash behaves in a way even more
  | inconsistent than what POSIX specifies.

For a long time it was believed (incorrectly) that () counting & matching
was sufficient to find the end of a $() command substitution - just as the
next unescaped ` finds the end of a `` cmdsub.   Parsing these things is
a bit tricky, as they occur in the middle of a word, which is nominally a 
lexical token (an indivisible unit to the grammar) - and so needs either
fairly horrid tricks, or a parser generator which allows suspending its
current parse and running a new one, so the end of the cmdsub can be
correctly found (for the $() case).   Because that's so hard, and because
() counting was assumed to work, some shells (I believe including bash
initially, though I have never examined its code) used that technique.

That doesn't work however, some kind of parser is required, not just character
scanning for parentheses (and quoting).    And it all has to be recursive,
as $() cmdsubs can occur in the middle of words in the middle of other cndsubs.

Chet can explain more if he feels it is warranted, but I believe that even
today, bash uses a very heuristic type of almost pretend to be a parser
to deal with these things - not really parsing them fully (not until they're
to be executed, where the issues all go away, as at that point, nothing
else is being parsed) but attempting to handle most cases so things work
for real code.

This is why commands that parse file and work properly outside cmdsubs
don't always parse correctly inside (while parsing the command that
contains them) - they're being done by two completely different parsers,
one that is correct, and works, and the other than is just mostly OK.

None of the (current anyway) problems represent code that anyone trying
to solve a real problem (as distinct from stressing the parser) would
ever write, so one can understand a certain reluctance to spending a lot
of time fixing things that only even happen in stress tests.

Really, who ever writes a real case statement with no patterns in it?

kre