Re: Unclosed quotes on heredoc mode

[Chet Ramey]

On 11/24/21 10:40 AM, Robert Elz wrote:
>      Date:        Tue, 23 Nov 2021 11:09:51 -0500
>      From:        Chet Ramey <chet.ramey@case.edu>
>      Message-ID:  <3a5f6f3a-aa73-d8ac-46f4-46467d5b398d@case.edu>
>
>    | > I'll run our tests against the newest (released) bash
>    |
>    | OK. However, since, as I said, the devel branch has a completely different
>    | implementation, this is not particularly useful.
>
> OK, then I won't bother ... running the tests is easy (about 5 mins
> elapsed time, after about 1 second setup) but analysing the results to
> separate out the real bugs from the places where bash is just different
> from our shell, and neither is right or wrong (our tests are testing to
> make sure we don't change things by accident) takes longer.

OK, if you do end up building the devel branch, I'd be interested in these
results.

>    | It's the build version: how many times have you built in this build tree?
>
> Once, of course ... why would I ever build it again?

Patches exist. There are vendors who take the original release, apply their
own special-sauce patches, then apply the patches I release as they come
out, as part of their own distribution release process.


>    | Whatever. You do you. Don't be surprised if many of my answers turn out to
>    | be "that's already fixed in the devel branch."
>
> First, thanks to those (several) people who indicated how I could
> fetch the devel code, I might look at that sometime, but in general I
> prefer to wait for the released versions (and then for those to get
> included in NetBSD's pkgsrc, which usually happens quite quickly).

Usually, that's ok. In this instance, where we're discussing a feature
whose implementation is substantially different between the released and
development versions, it's more relevant.


>    | Refer to my previous message about the reading-full-lines strategy.
>
> I have no problem with reading full lines, but whenever a "full line"
> includes a newline token, any pending here docs should be read. 

So the ultimate question is whether or not the act of reading a command
substitution should reset this requirement. That's where we disagree.
The grammar is, at that point, reading a different command.


>    | The devel branch produces
>    |
>    | TRACE: pid 78934: parse_comsub: need_here_doc = 1 after yyparse()?
>    | cat: abc: No such file or directory
>    | cat: def: No such file or directory
>
> That looks much better.

Like I said, it's a conscious choice that is still fluid.

>    | We talked about this. The command substitution starts a new parsing context
>    | to implement the "any valid shell script" part of the standard.
>
> Then we get to whether heredoc data is part of a valid shell script
> in that sense - when there is yet to be a newline token to introduce it.

What does this mean? In all cases, the here-documents are not read until
after a newline token. That's not the issue.

> This is where we started this, the question of which newline is the one
> after which heredoc data starts.   It isn't at all as clear as you make
> it appear to be.

Apparently not.


>    | The netbsd shell appears to be the outlier here. The parser reads the
>    | command substitution so it can parse the entire and-or list before trying
>    | to gather any here-documents.
>
> You cannot possibly really mean that I hope.   That is, in
>
>         cmd1 <<EOF &&
>         data
>         EOF
>                 cmd2
>
> you do agree that "data" is stdin to cmd1, that is, the herdoc data
> appears splat in the middle of the and-or list.   That's certainly the
> way it appears to work (in bash) to me.

There is no command substitution in this example.

> Once again, heredoc gathering has nothing at all to do with the grammar,
> and so obviously not the parser either, beyond it informing the lexer which
> heredocs are pending.

So, again, the question is whether or not input data that is logically
part of the command substitution (it appears between the opening and
closing parentheses) should affect the `outer' command. That's the
question. We have different answers.



>    | The fundamental point of disagreement is what to do if the lexer (after,
>    | presumably, calling the parser recursively) finds that it still has here-
>    | documents to read after reading the end of the command substitution.

It was. Now we have moved away from that and added the `should text in the
command substitution satisfy here-documents outside it?'

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/