Re: Changes to lexer invalidate documentation

[Rik]

John W. Eaton wrote:
> On 12-Apr-2009, Rik wrote:
>
> | The semicolon is used to indicate an end of line (and please don't print
> | the results of the previous command) or as a new row indicator in a
> | matrix.  The original example is,
> | 
> | functon y = f (x) y = x^2; endfunction
> | 
> | which I would think would parse as 
> |  A ; B
> | where A = 'functon y = f (x) y = x^2'
> | and B = 'endfunction'.
> | 
> | If I follow your lead and type line 'A' individually I get a nice
> | clean error message, but when I follow it with 'endfunction' I get
> | the generic parse error.  Should the parser really have a forward
> | dependency on what lies beyond the semicolon in this case?
>
> The error message comes from the part of Octave that interprets the
> syntax tree generated by the parser.  There is no syntax error in A,
> so the parser has no reason to complain about that.  But there is a
> syntax error in B, and parse errors happen before run-time erorrs, so
> you see the parse error instead of the run-time error even though the
> parse error occurs after the code that will generate the run-time
> error.  This happens because A line like
>
>   A ; B
>
> is parsed into a list of commands and then the list is evaluated, so
> that's why you see the parse error for B before the interpreter has a
> chance to try to evaluate A.
>   
What I imagined is that the sequence of inputs was parsed and as each
logical group of commands was divided the result was placed in a queue. 
After parsing a simple while (!empty) loop grabs the head item from the
queue and passes it to the evaluation engine.  This architecture has a
strict division between parsing, which occurs at one time, and
evaluation, which occurs at a different time.  I was imagining that it
would be simple to intervene when a single block of commands was
available at the head of the queue.  Instead of finishing all of the
parsing the engine would evaluate the first group 'A' and only if that
succeeded would it move to parse B.

The more I think about it the more pathological the example in the
documentation becomes.  First, I find it really rare to enter long
multiple commands on a single line.  I would normally code a function on
the command line like so:

octave:1> functon y = f (x)<CR>
> y = x^2;<CR>
> endfunction<CR>

In this case the first line WOULD be evaluated by itself and produce an
error before I ever got to lines 2 and 3.  Secondly, it has to be just
the right sort of parsing error to pass initial parsing checks but fail
a subsequent one.  Most of the time I have a much simpler typo which the
parser will catch.  As an example, the current parser correctly catches
a failure to close a parenthesis and indicates where.

a = sin(pi; b = cos(pi)<CR>
parse error:

  syntax error

>>> a = sin(pi; b = cos(pi)
              ^

In view of the rarity of the documentation example (and the fact that
I'm changing it to something simpler) it is probably not worth bothering
about.
> I don't see how we can change that behavior.  Imagine typing
>
>   missing_function (); x+++
>
> and having your typing interrupted by an error message about the
> missing function as you started to type "x+++", or typing
>
>   y = rand (10), x = 1
>
> and getting output (the display of Y) before you finished typing "x = 1".
>   
I wasn't thinking of a just-in-time parser that was processing
character-by-character.  The current parser is engaged by the carriage
return character and then works on the entire line.  There is no
intermediate output from one command obscuring another because the
semicolon is specifically there for silencing output.

The current parser already behaves pretty much the way I was envisioning it.

x = 0; x++; missing_function()

will produce an error about missing_function being undefined but the
value of x will be 1.

x = 0; missing_function(); x++

will also produce an error about missing_function but x will be 0.  So
it appears as if the commands are coming out of a queue and that
processing stops as soon as an error is encountered.  My only suggestion
was to limit the queue size between the parsing and evaluation engines.

--Rik