Re: significance of whitespace [WAS: LilyPond, Finale and Sibelius]

[Han-Wen Nienhuys]

On Mon, Apr 6, 2009 at 9:23 AM, Jan Nieuwenhuizen
<address@hidden> wrote:
>> as long as I have anything to do with LilyPond, I will veto changes
>> like this that introduce inconsistent whitespace handling in the
>> syntax.
>
> What was the problem with this again?  It is not so much inconsistent
> whitepace, it's the absence or presence of whitespace.  That's
> quite a difference.  We have that already
>
>   c 4 4 == c4 4  != c44  % first ws significance BAD, second GOOD?
>    ^ ^

This is actually a lexicographic difference: the input of lilypond is
a stream of tokens (ints, keywords, strings, etc.), and we have  a
grammar specified in terms of these tokens.

There are also two technical reasons why this might is a bad idea.

When you introduce a change like this, it often means that the parser
has to signal back to the lexer, to interpret

  c4 =>  PITCH=c , INT=4

and with space

  c 4 => PITCH=c, INT_AFTER_SPACE_PITCH=4

this is difficult to get right, since parser actions happen somewhat
unpredictably due to lookahead.

Secondly, if you do this, you now have an extra way
(INT_AFTER_SPACE_PITCH) to denote a number, on top of

 bare_unsigned:
        UNSIGNED {
                        $$ = $1;
        }
        | DIGIT {
                $$ = $1;
        }
        ;

(remember: digits are already special due to constructions such as
c4-5), with all the extra difficulties of constructing a grammar
without ambiguities.

>From a user perspective, I also don't see how this makes sense in
corner cases.  For example:

  c'=, ,4
  c4- 4

(note space between commas and after - ) - is this a
INT_AFTER_SPACE_PITCH? In this case, probably not, right?  (if it
would, it would be a parse error,  since comma on itself does not mean
anything.)

So, INT_AFTER_SPACE_PITCH is a number preceded by a space.

Great!  Now we have to keep track of all spaces in the lexer, to make
sure we catch all of the INT_AFTER_SPACE_PITCHes.

Great!  now \tempo 4 = 120 is actually taking a INT_AFTER_SPACE_PITCH
in its 3rd argument, just like c4 * 5 (the 5 is
INT_AFTER_SPACE_PITCH).  Good luck with not creating ambiguities.

Remember how much fun TeX is, where you have to cover up newlines in
definitions with %  (but only half of the time?) so you do not create
accidental spaces in macro expansions?  This is the same thing, and I
think it is a bad idea.

For crying out loud, why is this so difficult to understand?  Syntax
changes may be inspired by "would it not be nice if .. ", but this
mindset must then be discarded immediately to look whether you can
pull it off in the parser/lexer consistently.  Then, after introducing
it, run a 100 pieces of mutopia through the new lexer, and check if
they are not breaking in unobvious ways.

The lilypond syntax is already way more convoluted than is good for
it, and especially if there is an easy way to avoid the problem (using
& or similar), I see no reason to make it even more convoluted.

> I'm quite certain there is a good argument for not not making such
> changes--in fact I remember proposing this change about a decade
> ago and you talking me out of it :-) --but I don't remember
> the reason.
>
> It would be good for the archives too: a good reason
> may even hold it back should the veto ever fail ;-)
>



-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen