Re: Eliminate MARKUP_IDENTIFIER and MARKUPLIST_IDENTIFIER (issue 6561053)

[dak]

Reviewers: janek,

Message:
On 2012/09/28 06:22:40, janek wrote:
> The description looks really nice, but unfortunately i think i don't
grasp the
> consequences of this change.  Could you give some before/after input
examples?

Embarrassingly, not really.  The point of this change is to move forward
in removing the special-casing of various *_IDENTIFIER token types.  The
target is to arrive at only a single SCM_IDENTIFIER type.  Once that is
achieved, it is no longer necessary to evaluate identifiers in the
lexer, removing a source for awkward timing problems like

xxx = { c4 c4 c4 c4 }
\xxx

But as long as there remains more than one category of identifier, this
can't be done.  In a similar vein, it would be nice to be able to use a
scheme function call anywhere you use a markup: if we arrive there,
markup and markup list commands become mostly uninteresting for defining
commands intended to be "top-level" markups, usually called with \markup
\my-command ...  In that case, you can just use \my-command directly
without leading \markup and define it as a scheme function.

At the current point of time, this does not work at most places where
markups are allowed: you can use an identifier there (which basically is
an SCM_IDENTIFIER now) but you can't use a Scheme function as their
argument list parsing is a huge source of parser conflicts.  Once
Scheme/music functions are called somewhat transparently, they can be
used pretty much everywhere, also in and as markup.

This patch is part of preparing for that change, but it does not provide
significant improvements in usability by itself.  The main point is to
cut down on special cases provided by the lexer without _losing_
functionality.  But the actual awards for that will only come after a
number of similar patches.

> Does this have anything to do with what you proposed in "[GLISS] turn
xxx.yyy
> into ("xxx" "yyy")"?

Not really.  It likely helps a bit.  It also begs the question whether
the . syntax should work just for building string lists, or whether it
should be open to building arbitrary markup lists (of which string lists
are a special case).  But that decision does not really require advance
consideration: it will more or less be hardwired into the parser actions
without requiring to be anchored in the grammar itself.

Description:
Eliminate MARKUP_IDENTIFIER and MARKUPLIST_IDENTIFIER

Those and their LYRIC variants are no longer generated by the lexer.
Instead, SCM_IDENTIFIER takes over their function.  One advantage is
that standard Scheme strings and lists of strings (including the empty
list!) are no longer receiving special syntactic status when used as
identifier.

Another is that markups and markup lists can now also be produced and
interpreted by define-scheme-function.  The ultimate goal, of course,
is to have try_special_identifiers return nothing except
SCM_IDENTIFIER.  If that is the case, $... ("immediate Scheme") does
no longer need to get evaluated in the lexer and can be retitled into
"active Scheme".  That would get rid of the Scheme timing problems and
make basically _everything_ doable by define-scheme-function.

This issue is a two-patch issue with the first patch being named:

parser.yy: remove "embedded_scm" as one production for "assignment"

This saved a few lines in the grammar for the price of clarity.

Please review this at http://codereview.appspot.com/6561053/

Affected files:
  M lily/parser.yy