Re: Allow digits in identifiers (issue 6493072)

[David Kastrup]

Marc Hohl <address@hidden> writes:

> Am 29.10.2012 11:05, schrieb address@hidden:
>> [...]
>>
>> I can wave around my long-term plans which would allow for just writing
>> \violin1 by allowing arrays of violins (I have something in a branch
>> right now, but without further syntax changes I am working on it is not
>> really fitting seamlessly into LilyPond). But long-term plans are not
>> really a suitable excuse for blocking other developers indefinitely.
> I think it would help everyone quite a lot if you could post your
> future plans/roadmap – just some concise bullet points telling us what you
> want to do.
>
> I trust you (almost) completely in terms of developing lilypond along
> a way that is most fruitful for internal consistency and further external
> enhancements. It is sometimes just hard to see your goals by just
> trying to review your patches and remarks along that way.

Much stuff is in a Schrödinger's cat state: a superposition of
possibilities not necessarily compatible with one another.  The
long-term plans are only really getting more detailed as I work on them.
And of course, they change while I work on them.  I pushed just now a
branch dev/vector to look at: it is basically a design study for doing
procedures-with-setters in LilyPond.  There are parser conflicts (caused
by the weird octave check syntax overlapping with assignment syntax) I
have not yet bothered to address.  The main problem, in my opinion a
deal-killer, is that you need to have a separate function type for each
result type: the example in ly/setters.ly implements vectors of music,
but it turns out that you need separate defining functions for vectors
of every single type, which is unfeasible really.

So the main challenge is to just create a vector, and let LilyPond
decide about its component type only by actually referencing it and
looking at the result.

All the incomprehensible issues/commits saying something like "Eliminate
xxx_IDENTIFIER in the parser" or "reduce closed music expressions" are
leading to the goal of making music functions and variables work in a
type-agnostic manner regarding the lexer tokens and the data
_structures_.  Once that falls into place, I can establish a nice set of
features (like vectors of anything, or getting rid of the - in
-\tweak ...^4).  And once those are established, it will be quite harder
to make proposals that scare me silly.

> And if other developers just knew what you are about to do, they do
> not cross your way (in terms of work!) unintentionally.

It would be absurd to propose that developers have to refrain from any
development that I find unsettling.  Even if we wanted to state such an
interdictum based on some popish infallibility theorem, by exactly the
same theorem the other developers would not have the _means_ to figure
out in advance what would be unsettling me.

And it's not like Keith's proposal about violin.1 is fundamentally
flawed or bad.  It is really clever.  It just does not fit well with the
concepts and constructs it tries to harmonize with on a surface level.

In the "Earthsea Trilogy" of Ursula K. le Guin, there is one master on
Roke, the island of wizards, who is known as the Master Patterner.  As
opposed to the others of the nine masters, nobody really knows what art
he practices or teaches.

In the second volume, "The Tombs of Atuan", there is just a slight hint
from an archmage mending a magic artifact:

    "It will [hold]. I couldn't put a mere mending charm on the Ring of
    Erreth-Akbe, like a village witch mending a kettle. I had to use a
    Patterning, and make it whole. It is whole now as if it had never
    been broken."

In a manner, I feel like I am growing into the position of a "Master
Patterner" in LilyPond's syntax, making its design whole as if it had
never been broken, tying fragmented features into a coherent whole.  So
there is a lot of gut feeling involved here, and not all necessarily
scientifically quantifiable.

I have written something up in a reply to Han-Wen at some point of
time.  Let me take a look:

Thinks I have on my personal agenda:

a) largely simplify structure and working of parser, particularly in
connection with music functions.  (Music) function calls should
basically be resolved in the lexer, resulting in an expression that is
then used in the parser according to its type.  Of course, "basically
resolved in the lexer" implies that the syntax of the function arguments
needs to get resolved in an own copy of the parser.

The C parser has the parser stack in automatic arrays on the stack with
an initial non-trivial size.  There is one array for the location data,
and one for the semantic data and parse stack (also one for token types,
but that one is unimportant).  I have just now committed a patch where
the semantic data is consistently SCM.  I have a patch for a Clean_scm
type (to be used for semantic data) which differs from SCM only in that
uninitialized variables start with SCM_UNDEFINED and end up as
SCM_UNDEFINED after destruction.  This helps with the conservative
garbage collection problem in that exiting and reentering a parser does
not leave the same SCM values in the stack area.  However, it means that
exiting and entering the parser with a stack size of 200 entries or so
has a non-minimal startup and shutdown time even when the parser itself
only has just a bit of trivial work to do.  Also, stack entries don't
get destructed after use, and I am not fond of hand-overwriting them for
every reduction.  Would be possible, however.

So it would seem like I either need to find a way to keep two parsers
running in tandem (only feasible if we are using coroutines or one
parser is a push parser), or the per-call overhead of one parser needs
to go down close to zero, and that means a C++ parser with a
destructor-aware stack implementation, or moving to uninitialized data
for both location and semantic data stack, with all the consequences for
conservative garbage collection.  Or any other implementation of a stack
cooperating with the Scheme garbage collector: now that semantic values
are consistently SCM, one can call scm_gc_mark on the active stack
portion, wherever it may be located in memory.

We do call the garbage collector after exiting each source document (and
its associated parser and the mostly unmaintained automatic stack part),
so there is not much of a carry-over of garbage between jobs.  But while
the parser is still on-stack, namely _within_ a large job, its parse
stack keeps far too much garbage around in the non-maintained stack
areas.  Now that I moved the semantic stack to SCM solely, this problem
will be exacerbated.  So a C++ stack/parser is on the agenda soon.

b) move the management of context and grob properties and context
definitions to GOOPS.  The current system is intransparent and not
usefully extensible and predictable.  See scheme-text-spanner.ly for
code thrashing through internals in order to pretend this is feasible.

c) move engravers and listeners to GOOPS.  Similar reasoning here.
Listeners are basically just callbacks and should be implementable as
such.  Engravers suffer severely from C++' absence of per-subclass
initializers (which GOOPS has), and this is worked around with a complex
C macro framework that is neither Scheme nor C++ and is not usefully
documented or extensible.  As a consequence, Scheme engravers can't be
registered where they are callable by name (like C++ engravers) and
documented in the same manner.

d) move MIDI generation to the same GOOPS-managed system.

e) implement a MusicXML exporting global context modeled with the same
system and hooking into the parser like $defaultmidi/\midi and
$defaultlayout/\layout do.

f) move Music expressions to GOOPS.  Make the Scheme pattern recognition
and transformation systems work on Music expressions.  Use that for
reimplementing display-lily-music in an extensible way.

g) Make it possible to define new Music types including their print
forms, engravers, grobs, and whatever in Scheme.

h) retire while everybody uses the provided facilities to turn LilyPond
into something not making sense rather than being restrained to just
talking about it.

At any rate, there is a lot to do that makes a lot of sense in terms of
turning LilyPond into a collection of modular subsystems where one can
evade the consequences of badly implemented subsystems simply by not
using them.



Ok, that was what I wrote to Han-Wen some months ago.  Obviously, there
is still the very real possibility that I just have to drop out of
LilyPond for financial reasons.  Donations, while impressive regarding
the engagement and endurance of the individuals involved, are basically
stagnating at a level quite below what counts here as "poverty
endangered", I just manage to match expenses with that (which is
impressive), with nothing left for pension funds or personal hardships,
and that's before taxes.  So it is quite likely that at some point of
time, I have to cut my losses and leave for a more conventional job, and
of course I don't want to leave matters in a state nobody else could
hope to work with.

So the degree with which I can sensibly put out long-term directions and
goals is really limited, as I can't vouch to actually bring them to
fruition.

-- 
David Kastrup