Re: GOP2-5 - GLISS discussions

[David Kastrup]

Graham Percival <address@hidden> writes:

> David, after Waltrop I was really enthusiastic about lilypond.  I
> saw the problems that Rudolfo had in trying to make a simple
> change to the website, and I was really fired up to fix the
> problems in the CG.  I was also fired up to organize weekly
> discussions on irc and/or voice.  However, your insulting tone in
> the GLISS emails has completely negated that enthusiasm.
>
> You have told me that I'm not worthy of discussing the ly
> language.

No, I have said that you don't have the expertise, and when I try
providing some, I get shouted down because this is not supposed to be
taken seriously.  _I_ don't have expertise in the backend of LilyPond.
That does not mean that "I'm not worthy" discussing the backend, but
that I simply don't have what it takes to do so.  It does not mean that
I am unable to acquire that knowledge, but before I do so, I can
contribute really little of value, and I certainly am not in a position
to contribute sensibly to a roadmap.  And it certainly won't help when I
tell everybody that _has_ the knowledge to go elsewhere while others
discuss it.

> The *whole point* of having a separate discussion list is to avoid
> upsetting you.  I'm serious about this.  Han-Wen has wisely said that
> he will ignore the emails.  Great, that's what I wanted!  You're now
> the only person taking these discussions in the wrong way.  I never
> expected, nor wanted, to have somebody sitting at my metaphorical
> pot-table complaining that my inspired solution to the monetary crisis
> would never work because the chairman of the european commission
> doesn't have enough power to implement the changes that I wanted to
> make without approval from other committee members.

Graham, you said explicitly that this round of discussions was supposed
to lift the stay on discussions you put on syntax changes previously.

To quote:

<URL:http://lists.gnu.org/archive/html/lilypond-devel/2012-09/msg00049.html>

    The meta-target is "after spending 5 years very publicly telling
    people *not* to talk about changing the syntax because we would do
    so 'in a year or two', I think I should encourage such
    discussions.".  I mean, people trusted me when I said that there
    would be a time for discussing syntax changes.  It was advertized on
    our "help us" page for something like 3 years until Janek commented
    it out in 884194 on 2012-02-14 because "noone knows when GLISS will
    happen".  If I didn't fight to give people an *opportunity* to have
    a meaningful *discussion* about syntax changes before we stabilize
    things, I would be abusing that trust.

There would not have been any need to put discussions on hold in the
first place if the discussions were supposed to be just pipe dreams.
And you explicitly state, both here as well as in your GOP proposal, the
goal of "stabilizing syntax changes" without any further qualifications.

Now the current situation of LilyPond's parser is highly transitory.  I
probably have picked a work area not prone to positive feedback.  When
Mike does a swell job in the backend, 90% of existing scores get better
and 2% change for the worse.  That's progress.  When I do a swell job in
my parser work, 100% of existing scores stay the same.  I am changing
LilyPond's parser from a one-trick dog (or several separate tricks) to
something quite more flexible and universal while keeping its existing
tricks available under the existing commands.  People read about
heavy-handed changes and are glad that apparently they have not happened
yet, and want to keep the status quo.

What isn't visible or appreciated is that I am not really changing
LilyPond's syntax, but changing what it takes to change LilyPond since I
am putting a solid _foundation_ below the existing syntax, meaning that
it becomes easy to extend LilyPond's functionality and vocabulary along
the lines of its _existing_ functionality and vocabulary.

For better or worse, the key to drawing new people working on LilyPond
is the Scheme layer.  You might say "but more people know C++", but C++
is not a language in which you communicate with LilyPond, but rather a
language in which you do brain surgery on it.

And the "C++ parts" of LilyPond are not really written in C++, but in a
concoction of C preprocessing macros, Guile programming and arcane
garbage collection techniques and C++ classes in a fragile mixture.  If
you actually use straightforward C++ techniques like I tried in
<URL:http://code.google.com/p/lilypond/issues/detail?id=2689>, things
just fall apart.  There is a lot of "don't touch this, don't think about
it" in those layers.

phrasing-slur-engraver.cc states close to its top:

/*
  NOTE NOTE NOTE

  This is largely similar to Slur_engraver. Check if fixes
  apply there too.

  (on principle, engravers don't use inheritance for code sharing)

 */

"on principle, engravers don't use inheritance for code sharing" is
actually a euphemism for "if you try using inheritance for code sharing
of engravers, the C preprocessor parts of per-engraver-class
initialization will explode around your ears in totally unfathomable
ways".  I doubt that Han-Wen could violate that "principle" himself in a
reasonable amount of time and effort.

Part of the problems stem from deficiencies in C++ itself: the whole C
preprocessor macro mess is largely papering over the problem that C++
does not offer anything like per-subclass initializations or
per-subclass data.  Other stuff consists of workarounds for the lack of
closures.

So we would not be having a lot of those problems when moving more of
our data and processing to Scheme proper.  I have a lot of work on the
agenda here, but I am currently still mostly tied up with the syntax and
parser stuff.

Now the current GLISS proposals which are slated to become official
policy guiding LilyPond, and the current discussions revolve about the
theme "let's think about what we can now do _to_ LilyPond" rather than
"let's think about what we can now do _with_ LilyPond".  There is no
interest in actually exploring the large strides in extensibility that
LilyPond has made and putting them to best use for the sake of
LilyPond's users.  Instead of making use of the extensibility of
LilyPond, you propose a separately implemented ly++ language to be
converted by preprocessor.

That's what basically most graphical LilyPond frontends do in some
manner, and it leads to a fragmentation of languages, a fragmentation of
tools, a fragmentation of communities and codebases since then the
LilyPond source itself is no longer the preferred form of modification.
It also means that the actions of a ly++ do no longer happen in sync
with the language itself, and that tools like point-and-click and
LilyPond editors like Frescobaldi stop working in a dependable manner.

You can see the effects of this kind of fragmentation in the MusiXTeX
world, with preprocessors like PMX (the MusiXTeX preprocessor MPP
actually was Jan and Hanwen's first approach to typesetting music),
TTRM, and M-Tx.

It has become remarkably easy to extend LilyPond using music functions,
and a large number of LilyPond builtins have been converted to being a
music function (check out ly/music-functions-init.ly for examples).

However, there is no discussion how to extend LilyPond along the lines
that are now easily available.  Instead the discussion focuses about
arbitrary syntactic extensions that provide no additional functionality,
partly based on the illusion "if we create a syntax that we associate
some meaning with in simple cases, the computer will be able to guess as
well as implement our meaning better than when we try following some
system".

And it is not just the discussion, but also the official guidelines for
LilyPond, the GOP, now moves in a direction discouraging working with
the available functionality and tools and discovering shortcomings and
new goals based on the existing state of LilyPond, but rather favors
abandoning future work on LilyPond and instead creating wrappers outside
of LilyPond.

This is slated to become official project policy.  And there is also the
announcement to freeze syntax increasingly, and the expression of
distrust of where I am taking LilyPond with my work (which to a large
degree does not even change the language but rather generalizes it).

Now the problem is that a discussion is only feasible if people know
what they are talking about, or are willing to learn about it.  For
better or worse, the programming layer in LilyPond that has a chance to
become generally accessible, comprehensible and consistent to users and
entry-level programmers is Scheme.  Planning the future of LilyPond
while ignoring the Scheme-accessible layers and tools does not make
sense.  We don't _have_ any other layer which has the exposure,
accessibility and expressivity of Scheme.  LilyPond does not just use
Guile as an extension language but also is architectured around Guile's
memory model and data structures.

Like it or not, Scheme is where the music plays.  I am not responsible
for that design choice, but it is ingrained in LilyPond, and it has what
it takes to make LilyPond go far in regard of power, extensibility and
consistency.  Certainly more so than Emacs Lisp, and look at the
ecosystem of extensions that this editor has been able to sprout.

I have been working hard on making playing around with Scheme not
default to a flaming fiasco until one learns to avoid all the little
peculiarities that don't work for little peculiar technical reasons.

> Now, I imagine that you wouldn't enjoy getting drunk and BSing about
> possible ly language changes.

It would not change much for me to get drunk.  The effect of "just
ignore physical limitations for a while" is more caused by the need of
silencing repeated unreasable user requests by reverting to magic.

In any case, you are right that I might get annoyingly defensive here.
Part of the reason is that the discussion happens to take place ignoring
accomplished work of mine, hampering ongoing work and preventing future
work.  That's certainly not the underlying intention, but that's what it
amounts to, and I don't see that we actually have workable alternatives.
Now Han-Wen's approach is "ignore them all, the coder is the master of
his and their fate".  But that approach does not lend itself to a
feeling of common goals and objectives, and it does not lend itself to
recruiting new coders when the public view of a project's goals are in
discord with the actual work.

I am not doing a fabulous job of helping that impression, no question
about that.

-- 
David Kastrup