Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <address@hidden> writes:

> David Kastrup <address@hidden> writes:
>>> In general, the _right_ way to build a custom extension language using
>>> Guile 2 is to write a compiler that converts your language into one of
>>> the other languages that Guile 2 supports.
>>
>> Lilypond is not Scheme.  It has syntax ambiguities that are resolved by
>> lexical tie-ins and thus depend on the context.  You can't easily
>> compile it in advance.
>
> Lexical tie-ins require the use of a context-sensitive parser, but how
> does it prevent compilation in advance?  Guile 2 places no constraints
> whatsoever on the parser used to compile your language to Guile.  You
> could use the exact same Bison parser you are currently using, but with
> different actions.
>
>>> If there's something about Lilypond's language that you believe would
>>> make compilation impractical, let's talk about it.
>>
>> Its syntax and semantics.
>>
>> <URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>
>>
>> If I call a function with an optional argument of type integer? before
>> an argument of type ly:music? and I encounter #x, then the value of x
>> decides whether this argument will be used as the optional argument or
>> as the following argument.  The rest of the parsing has to follow.
>
> I don't see a serious problem here.  In general, anything that can't be
> done at compile time can be postponed to runtime easily enough.

We are running in circles here.  The problem is that I don't get to keep
the lexical environment from compile time for use at runtime.

> To address the specific example you give above, the compiled Lilypond
> procedure could look something like this:

You are putting me on, right?  I explain why Lilypond is not compiled,
and you talk about a "compiled Lilypond procedure".

> (define (myproc . args)
>   (extract-lyargs args `((#:optional ,integer?) (#:required ,ly:music?))
>     (lambda (x music)
>       body ...)))
>
> where `extract-lyargs' is a procedure (part of the runtime environment)
> that takes the list of arguments and a formal-parameter specification,
> and does the runtime tests needed to decide how the arguments should be
> put into `x' and `music'.

Very funny.  If x is not an integer, it is put into the music argument
instead, and the next "argument" is not an argument but independent
code.

>> And you are _totally_ putting the cart before the horse here.
>> Lilypond is not supposed to be an extension language for Guile, but
>> Guile is supposed to be an extension language for Lilypond.  The
>> acronym Guile stands for "GNU's Ubiquitous Intelligent Language for
>> Extension".  You are losing sight of what Guile is supposed to be.
>
> I don't know about that.  You seem to imply that Lilypond's use of
> Guile is very typical,

Not at all.  I imply that Lilypond's use corresponds to what Guile is
advertised as being useful for.  Not all that many people bother using
it in that way, and Guile 2 is taking a definite step backward with
regard to being useful for it.  The "Guile 2 migration project" for
Lilypond is solidly running into man-months of work with no end in
sight.  I have been working around killing capturable lexical
environments (quite more important for an extension language than
capturing continuations), but it is not like this is the only problem.

> and that other programs that use Guile for extension will run into
> similar difficulties, but as far as I can tell Lilypond is quite
> unique here.

Because nobody else uses Guile for serious extensions.  And not because
of its performance: that is _irrelevant_ for most extension purposes.
The performance angle is interesting when one uses Guile as a general
purpose _programming_ language.  You are sacrificing your target
clientele here.

> Typically, an application using libguile (or any other language
> library) allows the library to handle all aspects of parsing and
> running the supported extension language(s).  In Guile's case, the
> idea is that whenever a new language is added to Guile, applications
> using libguile can automatically make use of those new languages.
>
> You are using Guile in a very unusual way.  You have constructed a
> hybrid language of both Scheme and Lilypond,

That's what "extension language" as opposed to "implementation language"
means.

> where each can be nested within the other (so far so good), but -- and
> here's the kicker -- you apparently want to implement this hybrid
> language using two separate interpreters maintained by two separate
> groups that are each able to run code within lexical environments
> established by the other one.

Not really.  We run Guile code inside of lexical environments
established by Guile code, just with a difference in timing.  That is
exactly the same thing as Guile does when using macros.  But there is no
point in turning everything into macros (since we then need to run
primitive-eval for everything), and actually Guile v2 _also_ throws a
wrench into using macros (Ian Hulin has been working on migrating the
use of macros to Guile v2 and it has not exactly been fun and smooth
sailing up to now).

> This is a fundamentally bad idea, because this structure makes it
> impossible for either of these language implementations to evolve in
> any significant way.  It forces them both to remain simple
> interpreters.

We are talking about the syntactic front end here.  A miniscule amount
of the runtime is actually spent in it.  You are arguing for turning a
convenient user environment for processing an input language that can
make good and logical use of Scheme into a complex compiled mess for the
sake of hypothetical performance gains.  This is O(n) for n being the
size of the input.  It is irrelevant for the performance of the system.

>> You are working from the premise that Guile should govern the
>> architecture of the system it is supposed to be extending.
>
> I can understand why it appears that way to you, but this is only
> because you have built a system on Guile that places unreasonable
> constraints upon the internal workings of Guile.

Again, you are putting the cart before the horse.  If an "extension
language" dictates the structure and syntax of the system it is supposed
to extend, it is no longer doing the job of an extension.

> Please try to look at it from our perspective, and also from the
> perspective of other programs that use Guile in a more typical way.

Name a few other programs that use Guile as an _extension_ language
rather than as a _programming_ language.

> Most users of Guile 2 benefit from the architectural and efficiency
> improvements, and are not harmed by them.

You are trying to compete with systems like Chicken, Stalin and C rather
than Tcl, JavaScript and Lua.  But losing by a smaller margin in their
market niche is not going to buy you anything in exchange for ignoring
the needs of your existing users.

So you say we should not be using Guile anymore if we intend to extend
the functionality of Lilypond in a manner where Guile and Lilypond play
smooth and predictably hand in hand.

-- 
David Kastrup