Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Andy Wingo <address@hidden> writes:
>>> On Wed 14 Dec 2011 08:50, Mark H Weaver <address@hidden> writes:
>>>> I have successfully implemented the (capture-lexical-environment)
>>>> special form in the evaluator, and also primitive-local-eval.
>
> Let's call it `(the-environment)', as it is what it was called in Guile
> 1.8.

Ah, okay, I didn't know that.  It's also a better name :)

> It is my opinion -- and I could be wrong here, either by
> misunderstanding (again!) the issues, or for whatever reason -- that
> closures are the best solution to the #{#} problem.

I would love to use closures for this, but they simply are not powerful
enough to implement `local-eval' properly.

First of all, closures cannot capture the syntactic environment, so
things like (let-syntax ((foo ...)) (the-environment)) would not work.

Even if you don't use let-syntax, handling macro expansion properly
would be difficult using closures.  As you know, when you expand
(let ((xxx ...)) (the-environment)), the expander replaces `xxx' with a
gensym.  So `local-eval' needs to somehow map `xxx' to the same gensym.

If, on the other hand, you create a closure that instead recognizes the
original names found in the source code, then you will break hygiene and
cause unintended variable capture problems when macros are even _used_
by the code in question.

I think these problems are already serious enough to make the closure
method unworkable, but let's suppose that the macro expander wasn't an
issue.  Even so, there are other problems:

How do you build a closure that can be used to evaluate _arbitrary_
Guile code that is not known until after the surrounding code has
already been compiled and run?

It would apparently involve creating a monster closure that could both
access and mutate any lexical variable within its view (ugly, but
doable), and then -- and here's the really bad part -- we'd need to
create and maintain a `local-eval' that supports the entire Guile
language, but is able to handle some (but not all) of its lexical
variables being accessible only via these monster closures.

IMHO, maintaining this special local evaluator would be a massive
maintenance burden.

Finally, I'd like to argue that supporting `the-environment' and
`local-eval' is really not so bad.  Previously, I thought that
Lilypond's use of Guile was very intrusive, but now that I understand
the situation better, I no longer think so.

Lilypond has no need to inspect nor manipulate our lexical environment
objects.  They only need to be able to pass an opaque environment object
from `the-environment' to `local-eval'.  This still gives us plenty of
freedom to change the representation of these environment objects.

The only serious constraint this places on us is that we must keep a
full-featured interpreter around.  I don't think this is so bad.  I love
optimizing compilers as much as anyone, but sometimes they are the wrong
tool.  When efficiency is not a concern, evaluators can provide
additional flexibility, as demonstrated by the Lilypond case.

Personally, I don't view `the-environment' and `local-eval' as an ugly
hack, but rather as a cool feature like delimited continuations.  It's
something Guile 1.8 could brag about.  I'd like for Guile 2.0 to be able
to brag about it too :)

     Mark