Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <address@hidden> writes:

>> So this email is partly a:
>>  - What now?  What's the chance of work towards guilemacs moving over to
>>    an official emacs git branch, and that port happening, anytime soon?
>>  - Is anyone running it?  How's it going for you?
>
> Good questions.  I've had the opportunity to think a bit more about
> Emacs Lisp and its possible evolution and I'm still not sure what to
> think about it.
>
> I see a few different options for Emacs Lisp.
>
> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>
> This last point is for me the strongest motivation to try and move to
> some other system, where we could use other people's work.
>
> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient), would give us an FFI, and would give us more manpower
> since we'd be benefiting from the work done on Guile.

That's not all that much manpower.  If you take a look at the commits in
the master branch that are not merges from the stable branch, I think
that more than 90% are from Andy Wingo.  And the turnaround times on the
issue tracker are not all that impressive.  It doesn't help that Andy
does not generally bother with the bug tracker, so most bugs actually
having a chance of getting fixed are those in the stable branch.

> Note that while Guile does come with support for threading,

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14756>

Bug report about basic threading functionality not working, no response
since June 2013.

> it doesn't immediately let us use concurrency in Guile-Emacs, because
> of all the issues of synchronizing access to shared data, with all the
> existing Emacs code (both C and Elisp) assuming that this problem
> doesn't exist.  IOW, language support for concurrency is just a first
> step on the way to letting Emacs Lisp use concurrency.

Bug report about basic list functionality not working for lists of more
than 10009 members:

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>

The proposed fix is rejected, but no alternative fix is done.

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an example
for the inability to compile large expressions on Guile-2.0.  The basic
resume for this bug is "large expressions like that will only work with
Guile-2.2+", but since Guile-2.2 is pretty much a one-person venture, it
is very unclear how it will ever be suitable for a "stable release".

> Another detail that needs to be spelled out is the difference between
> the language and its implementation.  Guile-Emacs provides 2
> languages: Emacs Lips and Scheme (well, it also provides a few more,
> but that's not important).

It might very well become important if it leads to Emacs extensibility
in more procedural-syntax languages like Lua (the data model and basic
application of Emacs does not really call for functional programming, so
a functional programming language has no real advantage).  I see that as
mostly pie-in-the-sky material right now since getting Scheme right and
finding a workable balance between code complexity, performance,
scalability and reliability of the basic feature set seems more
important to focus on before fanning out.  But it is an interesting
long-term perspective.

> One of the main reasons for Emacs's enduring success is its large set
> of third party packages so obviously we can't drop support for Elisp
> any time soon.  And as much as I like Scheme, I'm very much
> unconvinced that it's really so much better that it's worth converting
> packages from Elisp to Scheme.

Scheme is not "better" than Elisp but it's certainly more standardized.
Since it is mostly employed for other tasks, code reusability for Emacs
will not be much of an ongoing advantage in practice.  I mean, how did
the Common Lisp based Emacs clone Hemlock fare?  I think nobody
remembers it any more.

> So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd have
> (old and new) packages that use Elisp, new packages that use Scheme,
> maybe yet other new packages that use, say, Javascript (or some other
> language support by Guile).  That would make the work of Emacs (and
> GNU ELPA) maintenance harder.
>
> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just
> Elisp.

And its dark corners are in areas entirely foreign to text editing.

> So, I think that ideally, we'd want to stick to Elisp, or some
> evolution thereof.  Sadly, I don't see how to evolve Elisp into
> Scheme: they are closely related languages, but the differences are
> large enough that it seems hard to reconcile them.

The main road to a common effort/codebase would be a common VM and byte
code I think.  The Guile compiler is layered, so one could hug the Elisp
byte compiler into a Guile compiler level considerably more remote from
the low-level machine than the VM and basically use all of its
compiler/optimizer structure without prescribing the high-level
language.

I think that this is the basic Emacs-Guile approach and it would make it
reasonably easy to extend Emacs with either Elisp or Scheme without
really having the languages merge in any manner.

I think that the basic idea is quite workable.  What I doubt is that the
GUILE project will readily scale to accommodate Emacs.  We've seen a lot
of growing pains when Emacs jumped on board of Bazaar, a project
unprepared for the challenges of a project of Emacs' size with regard to
both computational and social scalability.

Moving the responsibility for the Emacs low-level language layer and
management (for example, byte-compiled files will be created and managed
on disk by GUILE, and I am not convinced that GUILE is even close to
prepared to deal with a code base and add-on packages on the scale of a
typical Emacs-plus-extensions, plus multiple versions and installations)
into a different project will be a quite larger challenge and rather
permanent commitment than Bazaar was.

Given the current size of GUILE, this would require quite a large leap
of faith for which I see no return jump.

-- 
David Kastrup