[RFC] Adding threads to Eshell

[Jim Porter]

First, a bit of context. For those who don't know, Eshell has an 
"iterative evaluation" feature, where any time you run an asynchronous 
command (like an external process), it yields execution back to the rest 
of Emacs so you're not stuck waiting for a long-running command. This is 
handled by having Eshell evaluate Lisp forms step-by-step (see 
'eshell-do-eval' in lisp/eshell/esh-cmd.el). This requires a lot of code 
to handle various Lisp special forms, and the allowed forms are fairly 
limited. It also has a few bugs, likely due in part to being written 
before lexical binding.

Because of this (especially the bugs), I think it would be good to 
retire 'eshell-do-eval' and switch to something else. Initially, I filed 
bug#57635 to switch over to generator.el's machinery, but then I 
realized that Emacs threads might work even better. In addition to 
fixing the bugs (which would unlock several new features), I think 
threads would make it pretty straightforward to add job control to 
Eshell. I did a few quick tests, and found that it only took a bit of 
work to get the basics working with threads.

However, before I go too far, I wanted to check with other, more 
knowledgeable people: are Emacs threads available on all platforms? It 
looks like it requires the pthreads library (though my understanding was 
that Emacs threads aren't "real" threads, since there's no good way to 
avoid data races with globals). If we want to support asynchronous 
execution in Eshell on systems that don't have access to Emacs threads, 
that might be a problem. On the other hand, if those systems *also* lack 
support for asynchronous processes, then we're probably ok: on those 
systems, Eshell already does most execution synchronously anyway.

A second issue I noticed is that Emacs threads have their own, 
completely separate set of lexical bindings. Is it possible to tell a 
thread that I want to inherit the bindings from wherever I called 
'make-thread'? I might be able to avoid this issue, but it would be nice 
to be able to let-bind some defvars and then pass those bindings to an 
Eshell command to do its thing.

Depending on how people respond, I could go with either Emacs threads or 
some abstraction of generator.el's machinery to upgrade Eshell's 
iterative evaluation. Whichever way I go, I'm planning on making a 
feature branch once I have something ready for others to look at, since 
it's shaping up to be a pretty big change.