Update on the ‘wip-gash’ branch

[Timothy Sample]

Hi all,

I just pushed a major update to the ‘wip-gash’ branch.  It is rebased on
top of ‘v0.24.2’, and includes 123 commits.  The first 66 commits
(‘v0.24.2..538b0f5b1’) add support for Guile-style modules (these have
been discussed before).  The next 57 add system interfaces and fix bugs
to get Gash working.  In conjunction with Gash's ‘wip-modular-mes’
branch, Gash running on Mes can now stumble through an Autoconf
configure script!  In general, it’s working quite well.

Many thanks to NLnet for funding this work!

The bulk of the changes are new system interfaces to handle the minutiae
of shell scripts: ‘scandir’, ‘environ’, ‘execle’, ‘pipe’, etc.,
etc. (there are quite a few).  Nine of these are hidden from M2 builds
of Mes, and only apply when Mes is built with a more complete C
toolchain (e.g., MesCC and its C library).  There are downsides to this,
but knowing exactly which interfaces would need to be implemented in M2
lets us decide if it’s worth the effort.  We could also move the goal
posts and decide that we don’t need file globbing and skip porting
‘readdir’ to M2, say.

Besides system interfaces, there are many additions to the string and
character set handling procedures (SRFIs 13 and 14).  There’s a copy of
SRFI 11 from Guile.  I added a Scheme-level implementation of
‘dynamic-wind’ that actually winds.  In general, there are little Scheme
interface improvements here and there.

I also fixed (or worked around) a few gnarly bugs:

  • Mes’s ‘syntax-rules’ is not hygienic, so I replaced parts of the
  ‘match’ code with ‘define-macro’ and ‘gensym’ so that it is a little
  better behaved.  To make it work perfectly, the macro system would
  need some really deep fixes.

  • When Autoconf tries to find out how to print, it does a test that
  involves many hundreds of backslashes.  While trying to process all
  these escapes using the usual list processing procedures (‘map’ and
  friends) the Mes Scheme stack would overflow.  To fix this, I rewrote
  many list processing procedures to be iterative instead of recursive.

  • Gash assumes that file descriptors will be managed by the garbage
  collector, but Mes does not do this.  An Autoconf script opens /a lot/
  of files.  More than Mes expects.  So much more, in fact, that Mes
  starts clobbering the beginning of managed memory (the Scheme symbol
  table) as part of it’s I/O buffer handling.  It turns out Mes behaves
  really badly when you overwrite the ‘core:apply’ symbol with junk!  :)
  After a lot of consternation, I added a check to raise ‘EMFILE’ if Mes
  opens too many files.  I worked around how many files the scripts open
  by bumping the limit to 4096 (which is the maximum that my system will
  let me have with ‘ulimit’ by default).  The configure script I was
  testing with (Bash’s) opened more than 1K but less than 2K.

  • I rewrote ‘catch’ to unwind the stack before calling the error
  handler.  This is how Guile behaves, and it prevents infinite loops if
  the error handler itself raises an error.

I think the branch is in good shape.  There are some shims and
workarounds on the Gash side that could be addressed in Mes, but that
can happen later.  There are two commits that I need help with:

  • fc4a5c972 * DRAFT core: Use exceptions instead of asserts.
  • 31071dbb3 * xxx! core: Use 'throw' in 'error' unconditionally.

The first one is from Jan, and I don’t really know what it means by
“DRAFT”.  It’s three years old and part of the original module work.
The second one is from me.  I made it so that Mes would call ‘throw’ in
‘error’ even when compiled with M2 or MesCC.  It seems fine, but I don’t
know why the check was there in the first place.  Jan, I’m hoping you
can bail me out on these two!  :)

If it makes it easier, we could start by merging the first 66 commits.
That is, merge everything up to:

  538b0f5b1 * core: Add a C-only module lookup fast path.

That includes just the module system work.  I tested that the other day,
and was able to build everything “--with-bootstrap” using “boot-5.scm”
as the default.  I had to mark “psyntax.test” as an expected failure,
but all the other tests (the full “make check”) pass.  To be clear, the
‘wip-gash’ branch also passes all tests, but I haven’t tried using
“boot-5.scm” (I see no reason why it wouldn’t work).

That’s all for now.  There are a lot of commits here, and I’m happy to
answer any questions about them.  Let me know how I can help with
review/merging.


-- Tim