Re: The wip-mescc-module branch

[Jan Nieuwenhuizen]

[cc: bug-mes]
Timothy Sample writes:

Hi Tim,

> I just pushed a pretty clean version of wip-mescc-module:
>
>     https://git.ngyro.com/mes/log/?h=wip-mescc-module

Very nice!

> There’s one “DRAFT” commit from you I never bothered to clean up:
>
>     DRAFT core: Use exceptions instead of asserts.
>
> There’s also a few commits that I didn’t keep, but might be useful
> anyway, like “core: Fix FRAME_SIZE conflict.”

Sure, we can look at those later.

> In the end, it’s 66 commits ahead of wip-m2!

Sound good!

> Here’s a few high-level notes about the last round of changes.
>
> I got all the tests to run with ‘MES_BOOT’ set (where appropriate).  The
> psyntax test still fails with the module system.  I’m not quite sure
> why.

That's weird.  MesCC is running.  psyntax support has always been
somewhat troublesome, and also quite slow.  While it would be great to
have syntax-case, I mostly managed to avoid using it.

...later: While I thought that Nyacc was using psyntax, it turns out it
was only being loaded in

    mes/module/nyacc/lalr.mes

...and it turns out that file is not used.  Wow, that explains a lot:
Why Nyacc is not terribly slow and why the flakey psyntax seemed to be
good enough for Nyacc.  I've pushed a patch to remove that file to
wip-mescc-module @ gitlab.

So we can optionally look at psyntax later, maybe we don't need it for
Gash.

> Besides that, all the tests should work on Mes with and without
> the module system, compiled with either GCC or M2.  They should also
> work on Guile, too.

Awesome!

> I never replaced “boot-0” with “boot-5” (as the default).  You have to
> use ‘MES_BOOT’ to get the module system.  Mostly, I like that both work
> and are available.  For MesCC, I have the script check if the module
> system is booted and choose between ‘mes-use-module’ and ‘use-modules’
> that way.

Good.  I was possibly so invested in getting guile-modules to work, that
I opted for removing mes-module support after booting boot-5/boot-6 (as
you have probably seen).  Your choice makes much more sense.  We can
always flip the default and remove mes-modules later, if we wish.

> The performance degradation was unacceptable, too.  Testing indicated
> that MesCC with the module system took nearly 5 times as long to compile
> a file!  Because of this, I added one last commit to avoid Scheme during
> lookups.  With that last commit, modules are only 35% slower than no
> modules.

That's great.  I did some simple test and it seems to be much better
than that.

First, I found something very silly: mescc:main is being run twice, as
easily seen by using -v

    $ ./pre-inst-env mescc -c -v  scaffold/hello.c
    parsing: input
    compiling: input
        :main
    dumping: hello.s
    parsing: input
    compiling: input
        :main
    dumping: hello.s

That was introduced in commit

    6b841a0cab4aa4653f1b7fb4895f4de831f41523
    mescc: Run without shell.

so I fear that there is/was some scenario where mescc:main would not be
run (possibly in the Guix bootstrap).  I haven't looked into this
further but pushed and XXX commit removing this behaviour.  This makes
mescc twice as fast :-P

    ./pre-inst-env mes -c '(display "hello\n")'
real    0m0.186s
    MES_BOOT=boot-5.scm ./pre-inst-env mes -c '(display "hello\n")'
real    0m0.444s

    ./pre-inst-env mescc -c -v scaffold/hello.c
real    0m3.896s
    MES_BOOT=boot-5.scm ./pre-inst-env mescc -c -v scaffold/hello.c
real    0m4.891s

=> 1s (or 33%) slower

    ./pre-inst-env mescc -c -v -I include src/mes.c
real    0m16.239s
    MES_BOOT=boot-5.scm ./pre-inst-env mescc -c -v -I include src/mes.c
real    0m16.988s

=> still only 1s slower

So yeah, not the speed-up that I was hoping to get by using modules
(name lookups are one of the biggest costs, performance wise), but
certainly not bad.  Of course, it would be nice to bring guile-modules
speed up to par with mes-modules, but yeah.
    
> As far as I can tell, the slowdown comes from processing all
> the ‘define-module’ forms.  But that’s just a guess.

Yeah, let's look into that later.

> Variable lookup with the module system booted wasn’t obviously slower
> (when doing everything in C).  I also compared the wip-m2 branch with
> wip-mescc-module (without modules).  The new code is slightly slower:
> the test file took about 15% longer to compile.  I have a guess about
> that, too.  Using ‘<variable>’ records causes everything to get boxed
> twice: once by the binding and once by the variable.  To be honest,
> I’m happy with “only” 15% here.  I hardly gave one thought to
> performance while hacking, and I was worried it would be worse.

Yes, I agree.  Having a much more uniform and guile like interface is
more important.

> Let me know how you want to proceed.  For instance, should I send
> patches to bug-mes?  You could also pull from my repo to a branch
> Savannah or GitLab, and we could have a public discussion on bug-mes
> without posting 66 patches.

I don't think there's need for posting all these patches.  We can do
that for specific patches when one of us has the need to discuss them.

> I know you’re probably still thinking about getting wip-m2 finished.
> There’s no rush.  I’m happy with where this is, and I can move on to
> porting Gash now.

That's great.  Yes, I would like to finish and merge the
wip-full-source-bootstrap branch in Guix based on a mes-0.24 release
from wip-m2.

I propose to keep wip-mescc-module (or any another derived branch) at
least until that time, cleanup, dig up commits from wip-m2 and work
towards a 0.25 release (check it in the guix bootstrap, etc).  It will
most probably need some work to support Gash/Gash-Utils too.  As you
probably have seen, the SRFI support in Mes (e.g. SRFI-1) is quite
minimal: There's only what's needed for running mescc.  One reason for
being minimalistic here was that all bindings lived in a global scope,
and having usused definitions would only slow down mescc (which was
unbearably slow, initially).  I'm hoping that with guile-modules, adding
more of SRFI will not have (such) a negative impact.

Having seen your work on Mes, I'm happy to give you commit access to
savannah.  Maybe that's helpful to give a branch more exposure.

You probably won't be tempted to push to master any time soon, however
if you feel the need please use signed commits (there is no server side
check).

Please also don't push merge commits to master: I like to keep history
linear; this has come in very handy a couple of times bisecting a
problem in the bootstrap.  A regression in mes/mescc may surface even
after building tinycc in the bootstrap.

Many thanks,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com