Re: The wip-mescc-module branch

[Timothy Sample]

Hey Jan,

Since we’re CC’ing bug-mes, I should mention that this work is funded by
NLnet – thanks NLnet!  :)

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Timothy Sample writes:
>
>> I got all the tests to run with ‘MES_BOOT’ set (where appropriate).  The
>> psyntax test still fails with the module system.  I’m not quite sure
>> why.
>
> That's weird.  MesCC is running.  psyntax support has always been
> somewhat troublesome, and also quite slow.  While it would be great to
> have syntax-case, I mostly managed to avoid using it.
>
> ...later: While I thought that Nyacc was using psyntax, it turns out it
> was only being loaded in
>
>     mes/module/nyacc/lalr.mes
>
> ...and it turns out that file is not used.  Wow, that explains a lot:
> Why Nyacc is not terribly slow and why the flakey psyntax seemed to be
> good enough for Nyacc.  I've pushed a patch to remove that file to
> wip-mescc-module @ gitlab.

That’s actually pretty funny (in a frustrating sort of way)!

> So we can optionally look at psyntax later, maybe we don't need it for
> Gash.

I doubt I did much macrology in Gash, so we’re probably fine.

>> The performance degradation was unacceptable, too.  Testing indicated
>> that MesCC with the module system took nearly 5 times as long to compile
>> a file!  Because of this, I added one last commit to avoid Scheme during
>> lookups.  With that last commit, modules are only 35% slower than no
>> modules.
>
> That's great.  I did some simple test and it seems to be much better
> than that.
>
> First, I found something very silly: mescc:main is being run twice, as
> easily seen by using -v
>
>     $ ./pre-inst-env mescc -c -v  scaffold/hello.c
>     parsing: input
>     compiling: input
>         :main
>     dumping: hello.s
>     parsing: input
>     compiling: input
>         :main
>     dumping: hello.s
>
> That was introduced in commit
>
>     6b841a0cab4aa4653f1b7fb4895f4de831f41523
>     mescc: Run without shell.
>
> so I fear that there is/was some scenario where mescc:main would not be
> run (possibly in the Guix bootstrap).  I haven't looked into this
> further but pushed and XXX commit removing this behaviour.  This makes
> mescc twice as fast :-P

Woohoo!  That’s performance hacking at its finest.  ;)

>     ./pre-inst-env mes -c '(display "hello\n")'
> real  0m0.186s
>     MES_BOOT=boot-5.scm ./pre-inst-env mes -c '(display "hello\n")'
> real  0m0.444s
>
>     ./pre-inst-env mescc -c -v scaffold/hello.c
> real  0m3.896s
>     MES_BOOT=boot-5.scm ./pre-inst-env mescc -c -v scaffold/hello.c
> real  0m4.891s
>
> => 1s (or 33%) slower
>
>     ./pre-inst-env mescc -c -v -I include src/mes.c
> real  0m16.239s
>     MES_BOOT=boot-5.scm ./pre-inst-env mescc -c -v -I include src/mes.c
> real  0m16.988s
>
> => still only 1s slower

Maybe the overhead is mostly constant.  After booting the module system
and processing the fixed number of ‘define-module’ forms (which
apparently takes around a second here), the two versions run at about
the same speed.

> So yeah, not the speed-up that I was hoping to get by using modules
> (name lookups are one of the biggest costs, performance wise), [...]

>From what I saw profiling, I would argue that evaluation itself is the
biggest (all the dispatching maybe?), with lookups and stack management
coming next.  Also (looking at it again as I write this), it seems that
the lookup time is mostly due to lexically-bound identifiers.  The
module system seems to slow down ‘expand_variable_’ a bit, but that
might be because of a poor implementation of ‘module_variable’ (maybe
using the C stack to traverse the module “uses” graph would be faster
than fussing with ‘append2’).  Either way, ISTM that there are more
gains to be made by simplifying ‘eval_apply’ and speeding up non-module
lookups.

> [...] but certainly not bad.  Of course, it would be nice to bring
> guile-modules speed up to par with mes-modules, but yeah.
>
> [...]

>> I know you’re probably still thinking about getting wip-m2 finished.
>> There’s no rush.  I’m happy with where this is, and I can move on to
>> porting Gash now.
>
> That's great.  Yes, I would like to finish and merge the
> wip-full-source-bootstrap branch in Guix based on a mes-0.24 release
> from wip-m2.
>
> I propose to keep wip-mescc-module (or any another derived branch) at
> least until that time, cleanup, dig up commits from wip-m2 and work
> towards a 0.25 release (check it in the guix bootstrap, etc).  It will
> most probably need some work to support Gash/Gash-Utils too.  As you
> probably have seen, the SRFI support in Mes (e.g. SRFI-1) is quite
> minimal: There's only what's needed for running mescc.  One reason for
> being minimalistic here was that all bindings lived in a global scope,
> and having usused definitions would only slow down mescc (which was
> unbearably slow, initially).  I'm hoping that with guile-modules, adding
> more of SRFI will not have (such) a negative impact.

Sounds good!  I got a start porting Gash to Mes last year, so I’m hoping
I can reuse that and have the basics running shortly.

> Having seen your work on Mes, I'm happy to give you commit access to
> savannah.  Maybe that's helpful to give a branch more exposure.

That makes sense.  Having a wip-gash branch on Savannah will make my
work more visible to passers-by, which is always good.  As a complete
fluke, I caught some people discussing Gash and Mes at #bootstrappable,
and I was able to say “I’m working on it!”  I can’t always be on IRC,
though.  :)

> You probably won't be tempted to push to master any time soon, however
> if you feel the need please use signed commits (there is no server side
> check).
>
> Please also don't push merge commits to master: I like to keep history
> linear; this has come in very handy a couple of times bisecting a
> problem in the bootstrap.  A regression in mes/mescc may surface even
> after building tinycc in the bootstrap.

Agreed for signed commits and linear history (which is also my personal
preference, so no trouble there).  Also, you’re right that I probably
won’t be committing to master anyway.


-- Tim