bug#15602: Compiling several files in the same session [2.0.9]

[Ludovic Courtès]

Hi again!

Andy Wingo <address@hidden> skribis:

> On Sun 13 Oct 2013 15:51, address@hidden (Ludovic Courtès) writes:
>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) 
>> (for-each compile-file (list "one.scm" "two.scm" "three.scm"))'
>>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) 
>> (for-each compile-file (list "three.scm" "two.scm" "one.scm"))'
>>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) 
>> (for-each compile-file (list "one.scm" "three.scm" "two.scm"))'
>> Backtrace:
>
> I understand this is now fixed in Guix.

I think it’s worked around, not fixed.  :-)

>> I think the right thing would be to use a separate module hierarchy in
>> the dynamic extent of ‘compile-file’, somehow, such that all module side
>> effects are isolated.
>
> I don't think this is going to happen, for what that's worth :/

What do you mean?

Global state held in global variables is Bad.  To me, this is clearly an
area where things can and should be improved.  In a live-hacked,
long-running system, that ‘compile-file’ modifies the global state in
arbitrary ways is not just a theoretical problem.

WDYT?

>> Of course the above can be worked around by running ‘compile-file’ in a
>> child process, but forking alone is more expensive than ‘compile-file’,
>> so that’s not really a solution when there are many files.
>
> It's the N^2 expansion that's the problem, not the forking.  If you have
> N files which depend on each other, then compiling each one will require
> the expansion of approximately all N files, so N*N costs in number of
> files.  Or N log N if you have a tree ordering of your files.  Anyway
> it's badness.

Right.  However, when compiling a set of files in a single process, one
could expect previously-expanded files to be cached.

> Glad to know that Taylan has fixed this one in Guix.

I should mention that the hack in Guix to allow parallel compilation is
brittle since shared state (module obarrays) can, in some cases, be
accessed concurrently, and modules aren’t thread safe.

Concretely, building all the files in Guix works well ~95% of the time;
when only some files need rebuilding, it’s not uncommon to see it fail
weirdly (“no such language” or a similarly obscure error.)

Any idea how this could be addressed?

Thanks!

Ludo’.