bug#55305: 28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation

[Stefan Monnier]

>> So in order for the compilation to happen correctly, our async workers
>> need to mimic to some extent the currently running Emacs session.
> That was never the way byte-compilation worked in Emacs.

It has never been officially documented, you're right.

But all the files bundled in Emacs are byte-compiled in an Emacs session
where `lisp/loaddefs.el` has already been loaded and many packages rely
on that to minimize the amount of explicit `require` they use (and to
break some cyclic dependencies).

In ELPA packages, that translates into an assumption that the package's
`<pkg>-autoloads.el` has already been loaded (which is indeed the case
during `package-install` and is also the case when compiled via the code
in GNU ELPA's `elpa-admin.el`).

> We have all those 'require' and 'eval-when-compile' things precisely
> so a file can tell the compiler what is needed for the compilation.
> And we _need_ a way to make the compilation be completely independent
> of any local customizations or installed packages.

When a package uses some other package's macro, it necessarily depends on
the locally installed packages to be compiled correctly.

Until now `comp.el` limits the support for "local customizations or
installed packages" to the act of propagating the current session's
`load-path`.  In theory, it could be sufficient.  But this is not the
same as what has been provided for the last ten years when compiling
ELPA packages, so it will inevitably bump into packages for which it
breaks compilation (such as Hyperbole).

I'm not claiming that calling `package-activate-all` is right for
reasons of principle.  We sadly never clearly defined what it is that
a package can count on. In practice ELPA packages have been able to
count on the fact that their autoloads (and their dependencies's
autoloads) have all been loaded (which also implies that all those
packages have been added to the `load-path`).

> I'm firmly against doing this.

OK.

>> AFAIK the only way to make async compilation work reliably is to make it
>> generate the `.eln` file without using the `.el` file (i.e. using the
>> `.elc` file instead, which can be compiled without having to load any
>> user-installed file, expand any macro, or run any user-installed
>> function).  That will also save us from mis-compiling file and from
>> (re)emitting compilation warnings.
> This is unrelated,

I firmly disagree with this, but I agree that it's a side discussion
because we haven't yet found anyone motivated to tackle this problem in
our native-comp pipeline.  So in the mean time we have to paper over the
consequences.


        Stefan