bug#55305: 28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation

[Stefan Monnier]

>> Cc: rswgnu@gmail.com, Andrea Corallo <akrl@sdf.com>, 55305@debbugs.gnu.org
>> Date: Sun, 15 May 2022 11:59:05 -0400
>> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> I suspect we should add a call to `package-activate-all` somewhere
>> in the above code (and probably preserve `package-directory-list` and
>> `package-user-dir` as well).
>
> I don't see why comp.el should call package-related functions (or
> indeed know anything about packages and distinguish between packages
> and other Lisp files).  It makes no sense to me.  Compilation should
> not consider user customizations or be dependent on them.

Compiling a `.el` file requires loading files, running macros, and
calling functions, all of which may not come with Emacs and may depend
on the user's specific customizations and set of installed packages
(regardless of whether they're installed via package.el or some other
way).

So in order for the compilation to happen correctly, our async workers
need to mimic to some extent the currently running Emacs session.

The current code only does that to the extent that it preserves the
`load-path`, but this is not always sufficient.

My suggested patch adds the call to `package-activate-all` which is
usually executed in `startup.el` between loading `early-init.el` and
`init.el` and whose intention is to initialize things for the
user-installed packages much like things are unconditionally initialized
for the bundled packages (i.e. enough so it's easy to use them, but
sufficiently little that it doesn't get in the way if the user doesn't
want to use the package).

Just like the current code, my proposed change can fail to do the right
in some circumstances and I don't think there is a way to setup the sync
workers such that they'll always do the right thing.  My proposed patch
just gets a bit closer to reproducing the user's setup in "the usual
case" so I think it's less problematic than what we have.

AFAIK the only way to make async compilation work reliably is to make it
generate the `.eln` file without using the `.el` file (i.e. using the
`.elc` file instead, which can be compiled without having to load any
user-installed file, expand any macro, or run any user-installed
function).  That will also save us from mis-compiling file and from
(re)emitting compilation warnings.

But Someone™ would have to work on that (I can't think of any reason why
it should be difficult, tho it might require a few changes to the .elc
files we generate to preserve a bit more info from the source code, but
it wouldn't require a new .elc format of anything complicated like that).


        Stefan