Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org
From: Ken Brown <kbrown@cornell.edu>
Date: Thu, 23 Sep 2021 10:20:19 -0400
Is it really necessary to rebase the *.eln files before each startup?
Isn't it enough to rebase each of the .eln files just once, when it is
produced? If indeed this is needed every time, can you explain why?
We have to distinguish between system libraries and user libraries. Libraries
installed by Cygwin packages are system libraries. The *.eln files in
~/.emacs.d/eln-cache are user libraries. Cygwin maintains a database of all
system libraries and their base addresses. Whenever a Cygwin package is
installed or updated, Cygwin rebases all system libraries and updates the
database.
Each time Emacs starts, it has no way of knowing whether the system libraries
have been rebased since the last time Emacs was run. So the user's *.eln files
could now have base address conflicts with system libraries. Rebasing the *.eln
files fixes this problem.
What do you mean by "system libraries" here? Does it, for example,
include the DLLs distributed in the Cygwin port of libpng or libjpeg?
Or does that include only the basic libraries: the Cygwin DLL, the C
runtime, etc.? >
The non-preloaded *.eln files are all loaded by native-elisp-load, so
I guess the rebase should be launched from there? The preloaded *.eln
files are loaded in pdumper.c:dump_do_dump_relocation, but do we need
to support non-rebased preloaded *.eln files?
The preloaded *.eln files will be installed by Cygwin's package manager when
emacs is installed. They are therefore system libraries and are automatically
rebased as needed.
If the only problem is with non-preloaded *.eln files, why not rebase
them on the fly, when they are loaded. That is, run the 'rebase'
command from the native-elisp-load function, before it actually loads
the file. User libraries are never loaded during startup, only when
some Lisp requires them.
When an updated .eln file is produced for .eln that is loaded into
some running Emacs, Emacs on Windows renames the original .eln to
avoid a similar problem.
I hadn't thought of this issue. We may have to use a similar technique...
Can't you use the same technique, to avoid
the need of rebasing on each start?
...but it wouldn't eliminate the need for rebasing at each start for the reasons
explained above.
Perhaps that need could be eliminated after all, see above.
Please note that users could
place *.eln files in unusual locations (and customize
native-comp-eln-load-path to reflect that), so finding _all_ of the
relevant *.eln files from a shell script might not be easy. In fact,
even without customizing the load-path, I don't think I understand how
will that script you propose know where to find all the *.eln files.
The current proposal that Achim and I are looking at would require each user to
maintain a file ~/.config/rebase/dynpath.d/emacs containing a list of
directories where the .eln files can be found. By default, this file would
contain one line, which is the path to the standard eln-cache directory. Users
who customize native-comp-eln-load-path would have to modify that file
accordingly.
That's tough on users, because Emacs by default automatically compiles
every .el file it loads into .eln, if there's no up-to-date .eln file
already, and the compilation runs in the background (by forking
additional Emacs sub-processes that run in batch mode). In addition,
the native-compilation process sometimes decides that it needs to
create a special "trampoline" .eln file (if you want to know why, I'm
sure Andrea can explain) that correspond to parts of the *.el files.
The upshot is that users may not even be aware that new *.eln files
have been created as part of their session, and may not know their
names. Unless the automatic rebase process, which runs from
native-elisp-load, will also update the file in dynpath.d, I don't see
how users could maintain such a database by hand in practice.
There's one more aspect that you should be aware of. A single file
FOO.el could give birth to several different .eln files, for example
if they are compiled by different Emacs binaries and/or from different
source directories and/or from somewhat different versions of FOO.el.
For example, I now have 3 different versions of .eln files
corresponding to window.el, in the same directory:
window-0d1b8b93-3370bedb.eln
window-0d1b8b93-7d08b7b4.eln
window-0d1b8b93-f8fc9683.eln
This makes the job of maintaining the database by hand even harder and
more error-prone.
Finally, as a side note, I don't think it would be a tragedy if this just turns
out to be too complicated and we have to disable native compilation on 32-bit
Cygwin. The Cygwin home page at https://cygwin.com/ already contains the
following:
Address space is a very limiting factor for Cygwin. These days, a full
32 bit Cygwin distro is not feasible anymore, and will in all likelihood
fail in random places due to an issue with the fork(2) system call.
Therefore we recommend using 32 bit Cygwin only in limited scenarios, with
only a minimum of necessary packages installed, and only if there's no way
to run 64 bit Cygwin instead.
My point is that maybe we should make that decision already, before
burning too much time and energy on it.