bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
>> Date: Fri, 08 Jan 2021 22:02:38 +0000
>> 
>> I've compiled current native-comp with and without --with-nativecomp
>> repeating the experiment with and without X.  These are the data-points:
>> 
>>   |         | --without-x | --without-x --with-nativecomp |      |
>>   |---------+-------------+-------------------------------+------|
>>   | -Q      | 49M         | 92M                           | 1.9x |
>>   | my-conf | 92M         | 179M                          | 1.9x |
>>   
>>   
>>   |         |      | --with-nativecomp |      |
>>   |---------+------+-------------------+------|
>>   | -Q      | 536M | 756M              | 1.4x |
>>   | my-conf | 585M | 1453M             | 2.4x |
>> 
>> So yes shared are using considerably more memory, I think this is
>> expected as also the file footprint suggests native code is less dense
>> that byte-code (actually with a quite similar relative results).
>
> Thanks for the data points.
>
> What about memory usage when there's a background compilation of Lisp
> going on?  GCC is known to be a memory hog in some cases, so I wonder
> what happens in this case with libgccjit.

In June we changed the way we store immediate objects in the shared and
this makes the compilation way lighter on the GCC side (both in time and
memory).  I've no precise data on this other than the experimental
observation that compiling all Elisp files in Emacs on 32bit systems is
not anymore an issue.  This IIUC implies that the memory footprint for
each compilation is always < 2GB.

As a note: in all cases except bootstrap the final pass (the one driving
libgccjit) is executed as a sub-process, this to protect us from
eventual GCC leaks and not to generate unnecessary fragmentation.  In
async compilation we indeed run all the compilation (also the Lisp
computation) in the child process, so compiling should not have impact
on the memory footprint of the main Emacs session.

> (Do we allow multiple async compilations, btw? if so, how many
> concurrent compilations can be running, and how do we or the user
> control that?)

Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

> Also, what are the numbers for a session that has been running for
> several days?  I understand that it would be hard for you to collect
> such numbers about all the configurations, but could you show the
> growth of the configuration you are routinely using, which I presume
> is --with-x --with-nativecomp and with your config?  As your numbers
> above show, it starts at 1.5 GiB, but what is the footprint after a
> day or a week?

ATM I can provide this number, this is an Aarch64 daemon compiled with
'--without-x' with an up-time of 25 days and is showing a footprint of
765M.

>> Indeed *with use the delta should decay as data are the same and there's
>> no difference in its representation*, so this picture should be more on
>> the worst case side than on the average.
>
> That's why I asked to see the memory footprint after the session has
> run for some time, yes.

The hard part is to have a reference to compare against as the memory
footprint is strictly connected to the usage.  One with very regular
working habits should work like one week on vanilla and one week on
native-comp to make a comparison.  I've no regular working habits so I
fear I'm not the best fit for this comparison.

Thanks

  Andrea