bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
>> Date: Sat, 09 Jan 2021 10:55:23 +0000
>> 
>> > What about memory usage when there's a background compilation of Lisp
>> > going on?  GCC is known to be a memory hog in some cases, so I wonder
>> > what happens in this case with libgccjit.
>> 
>> In June we changed the way we store immediate objects in the shared and
>> this makes the compilation way lighter on the GCC side (both in time and
>> memory).  I've no precise data on this other than the experimental
>> observation that compiling all Elisp files in Emacs on 32bit systems is
>> not anymore an issue.  This IIUC implies that the memory footprint for
>> each compilation is always < 2GB.
>
> You assume that the compilations are all done serially?  AFAIK, most
> people build Emacs with "make -jN", so parallel compilation is an
> important use case.

Yeah, we can say this loose information is only a per compilation unit
upper-bound.

> I guess we will have to collect the information about that, if you say
> we don't have it now.

I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
produce quite accurate results on memory footprint during the
compilation process.  Perhaps he has time and he's so kind to gather
some data on the current state, that would be extremely helpful.

>> As a note: in all cases except bootstrap the final pass (the one driving
>> libgccjit) is executed as a sub-process, this to protect us from
>> eventual GCC leaks and not to generate unnecessary fragmentation.  In
>> async compilation we indeed run all the compilation (also the Lisp
>> computation) in the child process, so compiling should not have impact
>> on the memory footprint of the main Emacs session.
>
> That's fine, but the memory footprint of such a subprocess is also of
> interest, as it could be relevant to the overall memory pressure of
> the OS, and thus indirectly on the parent Emacs process as well.

Indeed, I thought was relevant to make you aware of this mechanism.

>> > (Do we allow multiple async compilations, btw? if so, how many
>> > concurrent compilations can be running, and how do we or the user
>> > control that?)
>> 
>> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>
>
> Thanks.  This needs further tuning, IMO, both per the FIXME
> (i.e. provide a primitive to return the number of execution units),
> and wrt the default value being half of the available units.  We
> should pay attention to the system's load average as well, I think.

Agree, I guess we'll be able to tune it better when we'll have more
data.  We could even think of using the memory pressure on the system to
make such a decision.

>> > Also, what are the numbers for a session that has been running for
>> > several days?  I understand that it would be hard for you to collect
>> > such numbers about all the configurations, but could you show the
>> > growth of the configuration you are routinely using, which I presume
>> > is --with-x --with-nativecomp and with your config?  As your numbers
>> > above show, it starts at 1.5 GiB, but what is the footprint after a
>> > day or a week?
>> 
>> ATM I can provide this number, this is an Aarch64 daemon compiled with
>> '--without-x' with an up-time of 25 days and is showing a footprint of
>> 765M.
>
> OK, thanks.
>
>> The hard part is to have a reference to compare against as the memory
>> footprint is strictly connected to the usage.  One with very regular
>> working habits should work like one week on vanilla and one week on
>> native-comp to make a comparison.  I've no regular working habits so I
>> fear I'm not the best fit for this comparison.
>
> I agree, these numbers still need to be collected.  Maybe we should
> ask on emacs-devel that people who use the branch report their
> numbers?

Is a good idea, my fear would be only to have very noisy or hard to
interpret results.  A priori I think would be nice to collect such data
also for master too.

Thanks

  Andrea