bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Kévin Le Gouguec]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> 
>>> In June we changed the way we store immediate objects in the shared and
>>> this makes the compilation way lighter on the GCC side (both in time and
>>> memory).  I've no precise data on this other than the experimental
>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>> each compilation is always < 2GB.
>>
>> You assume that the compilations are all done serially?  AFAIK, most
>> people build Emacs with "make -jN", so parallel compilation is an
>> important use case.
>
>> I guess we will have to collect the information about that, if you say
>> we don't have it now.
>
> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
> produce quite accurate results on memory footprint during the
> compilation process.  Perhaps he has time and he's so kind to gather
> some data on the current state, that would be extremely helpful.

See also bug#41194#20 and bug#41194#28 where I outlined how the
improvements reduced compilation time and memory usage.

I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
in need of… something (probably exorcism, given the noise).

Until I figure that out, here are the (very hacky) scripts I used to
measure and plot the RAM usage, in case someone else wants to take some
measurements:

- ./monitor.sh $PID finds the most RAM-consuming process among $PID and
  its children, and logs its memory usage (VSZ and RSS) and its
  command-line.

  (Logs are collected every 10 seconds; this probably needs to be
  reduced for faster machines)

- ./plot.py uses matplotlib to make graphs out of these measurements; it
  attempts to replace the command line with the less-verbose diagnostics
  from "make".

- My workflow was to start an emacs session, run M-x compile RET make,
  then ./monitor.sh $PID_OF_EMACS_SESSION.

  (PARENT_RE in plot.py should match the command-line of this parent
   session; its RAM consumption is then labeled as "noise floor" on the
   graph.

   This serves no real purpose and should be removed; monitor.sh should
   be amended to filter the parent session out of monitored PIDs, with
   some error control to handle the lack of child processes when
   compilation is finished.)

- There are some hardcoded things to tweak at the bottom of plot.py,
  e.g. how long should a child process last for it to have a label on
  the graph.

monitor.sh
Description: application/shellscript

plot.py
Description: Text Data