Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> This suggests that for batch jobs maybe we should bump up
>> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.
> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?

Yup.

> Would there be any likely major repercussions

That's the question we should investigate as well.

> -- i.e., jobs that used to run fine would run out of memory?

I added the patch below to try and see a bit more what's going on:

    diff --git a/src/alloc.c b/src/alloc.c
    index 55e18ecd77e..d954e928eed 100644
    --- a/src/alloc.c
    +++ b/src/alloc.c
    @@ -6245,6 +6245,11 @@ garbage_collect (void)
       consing_until_gc = gc_threshold
         = consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0);

    +  fprintf (stderr, "GC-%d p=%.1f total=%.1fM thresold=%.1fM\n",
    +          getpid (), XFLOAT_DATA (Vgc_cons_percentage),
    +          total_bytes_of_live_objects () / 1048576.0,
    +          consing_until_gc / 1048576.0);
    +
       /* Unblock *after* re-setting `consing_until_gc` in case `unblock_input`
          signals an error (see bug#43389).  */
       unblock_input ();

and then I ran

    rm **/*.elc src/bootstrap-emacs lisp/loaddefs.el
    make -j24 BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage 2.0)'

The first thing I see is that with the number of GCs where p=0.1
outweighs those with p=2.0 by a factor 10, so clearly the number of GCs
is significantly reduced.  I also looked for the number of GCs performed
in a given process:

    % grep 'p=2.0' build-log | sed 's/ .*//' | sort | uniq -c
      1 GC-15817
      1 GC-15837
      1 GC-15841
      1 GC-15842
      1 GC-15985
      1 GC-15990
      1 GC-15991
      1 GC-15993
      1 GC-15995
      1 GC-15998
     18 GC-16009
     18 GC-16021
     18 GC-16023
     18 GC-16026
     18 GC-16028
     18 GC-16042
     18 GC-16045
     18 GC-16048
     18 GC-16056
     18 GC-16058
      1 GC-16062
      1 GC-16064
     18 GC-16067
     17 GC-16079

This says that only 24 processes performed a GC with p=2.0, so the
vast majority of the byte-compilation processes finished without
performing any GC at all (well, maybe not quite: see below).

It's also odd how most performed 0 GC, then 12 performed 1 GC, then
1 performed 17 GCs and 11 performed exactly 18 GCs.  What's so magical
about 18?

Then I looked at those that performed 18 GCs and they all look quite similar:

    % grep GC-16026 ./+make-2.0.log
    GC-16026 p=0.1 total=0.5M thresold=0.8M
    GC-16026 p=2.0 total=3.9M thresold=7.8M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M

The first GC seems to happen before we set percentage to 2.0 (so
apparently all compilation processes performed at least one GC before we
set percentage to 2.0 and then the majority of them performed no further
GC before exiting).  [ So if we set percentage to 2.0 a bit earlier than
what happens with `--eval` we may gain yet a bit more time.  ]

9.5M is actually exactly 10000000 bytes so I suspect these come from
either `cedet/semantic.el` or `international/mule-cmds.el` which both
set gc-cons-threshold to (max gc-cons-threshold 10000000).


        Stefan