Re: Opportunistic GC

[Pip Cet]

On Wed, Mar 10, 2021 at 8:41 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Wed, 10 Mar 2021 20:21:16 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> >
> > > > > What happens when, on a 8GB machine, we have an Emacs session with a
> > > > > 5GB memory footprint, and you fork and start marking (which triggers
> > > > > copy-on-write)?
> > > >
> > > > That is, indeed, one of the cases in which we have to use the
> > > > synchronous GC or suffer the consequences.
> > >
> > > How do we know if/when we are in that case?  Because if we err, we
> > > have a meeting with the OOM killer and a swift death.
> >
> > Yes, but there's a good chance we would have had that anyway, since
> > it's "only" a factor of two, at most.
>
> I don't see how that helps to get out of the conundrum.  The
> synchronous GC doesn't enlarge the memory footprint of the Emacs
> process, so if we had enough memory when we started GC, there are good
> chances we will have it throughout the GC.  By contrast, when you
> fork, you enlarge the memory pressure in the system, and the amount of
> the increment is unknown in advance (because it's determined by the
> degree of nesting of the Lisp data you have in the session at that
> moment).  You could enlarge the memory enough to cause OOM killer to
> kick in.

All of these problems apply to all forks, not just GC ones.

> So it's important to know when not to attempt the fork.

I don't think "normal" GNU/Linux systems should ever be in that
situation, since they have virtual memory. Exotic systems may be, but
that's a system dependency which, as you correctly say, cannot be
easily resolved by generic code.

And, of course, it's possible even on GNU/Linux system for the fork()
not to succeed, which we need to handle.

> What is the
> algorithm for that?  If you want to use the fraction of the memory the
> Emacs process takes, and set the threshold at 50%, then we now need to
> discuss how to know that fraction reliably, what with some of the
> memory coming from sbrk and some from mmap -- this is why we all but
> eliminated memory-limit as not being very useful, and you now want to
> use the same technique to base a critical decision.

Ideally, we'd use the same (broken) technique as the (broken) OS uses
to determine whether to kill us.

> > If you're arguing that there is a strong case that synchronous GC
> > should remain the default, I'm inclined to agree
>
> No, I'm not arguing that yet, not until I see the implementation.  I'm
> just thinking aloud about some pesky problems we'd need to solve on
> the way.

Thanks for explaining.

I don't think we can answer, at this point, the question of what the
best circumstances are for enabling a feature that I haven't even
shown a prototype patch for (turns out sigblock is no longer
recommended! Who'd have known?)

It's possible the best answer is "only when the user tells us to
fork", but that's okay. Many features are disabled by default, and I
don't see why that shouldn't be true for a GC trick.

> > > > > Also, I quite frequently need to run Emacs on a system where I'm
> > > > > forbidden to run more than 2 processes simultaneously ("make -j3"
> > > > > aborts with an error message), and you propose to take those two
> > > > > processes with 2 copies of Emacs?
> > > >
> > > > That is, indeed, one of the cases in which we have to use synchronous 
> > > > GC.
> > >
> > > Again, how do we know we are in that case?
> >
> > I assume you don't have a reliable getrlimit that lets you know?
>
> I don't know -- does getrlimit always report this stuff, with all the
> virtualization that goes on nowadays?

It should, so it probably doesn't.

Pip