Re: Opportunistic GC

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Thu, 11 Mar 2021 08:06:50 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> > I don't see how that helps to get out of the conundrum.  The
> > synchronous GC doesn't enlarge the memory footprint of the Emacs
> > process, so if we had enough memory when we started GC, there are good
> > chances we will have it throughout the GC.  By contrast, when you
> > fork, you enlarge the memory pressure in the system, and the amount of
> > the increment is unknown in advance (because it's determined by the
> > degree of nesting of the Lisp data you have in the session at that
> > moment).  You could enlarge the memory enough to cause OOM killer to
> > kick in.
> 
> All of these problems apply to all forks, not just GC ones.

You mean, when we start a sub-process?  In that case the memory of the
Emacs process is not cloned, because we immediately call 'exec'.  But
in the case of forking and running GC, some part of the Emacs memory
will have to be cloned.  Which part is that is hard to predict in
advance; it could be a large part.

> > So it's important to know when not to attempt the fork.
> 
> I don't think "normal" GNU/Linux systems should ever be in that
> situation, since they have virtual memory.

I don't think I follow.  Virtual memory can be exhausted as well,
right?  That's the danger I'm talking about.

> And, of course, it's possible even on GNU/Linux system for the fork()
> not to succeed, which we need to handle.

The problem is with the scenario where the trouble begins long after
the 'fork' call succeeded.  Or are you saying that the OS will fail
the 'fork' call right when we call it, and if it doesn't, there's no
chance the forked child will come close to exhausting VM?

> > What is the
> > algorithm for that?  If you want to use the fraction of the memory the
> > Emacs process takes, and set the threshold at 50%, then we now need to
> > discuss how to know that fraction reliably, what with some of the
> > memory coming from sbrk and some from mmap -- this is why we all but
> > eliminated memory-limit as not being very useful, and you now want to
> > use the same technique to base a critical decision.
> 
> Ideally, we'd use the same (broken) technique as the (broken) OS uses
> to determine whether to kill us.

Is that possible?  Does the OS expose enough APIs to glean this info?
If so, perhaps we could use that to resurrect the usefulness of
memory-limit?