bug#45200: Memory leaks: (garbage-collect) fails to reclaim memory

[Konstantin Kharlamov]

On Sun, 2021-01-24 at 14:12 -0500, Stefan Monnier wrote:
> > I'm fine with Emacs possibly keeping a dozen of megabytes for personal
> > use. The problem though is that the issue easily manifests itself in
> > hundreds of megabytes, as can be seen with the testcase marked as "Message
> > #11" in web-ui.
> 
> I don't think the exact number matters: you were willing to make use of
> *any* amount of extra memory by delaying all GC until idle time.
> The fact that the GC you finally do after that long wait doesn't recover
> that memory is not a failure in my book: there's no reason Emacs or
> glibc should presume that you won't be reusing just as much heap space
> before the next GC.
> 
> IOW in my book, you got what you asked for ;-)

This doesn't look right. I mean, sure, I might need the heap again. Or I might
not. I don't know. Neither you can. Nobody knows. If glibc can predict the
future then fine. But my experience as part of this report suggests glibc can't.

Glibc could for example use statistics of previous allocation patterns to see if
it's needed to release memory and how much, but this is not what happens. And I
am not the only dissatisfied user. Ruby for example too¹. Also, many people
experience strange memory usage grow with Qutebrowser that nobody knows where
it's coming from; and after today's research I suspect Glibc is the culprit (I
don't have yet enough evidence because my Qutebrowser instance doesn't have much
uptime ATM, but attaching to it with gdb and calling `malloc_trim` inside it
already freed ≈40M of memory to me today). Glib also seems affected².

With so many unhappy users, do you still think nothing wrong with the glibc
allocator? It seems to me, we're doing perfectly valid usecase: we used memory,
we don't need it anymore, we free it. If somebody ever notes that malloc/free
cycle takes too much time for them, then they know they need to optimize the
specific codepath where this happens. It's a nice bonus if Glibc could do
malloc/free faster, but not so much if this causes other problems, especially
given it's a core system library.

> BTW, I wish `malloc_trim` could be passed an argument that says
> something like "release about half of the free memory", which would
> provide a better balance between "hoard free memory indefinitely" and
> "constantly free&reallocate memory from the OS".
> 
> Also, the manpage of mallopt suggests that glibc should automatically
> do the trim on its own every once in a while anyway, so maybe the
> problem won't be solved by `malloc_trim`.

Yeah, I've seen that too. Idk why it doesn't work, but that `malloc_trim()` does
I know because I tested it with the testcase where previously Emacs was never
returning ≈200M of memory.

1:
https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#investigating-fragmentation-at-the-memory-allocator-level
2: https://gitlab.gnome.org/GNOME/glib/-/issues/2057