Re: out-of-control GC

[Arne Babenhauserheide]

Hi Linas,

Linas Vepstas <address@hidden> writes:
> On Sun, Sep 10, 2017 at 11:50 AM, Arne Babenhauserheide <address@hidden>
>> Linas Vepstas <address@hidden> writes:
>>
>> > To make this conversation less crazy, and more down-to-earth, here
>> > is a demo that seems to spend most of its time in GC. Hard for me
>> > to to say that this is "wrong", right now, but it illustrates the
>> > scandalous nature of the problem.
>>
>> Do you have many string operations in your code?
>>
>> If I replace
>>
>>     (cons (format #f "~A" cnt) lis)
>>
>> with
>>
>>     (cons cnt lis)
>>
>> GC is down to ~20% (from around 130%).
>>
>
> That demo tried to capture the spirit of a more complex system. The cons
> that I have is actually (cons (wrapper-for-some-c++-code foobar) lis)  or
> maybe (cons (scheme-that-calls-other-scheme-that-calls-c++ foo) lis)  ...
> but its not just cons -- it can also be for-each, map, or fold.

I looked at format, because it seems to create a lot of values which
need to be collected.

Just try to run your function like this:

    (gc-disable)
    (define foo (make-bgl '() (* 1235 1000 1000)))

… and see memory usage go through the roof almost instantly.

Then try the same without the format to get a mostly tame memory
behavior.

> The actual printouts, in my instrumented code look like this:
>
> guile-en-r> (avg-gc-cpu-time)
> Elapsed: 4461. secs. Rate: 4.236 gc/min %cpu-GC: 280.8%  %cpu-use: 284.0%

That looks painful indeed.

> Anyway, this demo gives a scenario where gc seems to run too often.  I have
> other scenarios where it does not run often enough -- I have one where the
> working set is about 200MBytes, but guile happily grows to 10GB or 20GB RAM
> over a few hours.  My solution for that other case is to manually GC
> whenever gc-stats gets above 1GB, because I know a priori my working set is
> smaller than that.
>
> The overall message seems to be that gc either runs too often, or not often
> enough.  Some sort of better planning is needed.

Maybe Guile needs some additional information from the C++
datastructures about the total size of the managed memory (and the
actual size of datastructures in foreign code) so it can judge whether
the added memory usage it sees is actually already relevant.

> I'm motivated to explore this, as it actually impacts my work.

That’s great! And it is also awesome that you’re pushing Guile this far!
This helps Guile become a reliable tool for many more people.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

signature.asc
Description: PGP signature