Re: out-of-control GC

[Arne Babenhauserheide]

Hi Linas,

Linas Vepstas <address@hidden> writes:

>> If I replace
>>
>>     (cons (format #f "~A" cnt) lis)
>>
>> with
>>
>>     (cons cnt lis)
>>
>> GC is down to ~20% (from around 130%).
>>
>
> That demo tried to capture the spirit of a more complex system.

That’s great, by the way!

I did that experiment, because the string formatting seems to create
lots of values the GC has to collect. If this triggers an investigation
of all memory, any reduction of disposable value creation should help.

Just try putting a (gc-disable) just before you run make-bgl and see the
memory usage shoot through the roof almost instantly.

For the real-code: throw in a 

> The cons
> that I have is actually (cons (wrapper-for-some-c++-code foobar) lis)  or
> maybe (cons (scheme-that-calls-other-scheme-that-calls-c++ foo) lis)  ...
> but its not just cons -- it can also be for-each, map, or fold.
>
> The actual printouts, in my instrumented code look like this:
>
> guile-en-r> (avg-gc-cpu-time)
> Elapsed: 4461. secs. Rate: 4.236 gc/min %cpu-GC: 280.8%  %cpu-use: 284.0%

This indeed looks pretty painful.

> Anyway, this demo gives a scenario where gc seems to run too often.  I have
> other scenarios where it does not run often enough -- I have one where the
> working set is about 200MBytes, but guile happily grows to 10GB or 20GB RAM
> over a few hours.  My solution for that other case is to manually GC
> whenever gc-stats gets above 1GB, because I know a priori my working set is
> smaller than that.
>
> The overall message seems to be that gc either runs too often, or not often
> enough.  Some sort of better planning is needed.

A short term solution could be to disable automatic triggering of GC in
cases where it fires too often (compared to the total memory
consumption).

> I'm motivated to explore this, as it actually impacts my work.

That’s great! And it’s awesome that you’re pushing Guile that far. This
is what it needs to become a reliable real-world battered workhorse for
many more people.

A start might be to look at the problem description: GC is triggered too
often when your C++ datastructures are so huge that the growth in
Guile-managed memory is irrelevant, and it is triggered too seldomly
when you only manage a small amount of data.

So maybe GC could benefit from a kind of hint from C++ about the size of
the datastructures which allows tuning the GC frequency to the total
amount of memory.

-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

signature.asc
Description: PGP signature