Re: Guile bugs

[Linas Vepstas]

Hi Marko,

On Sat, Sep 9, 2017 at 5:31 PM, Marko Rauhamaa <address@hidden> wrote:

> Linas Vepstas <address@hidden>:
> > So if you used GC_malloc_atomic() in your code, then gc will NOT scan
> > that region for pointers. guile-2.2 does this correctly for strings (I
> > checked) because strings will never contain pointers. I have not
> > checked other uses.
>
> The question about mmap(2) is addressed more directly by the README
> under <URL: https://github.com/ivmai/bdwgc>:
>
>    Any objects not intended to be collected must be pointed to either
>    from other such accessible objects, or from the registers, stack,
>    data, or statically allocated bss segments. [...]
>
>    WARNING: pointers inside memory allocated by the standard malloc are
>    not seen by the garbage collector. Thus objects pointed to only from
>    such a region may be prematurely deallocated. It is thus suggested
>    that the standard malloc be used only for memory regions, such as I/O
>    buffers, that are guaranteed not to contain pointers to garbage
>    collectible memory.
>
>    [...]
>
>    WARNING: the collector does not guarantee to scan thread-local
>    storage (e.g. of the kind accessed with pthread_getspecific). The
>    collector does scan thread stacks
>
> Now, the question is, can you make your multigigabyte allocation go into
> these excluded memory segments? Are you still hit by the pathological GC
> behavior you reported?
>

What, my app, specifically? Uh yeah. All of my c++ objects go through
plain-old malloc and should be 100% invisible to guile and to bdwgc and
that's perfect.

I do have a layer where guile interacts with the C++ code, and all the
various bits and pieces are correctly declared everywhere.  That code is
about 8-10 years old, its been maintained all that time, its been in use
all that time, and works without issues.

"works without issues" means this: A typical run for me uses some 50-100
threads, maybe 1/2 of these in guile. Due to lock contention, in practice
only 3 to 10 of these threads ever get scheduled, and this is fine, and not
unexpected.  My algos are the opposite of "embrassingly parallel", so low
parallelism is expected. Each processing run goes for a few days to weeks
or over a month, uses 2 to 50GB RAM, does not leak memory, does not crash,
its stable, predictable, does what its supposed to do, terminates cleanly.

It does sometimes run much slower than expected (about 1/2x or 1/4x) and I
don't know why.  I don't usually monitor GC, but when I do, it seems that a
lot of cpu time gets wasted there. Sometimes other people also notice this,
bitch at me, and propose silly solutions. I've been ignoring that as
lower-priority.

Only now it's risen up and is biting hard, and I don't understand how or
why gc gets triggered in guile.  For my app, it seems that it is running
far, far too often, and is severely slowing/preventing forward progress.

--linas

>
>
> Marko
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *