Re: GC Warning: Repeated allocation of very large block

[Olivier Dion]

On Mon, 19 Sep 2022, Damien Mattei <damien.mattei@gmail.com> wrote:
> is this message appearing when a single scheme variable reach a given
> size?

This message is from the bdwgc and not from Guile itself.  From their
documentation:
--8<---------------cut here---------------start------------->8---
The garbage collector generates warning messages of the form:


    Repeated allocation of very large block ...
    May lead to memory leak and poor performance


when it needs to allocate a block at a location that it knows to be referenced
by a false pointer. These false pointers can be either permanent (e.g.
a static integer variable that never changes) or temporary. In the latter
case, the warning is largely spurious, and the block will eventually
be reclaimed normally. In the former case, the program will still run
correctly, but the block will never be reclaimed. Unless the block is intended
to be permanent, the warning indicates a memory leak.

  1. Ignore these warnings while you are using GC_DEBUG. Some of the routines
  mentioned below don't have debugging equivalents. (Alternatively, write the
  missing routines and send them to me.)
  2. Replace allocator calls that request large blocks with calls to
  `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You may
  want to set a breakpoint in `GC_default_warn_proc` to help you identify such
  calls. Make sure that a pointer to somewhere near the beginning of the
  resulting block is maintained in a (preferably volatile) variable as long
  as the block is needed.
  3. If the large blocks are allocated with realloc, we suggest instead
  allocating them with something like the following. Note that the realloc
  size increment should be fairly large (e.g. a factor of 3/2) for this to
  exhibit reasonable performance. But we all know we should do that anyway.


        void * big_realloc(void *p, size_t new_size) {
            size_t old_size = GC_size(p);
            void * result;
            if (new_size <= 10000) return(GC_realloc(p, new_size));
            if (new_size <= old_size) return(p);
            result = GC_malloc_ignore_off_page(new_size);
            if (result == 0) return(0);
            memcpy(result,p,old_size);
            GC_free(p);
            return(result);
        }


  4. In the unlikely case that even relatively small object (<20 KB)
  allocations are triggering these warnings, then your address space contains
  lots of "bogus pointers", i.e. values that appear to be pointers but aren't.
  Usually this can be solved by using `GC_malloc_atomic` or the routines
  in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc.
  Sometimes the problem can be solved with trivial changes of encoding
  in certain values. It is possible, to identify the source of the bogus
  pointers by building the collector with `-DPRINT_BLACK_LIST`, which will
  cause it to print the "bogus pointers", along with their location.
  5. If you get only a fixed number of these warnings, you are probably only
  introducing a bounded leak by ignoring them. If the data structures being
  allocated are intended to be permanent, then it is also safe to ignore them.
  The warnings can be turned off by calling `GC_set_warn_proc` with
  a procedure that ignores these warnings (e.g. by doing absolutely
  nothing).
--8<---------------cut here---------------end--------------->8---

It's my understanding that either A) there's a leak or B) its a false
positive.

> i wanted to debug and trace because i know this algorithm expand a lot
> expressions (perheaps too much) sometimes (but it is an NP-problem and
> exponential, so perheaps stop the program is the only solution and run it
> on more little data)
> i do not think there could be a memory leak in a recursive scheme
> program,

Gotta be careful with a conservative GC I guess.  Your memory will get
reclaim at some point, but it's not guaranteed when since any value on
the C stack -- and global variables -- could reference your allocation
by accident.  So if by accident there's a static constant value in C
that has the value of an allocation by the GC, it will never reclaim it.
Since it's the Scheme stack and I suppose it's tail call optimized, that
should not impact the GC.

The most important thing would be to check the memory usage of the
program with a tool like `htop'.  If there's leak, you will see that the
memory usage percentage keep increasing.

-- 
Olivier Dion
oldiob.dev