bug#58631: [Shepherd] Indefinite heap growth (memory leak)

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> The attached Fibers program illustrates the problem: heap grows even
>>> though it’s not supposed to.
>>
>> The saga continues at <https://github.com/wingo/fibers/issues/65>.
>
> … and then at <https://issues.guix.gnu.org/59021>, which appears to have
> a happy end: these Guix commits deploy the fix:
>
>   072fd1124a gnu: shepherd@0.9: Use 'guile-3.0-latest' to address memory leak.
>   7138ba34fa gnu: guile@3.0.8: Add patch to address continuation memory leak.
>
> In my testing it fixes the problem.  It’s now deployed it on
> bayfront.guix.gnu.org and we’ll deploy to berlin.guix.gnu.org later on
> and see how it goes in the coming days.

Turns out it’s neither the end nor happy!  On bayfront, shepherd is
apparently still leaking:

--8<---------------cut here---------------start------------->8---
ludo@bayfront ~$ date && sudo herd eval root '(gc-stats)'
Sun Nov  6 11:50:48 PM CET 2022
Password:
Evaluating user expression (gc-stats).
((gc-time-taken . 2193894749) (heap-size . 16404480) (heap-free-size . 5505024) 
(heap-total-allocated . 119183424) (heap-allocated-since-gc . 4071216) 
(protected-objects . 0) (gc-times . 21))
ludo@bayfront ~$ date && sudo herd eval root '(gc-stats)'
Mon Nov  7 09:45:35 AM CET 2022
Password:
Evaluating user expression (gc-stats).
((gc-time-taken . 741717175797) (heap-size . 47304704) (heap-free-size . 
10625024) (heap-total-allocated . 17295529056) (heap-allocated-since-gc . 
10237808) (protected-objects . 0) (gc-times . 1700))
ludo@bayfront ~$ date && sudo herd eval root '(gc-stats)'
Mon Nov  7 11:18:05 AM CET 2022
Password:
Evaluating user expression (gc-stats).
((gc-time-taken . 1090097976826) (heap-size . 55693312) (heap-free-size . 
5206016) (heap-total-allocated . 23190066736) (heap-allocated-since-gc . 
18995744) (protected-objects . 0) (gc-times . 1996))
--8<---------------cut here---------------end--------------->8---

There seems to be an unusually high number of “program” objects in the
heap:

--8<---------------cut here---------------start------------->8---
ludo@bayfront ~$ sudo herd eval root '(begin (load 
"/home/ludo/heap-profiler.scm") (profile-heap))'
Password:
Evaluating user expression (begin (load "/home/ludo/heap-profiler.scm") (#)).
  %   type                               self    avg obj size
 48.0 program                               1188240    32.1
 21.8 pair                                   539904    16.0
  7.4 bytevector                             184368   784.5
  5.6 vector                                 138373     7.1
  4.6 struct                                 113328    85.8
  3.5 unknown                                 86912    39.8
  2.0 stringbuf                               50144    60.5
  1.3 symbol                                  33152    32.9
  0.6 string                                  13728    34.1
  0.5 smob                                    12656    39.9
  0.5 variable                                12432    22.6
  0.4 frame                                   10720    34.6
  0.4 pointer                                 10256    32.5
  0.4 weak-vector                              9968    32.8
  0.4 bitvector                                9936    34.4
  0.4 hash-table                               9840    34.5
  0.3 weak-table                               7504    34.0
  0.3 vm-continuation                          6976    31.7
  0.3 heap-number                              6848    35.3
  0.2 atomic-box                               5488    36.1
  0.2 port                                     5248    36.4
  0.2 dynamic-state                            3760    30.1
  0.1 fluid                                    3664    70.5
  0.1 syntax                                   3312    48.0
  0.1 keyword                                  2016    40.3
  0.1 weak-set                                 1872    48.0
  0.1 array                                    1776    48.0
  0.1 partial-continuation                     1760    39.1
  0.0 primitive                                 528    16.0
  0.0 foreign-program                           448    44.8
  0.0 continuation                              336    48.0
  0.0 primitive-generic                         176    35.2
sampled heap: 2.36098 MiB (heap size: 53.11328 MiB)
--8<---------------cut here---------------end--------------->8---

(Note that ‘partial-continuation’, which are programs with the
SCM_F_PROGRAM_IS_PARTIAL_CONTINUATION flag, are counted separately.  So
here we really talking about programs with flags == 0.)

Ludo’.