Re: Grow heap aggressively during music interpretation (issue 561390043 by address@hidden)

[Han-Wen Nienhuys]

Thanks, I ran the carver score successfully now.

I took some pauses between runs so thermal throttling wasn't an issue.

My standard guile2.2 branch (with 40M initial heap), takes about 2m wall
time, 1m of GC time.

The stats printing shows that we have a very full heap:

In-use heap: 85% (370824 KiB pointers + 60065 KiB other)
In-use heap: 85% (370725 KiB pointers + 59737 KiB other)
In-use heap: 85% (370143 KiB pointers + 59687 KiB other)
..

so we have to scan a lot of live data to reclaim just 15% of the heap.

Tinkering with INITIAL_HEAP lets us tune things a little bit:

 initial heap 2G: 50s (2.69s GC)
 initial heap 900M: 0:54 (6s GC)
 initial heap 500M: 1:09 (33s GC)

The GC timings seem wonky, but since BDW is multithreaded, it's possible
that the computation times don't completely add up.

GUILE 1.8 in default configuration:

 wall time 40s, 100% CPU.



On Thu, Feb 6, 2020 at 1:45 PM Jonas Hahnfeld <address@hidden> wrote:

> Am Donnerstag, den 06.02.2020, 00:27 +0100 schrieb David Kastrup:
> > Han-Wen Nienhuys <
> > address@hidden
> > > writes:
> >
> > > On Wed, Feb 5, 2020 at 12:33 PM <
> > > address@hidden
> > > > wrote:
> > >
> > > when you see things like:
> > >
> > > World-stopped marking took 187 msecs (77 in average)
> > > In-use heap: 51% (40071 KiB pointers + 6357 KiB other)
> > >
> > > it means it took 187ms for GC, and then reclaimed 49% of the memory.
> > >
> > > If you often see higher percentages, we'll spend more CPU in marking
> memory
> > > without getting fresh memory.
> > >
> > > I tried to run the carver score, but it needs updating and,
> > >
> > >  [hanwen@localhost lilypond]$ ./out/bin/convert-ly  carver/*ly
> > > convert-ly (GNU LilyPond) 2.21.0
> > >
> > > Traceback (most recent call last):
> > >   File "./out/bin/convert-ly", line 413, in <module>
> > >     main ()
> > >   File "./out/bin/convert-ly", line 387, in main
> > >     f = f.decode (sys.stdin.encoding or "utf-8")
> > > AttributeError: 'str' object has no attribute 'decode'
> > > [hanwen@localhost lilypond]$ python2 ./out/bin/convert-ly  carver/*ly
> > > Traceback (most recent call last):
> > >   File "./out/bin/convert-ly", line 59, in <module>
> > >     import lilylib as ly
> > >   File
> > > "/home/hanwen/vc/lilypond/out/lib/lilypond/current/python/lilylib.py",
> line
> > > 216
> > >     print('command failed:', cmd, file=sys.stderr)
> > >                                       ^
> > > SyntaxError: invalid syntax
> > >
> > > Is there still work left for the python3 conversion?
> > >
> > > > https://codereview.appspot.com/561390043/
> > > >
> > > >
> >
> > Yes, separate issue.  Jonas, I think you should push the fix for that
> > one to staging: from "doesn't work" there cannot be much more of a
> > regression to be afraid of.
>
> Right, in staging now.
>


-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen