Re: Guile foreign object interface

[David Kastrup]

address@hidden (Ludovic Courtès) writes:

> David Kastrup <address@hidden> skribis:
>
>> address@hidden (Ludovic Courtès) writes:
>>
>>> I’m sure we already discussed it and then I forgot, but would
>>> anything prevent the use of specific C++ allocators in this case?
>>> The STL data structures could be allocated on GC-scanned memory, in
>>> which case mark procedures are unneeded.
>>
>> We are not talking about STL data structures containing SCM values
>> but data structures containing other data structures and pointers to
>> other data structures.  You cannot sensibly garbage collect when
>> whole memory areas in which allocation and deallocation is done are
>> in GC-scanned memory and there is no difference between data
>> structures that have been freed and data structures that haven't.
>
> Not sure what you mean.

The code is out there.  Typical stuff looks like

SCM
Context::mark_smob () const
{
  scm_gc_mark (context_list_);
  scm_gc_mark (aliases_);
  scm_gc_mark (definition_);
  scm_gc_mark (definition_mods_);
  scm_gc_mark (properties_scm_);
  scm_gc_mark (accepts_list_);
  scm_gc_mark (default_child_);

  if (implementation_)
    scm_gc_mark (implementation_->self_scm ());

  if (event_source_)
    scm_gc_mark (event_source_->self_scm ());

  if (events_below_)
    scm_gc_mark (events_below_->self_scm ());

  derived_mark ();

  return properties_scm_;
}

The first members are marked directly as SCM.  The others are marked
indirectly.

There is also stuff like

void
Slur_engraver::derived_mark () const
{
  for (vsize i = start_events_.size (); i--;)
    {
      scm_gc_mark (start_events_[i].slur_->self_scm ());
      if (start_events_[i].note_)
        scm_gc_mark (start_events_[i].note_->self_scm ());
    }
  for (vsize i = stop_events_.size (); i--;)
    {
      scm_gc_mark (stop_events_[i].slur_->self_scm ());
      if (stop_events_[i].note_)
        scm_gc_mark (stop_events_[i].note_->self_scm ());
    }
}

where the involved arrays are actually STL vectors.

>> LilyPond is a large-scale application which will eat up a significant
>> ratio of the available address space in 32 bit applications and will
>> process, in a documentation run, thousands of files with little overlap.
>> Already in Guile 1.8 where _only_ the stack is conservatively marked and
>> gc is called explicitly at known points of low-stack usage, we have
>> occasional false positives in garbage collection (the debugging runs
>> flag them).  With large memory areas being conservatively scanned, this
>> is going to get a whole lot worse.
>
> This sounds like speculation to me.  I don’t think we’ve seen
> slowdowns when Guile then 1.9 switched to BDW-GC, quite the opposite.

This is not a problem of slowdowns but of unreclaimed memory because of
false positives in reference detections.  LilyPond has large sessions
where it basically frees all memory after the startup thousands of times
over.

>> The point of time where it makes sense to try to evaluate the impact
>> or feasibility of such functionality removals (and Boehm GC _does_
>> support finalization) is when LilyPond is otherwise running fine on
>> Guile 2.  With the current slowdown by a factor of 5-10 and a number
>> of other problems preventing the LilyPond test suite from running,
>> there just isn't any setup where a useful evaluation could take
>> place.
>
> LilyPond on Guile 2.0 is 5–10 times slower than on 1.8?

Yes.

>> As there is no technical necessity for planning the wreckage of
>> existing functionality when there hasn't even been a stable release
>> containing the prospective successor, I cannot see a viable point in
>> raising yet more hurdles for existing applications to migrate to
>> Guile 2 rather than bite the bullet and fork Guile 1.8 in order to
>> actually have some dependable functionality to base other work on.
>
> Note that SMOBs are still around in 2.2, not even deprecated, only
> discouraged.

I have no problem with discouragement.

>>> As an aside, please keep the tone friendly as is the norm on this
>>> mailing list.
>>
>> Disagreement is not the same as unfriendliness.
>
> I agree.  However I found the tone of your messages patronizing and
> aggressive, assuming bad faith and incompetence on the side of the
> Guile developers (“planning for wreckage”, “pretty pointless”,
> etc. etc.)
>
> This is not OK on the Guile mailing lists.

Given the current state of things and the current discussion and
wordings being used, the terms I used are describing the effects as they
appear when viewed through the lens of LilyPond development.  I saw no
convincing _technical_ reason for the enthusiasm towards stopping users
from being able to use their existing code without a workable
replacement in light of the existing mechanisms.

As I said: LilyPond has its Guile object management encapsulated and
modularized to a degree where it was simple to implement some finalizer
workaround on the LilyPond side until the problem finally also got fixed
in Guile 2.0.13 or so.

But it would be a huge step backwards if you had to use STL containers
with separate allocation if and only if SCM data structures got
involved.  It would mean that creating and maintaining standard LilyPond
code would require a lot more low-level expertise from programmers than
is the case now.  I've been working hard on encapsulating special
knowledge into modules that make creating a lot of LilyPond code
accessible to programmers with moderate skill levels.

I'll readily grant that foregoing mark procedures would be beneficial in
that respect, but it would be more than offset by having to meddle with
custom allocators: that is _way_ more complex for average programmers.

As long as there are no actual good answers to the problems posed by
LilyPond's manner of wrapping C++ classes containing STL data types
referencing SCM with various degrees of indirection (which is done for
good reasons), the lighthearted manner in which burning bridges with
existing code (which addresses existing problems in reasonably
straightforward manner) is proposed is disconcerting.

I am aware that not working with finalizers/mark routines has its own
attractions: if it didn't, bugs like the one addressed with

commit 8442211ef0029581b35f784489afcf210491fc41
Author: David Kastrup <address@hidden>
Date:   Sat Sep 20 05:17:54 2014 -0400

    Fix SCM_SMOB_OBJECT{_,_0_,_1_,_2_,_3_}LOC.
    
    Fixes <http://bugs.gnu.org/18495>.
    
    * libguile/smob.h (SCM_SMOB_OBJECT_LOC, SCM_SMOB_OBJECT_0_LOC)
      (SCM_SMOB_OBJECT_1_LOC, SCM_SMOB_OBJECT_2_LOC)
      (SCM_SMOB_OBJECT_3_LOC): These elementary API macros have been broken
      by commit 56164dc47f6616b359f0ad23be208f01a77b55fa in 2009.
    
    Signed-off-by: David Kastrup <address@hidden>

wouldn't have survived as long.

-- 
David Kastrup