Re: The future: accessing vectors, arrays, etc from C

[Marius Vollmer]

Neil Jerram <address@hidden> writes:

> Firstly, I recall an old discussion between Jim and Mikael about a
> "leasing" interface in this area, which struck me at that time as
> very elegant.

Yes, I didn't remember this discussion.  Thank you for pointing it
out!

> I've not analysed how close it is to your thoughts, or how relevant
> it might be, but I thought you might like to review that and
> consider whether it affects your thinking at all.

The proposed leasing interface doesn't try to solve the problem that I
am adressing now: allowing changes to the representation of arrays
(which I take here to include vectors, uniform or not, bitvectors, and
strings) in a multi-threaded program.[1]

The leasing interface addresses the situation where you want to access
a Scheme vector as a vector of doubles, and want to avoid to making a
copy of the elements in the case the Scheme vector is a uniform
numeric vector already.  It doens't really deal with non-local exits,
I think, for example, which are important, too.

A change in representation of an array might happen when the copy for
a copy-on-write array takes place, or when the first 16-bit Unicode
character is stored in a 8-bit string.  In general, when we can come
up with a C API to arrays that is convenient and allows such changes
to happen, we should install it.  Once we can change represetation, we
can implement a lot of cool things (whether they are a good idea or
not), like growing arrays in place.

I think we can have such an API.  I will post documentation for it
soonish.

It will be more tedious to use than the old one mainly because it is
more general, something which has nothing to do with allowing
representations to change: When using the new API, your code must be
able to deal with non-contiguously stored vectors, while previously
you could just assume that all vectors are stored contiguously.

> Secondly, I am slightly concerned about the level of change that this
> means in the C API.

Yes, I am too.

> Actually, to be precise, I'm not sure I'm bothered about the level
> of change per se, as I believe the current reality is that a Guile
> application writer has no choice but to select an available major
> version of Guile (e.g. 1.6) and to target their application at
> specifically that version; and fortunately the bugs in Guile are few
> enough that it is feasible to stick with an older version.

Yes, you will be missing all the cool new features, tho.

> However, I wonder if perhaps we should have a more explicit and
> public policy on the level of change that is (i) acceptable and (ii)
> likely, from one major release to the next, so that developers know
> what to expect?

Hmm, that would be hard to formalize, no?

I think I get a feel for how intrusive these changes are, since I have
to change all of Guile itself to use the new API.  I hope that
restrains me from doing large, unmotivated changes.  Everytime when
installing a new API and changing Guile to use it, the code got much
cleaner (in my opinion), more robust, and subtle bugs were fixed.

What I am worried about is that I have not done the deprecation of the
old interfaces as good as it could be done.

For example, instead of the old SCM_VECTORP, SCM_VECTOR_REF and
SCM_VECTOR_SET, you now should use scm_is_simple_vector,
SCM_SIMPLE_VECTOR_REF, and SCM_SIMPLE_VECTOR_SET.  That might appear
to be arbitrary, since only names have changed.

On the other hand, this name change makes it clear that only the
special case of simple vectors is being dealt with, and people might
be motivated to look up in the manual how to deal with the general
case.

Hmm.

[1] Many of the problems also appear in single-threaded programs when
the program is allowed to do arbitrary things while having a lease for
an array.