Re: mmap in guile -- guile memory management question

[Daniel Ridge]

Ludovic,

On Nov 13, 2006, at 4:15 AM, Ludovic Courtès wrote:

> Hi,
>
> Daniel Ridge <address@hidden> writes:
>
> > I use mmap in guile via a very simple C module and I have a piece of
> > complicated Guile hackery that I would like to simplify. I'm hoping
> > someone will embarrass me with 3 rational lines to solve my problem.
> >
> > My problem is that I don't know how to manufacture a new Guile string
> > with a pre-existing pointer (returned by mmap) and a length that I
> > specify. Instead, I grab a string from Guile and whack its contents
> > and length with my own and then put back a reasonable value in a
> > guardian at GC time.

<snip>

>   In
> 1.8, you would use SRFI-4 octet vectors to that end (passing the  
> address
> returned by `mmap ()' to `scm_take_u8vector ()' and making the  
> returned
> vector uncollectable so that `free ()' isn't eventually invoked on the
> `mmap' address).

A single mmap may be 20GB. It would be a bit rude to prevent garbage  
collection on such an object. I think the mechanics of the problem  
are the same whether I use take_u8vector or take_string.

BTW: Guile builds easily as a 64-bit executable under Mac OS X on my  
Core2Duo MacBook Pro. Large memory maps are finally simple on  
consumer hardware!

> However, ports are generally more appropriate than vectors to handle
> arbitrarily large file contents.

Ports are useful, but have different semantics than memory maps. I  
agree that Guile could have a file port implementation built on  
memory map but the result would be non-portable, would work poorly on  
32 bit machines, and would still not provide the semantics I expect.  
In particular, mmap lets me do shared memory programming with Guile.

>   Thus, I would recommend using a
> regular file port and then using something like `uniform-array-read!'
> (in 1.8) to get part of the file contents into a vector.  Note that  
> the
> current implementation of file ports doesn't use `mmap ()' behind the
> scenes (but it could be changed to do so).

Thanks,
        Dan Ridge