Re: Modules

[Thien-Thi Nguyen]

() Neil Jerram <address@hidden>
() Sat, 29 Jan 2011 23:17:08 +0000

   If the modules are installed, that's true.  What if they are not?

   [...]

   For scripts that use uninstalled modules, then, some kind of
   solution is needed; ideally one that works for both 1.8 and
   1.9/2.0, allows the code needed to live in a single common
   file, rather than duplicated at the top of each script; and
   continues to work if the script+module tree as a whole is moved
   to a different place in the filesystem.

This is what "module catalogs" from Guile 1.4.x provides.

[verbose explanation follows, skip to end for summary]

Basically, you associate with each element in ‘%load-path’ a
sub-association between module name (list of symbols) and
implementation resolution method.  For Guile 1.4.x, there are two
types of modules supported:
 - scheme source (text)
 - shared object library

Of course the implementation must provide the appropriate
interface (as defined by the support in ‘resolve-module’) for
loading; this system cannot be used for arbitrary scheme source or
shared object libraries.

At load-time, the interpreter consults these catalogs
preferentially, falling back to old-style filesystem groping on
lookup failure.  A module (of any form) need not include its
location information.

To see how this helps, i use two small scripts:

st:

  #!/bin/sh
  exec strace -f -e open "$@" 2>&1 | grep -v o.such.file

sta:

  #!/bin/sh
  exec strace -f -e open "$@" 2>&1

Here is a run of loading (database postgres-table) [scheme code]
from the Guile-PG build tree.  Note that it in turn requires
(database postgres) [shared object library].  Also note ‘-L .’
which is explained further down.

  $ cd ~/build/guile-pg/.b
  $ st guile -L . -c '(use-modules (database postgres-table))'
  open("/home/ttn/local/lib/libguile.so.9", O_RDONLY) = 3
  open("/home/ttn/local/lib/libltdl.so.7", O_RDONLY) = 3
  open("/etc/ld.so.cache", O_RDONLY)      = 3
  open("/lib/i686/cmov/libm.so.6", O_RDONLY) = 3
  open("/lib/i686/cmov/libdl.so.2", O_RDONLY) = 3
  open("/lib/i686/cmov/libc.so.6", O_RDONLY) = 3
  open("/home/ttn/local/lib/guile/site/.module-catalog", O_RDONLY) = 3
  open("/home/ttn/local/share/guile/site/.module-catalog", O_RDONLY) = 3
  open("/home/ttn/local/lib/guile/1.4.1.122/.module-catalog", O_RDONLY) = 3
  open("/home/ttn/local/lib/guile/site/init.scm", O_RDONLY) = 3
  open("./.module-catalog", O_RDONLY)     = 3
  open("/home/ttn/build/guile-pg/src/postgres-table.scm", O_RDONLY) = 3
  open("/home/ttn/local/lib/guile/1.4.1.122/ice-9/common-list.scm", O_RDONLY) = 
4
  open("/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0", O_RDONLY) = 4
  open("/home/ttn/local/lib/libpq.so.3", O_RDONLY) = 4
  open("/etc/ld.so.cache", O_RDONLY)      = 4
  open("/lib/i686/cmov/libcrypt.so.1", O_RDONLY) = 4
  open("/lib/i686/cmov/libresolv.so.2", O_RDONLY) = 4
  open("/lib/i686/cmov/libnsl.so.1", O_RDONLY) = 4
  open("/home/ttn/build/guile-pg/src/postgres-types.scm", O_RDONLY) = 4
  open("/home/ttn/build/guile-pg/src/postgres-col-defs.scm", O_RDONLY) = 4
  open("/home/ttn/build/guile-pg/src/postgres-qcons.scm", O_RDONLY) = 4
  open("/home/ttn/build/guile-pg/src/postgres-resx.scm", O_RDONLY) = 4

If i use ‘sta’, there are many "no such file" messages, but NOT
with the modules.  Strictly speaking, open(2) savings proves
improvement only for shared object libraries, so you'll have to
trust me (or confirm by looking at the source code) that the
scheme source modules are treated analogously (i.e., no probing).

  $ cd ~/build/guile-pg/.b
  $ st  guile -L . -c '(use-modules (database postgres-table))' > A
  $ sta guile -L . -c '(use-modules (database postgres-table))' > B
  $ diff -u A B | uniq
  --- A 2011-01-30 10:41:25.000000000 +0100
  +++ B 2011-01-30 10:41:32.000000000 +0100
  @@ -1,8 +1,29 @@
  +open("/home/ttn/local/lib/tls/i686/sse2/cmov/libguile.so.9", O_RDONLY) = -1 
ENOENT (No such file or directory)
  +open("/home/ttn/local/lib/tls/i686/sse2/libguile.so.9", O_RDONLY) = -1 
ENOENT (No such file or directory)
  +open("/home/ttn/local/lib/tls/i686/cmov/libguile.so.9", O_RDONLY) = -1 
ENOENT (No such file or directory)
  +open("/home/ttn/local/lib/tls/i686/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
  +open("/home/ttn/local/lib/tls/sse2/cmov/libguile.so.9", O_RDONLY) = -1 
ENOENT (No such file or directory)
  +open("/home/ttn/local/lib/tls/sse2/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
  +open("/home/ttn/local/lib/tls/cmov/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
  +open("/home/ttn/local/lib/tls/libguile.so.9", O_RDONLY) = -1 ENOENT (No such 
file or directory)
  +open("/home/ttn/local/lib/i686/sse2/cmov/libguile.so.9", O_RDONLY) = -1 
ENOENT (No such file or directory)
  +open("/home/ttn/local/lib/i686/sse2/libguile.so.9", O_RDONLY) = -1 ENOENT 
(No such file or directory)
  +open("/home/ttn/local/lib/i686/cmov/libguile.so.9", O_RDONLY) = -1 ENOENT 
(No such file or directory)
  +open("/home/ttn/local/lib/i686/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
  +open("/home/ttn/local/lib/sse2/cmov/libguile.so.9", O_RDONLY) = -1 ENOENT 
(No such file or directory)
  +open("/home/ttn/local/lib/sse2/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
  +open("/home/ttn/local/lib/cmov/libguile.so.9", O_RDONLY) = -1 ENOENT (No 
such file or directory)
   open("/home/ttn/local/lib/libguile.so.9", O_RDONLY) = 3
   open("/home/ttn/local/lib/libltdl.so.7", O_RDONLY) = 3
  +open("/home/ttn/local/lib/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or 
directory)
   open("/etc/ld.so.cache", O_RDONLY)      = 3
   open("/lib/i686/cmov/libm.so.6", O_RDONLY) = 3
  +open("/home/ttn/local/lib/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file 
or directory)
   open("/lib/i686/cmov/libdl.so.2", O_RDONLY) = 3
  +open("/home/ttn/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or 
directory)
   open("/lib/i686/cmov/libc.so.6", O_RDONLY) = 3
   open("/home/ttn/local/lib/guile/site/.module-catalog", O_RDONLY) = 3
   open("/home/ttn/local/share/guile/site/.module-catalog", O_RDONLY) = 3
  @@ -13,9 +34,21 @@
   open("/home/ttn/local/lib/guile/1.4.1.122/ice-9/common-list.scm", O_RDONLY) 
= 4
   open("/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0", O_RDONLY) = 4
   open("/home/ttn/local/lib/libpq.so.3", O_RDONLY) = 4
  +open("/home/ttn/local/lib/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such 
file or directory)
   open("/etc/ld.so.cache", O_RDONLY)      = 4
   open("/lib/i686/cmov/libcrypt.so.1", O_RDONLY) = 4
  +open("/home/ttn/local/lib/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such 
file or directory)
   open("/lib/i686/cmov/libresolv.so.2", O_RDONLY) = 4
  +open("/home/ttn/local/lib/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file 
or directory)
   open("/lib/i686/cmov/libnsl.so.1", O_RDONLY) = 4
   open("/home/ttn/build/guile-pg/src/postgres-types.scm", O_RDONLY) = 4
   open("/home/ttn/build/guile-pg/src/postgres-col-defs.scm", O_RDONLY) = 4

The ‘-L .’ means that ~/build/guile-pg/.b/.module-catalog is
consulted:

  ;;; .module-catalog
  ;;; generated 2010-12-18 20:35:15 UTC -- do not edit!
  
  (
   ((database postgres-resdisp) . 
"/home/ttn/build/guile-pg/src/postgres-resdisp.scm")
   ((database postgres-col-defs) . 
"/home/ttn/build/guile-pg/src/postgres-col-defs.scm")
   ((database postgres-table) . 
"/home/ttn/build/guile-pg/src/postgres-table.scm")
   ((database postgres-meta) . "/home/ttn/build/guile-pg/src/postgres-meta.scm")
   ((database postgres-gxrepl) . 
"/home/ttn/build/guile-pg/src/postgres-gxrepl.scm")
   ((database postgres) scm_init_module "scm_init_database_postgres_module" () 
. "/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0")
   ((database postgres-types) . 
"/home/ttn/build/guile-pg/src/postgres-types.scm")
   ((database postgres-qcons) . 
"/home/ttn/build/guile-pg/src/postgres-qcons.scm")
   ((database postgres-resx) . "/home/ttn/build/guile-pg/src/postgres-resx.scm")
  )
  
  ;;; .module-catalog ends here

This (version 1) format is not very compact.  For version 2, we
add a magic number, factor the root and all intervening parent
directories, provide other meta info in a "header" section, and
include other meta info (exports list being the most interesting)
in the body.  In some sense version 2 is like /etc/ld.so.cache.

I imagine Guile 2.x could adopt module catalogs, extending to
handle also compiled scheme (.go) and the (future) JIT files as
well, but perhaps it's too late.  I wonder how module catalogs
would fit w/ name canonicalization, which IIUC is specific to
compiled scheme.

In sum, build-time caching plus run-time indirection as a means to
facile module relocatability (plus some performance gain).