-*-text-*- Dynamically-loaded native code modules for Emacs ------------------------------------------------ Dave Love
2002-02-27 * Features These changes add the ability to dynamically load modules coded in C, like the Emacs interpreter core using Libtool facilities. This works just by extending `load' to accept file names with extension `.la' which describe Libtool-built modules. These are found on `load-path', like Lisp files, taking precedence over Lisp of the same base name if an extension isn't specified. They work with `require' (if the module provides a feature) and `eval-after-load' (should that ever be useful). They can't be loaded from remote filesystems. There is a feature test `dynamic-modules-p' added for the facility, but I'm not sure that's actually useful. Modules can only be loaded once and cannot be unloaded; it may be possible to remove these constraints. The script `build-emacs-module' cans the Libtool and doc-snarfing steps for building modules from source. XEmacs has a rather more complicated native code module system. I'm not sure the complication is necessary, and it doesn't build on libtool. (Bill Perry, who did the original XEmacs implementation, agrees that using Libtool is the right thing.) * Installation and building The diffs are against the Emacs 21.1 code base. To use them, you need a recent version of GNU Libtool and autoconf 2.13. I used Libtool 1.4.2, and presumably anything more recent will work; version 1.3.x won't work. Autoconf 2.5x won't work. After applying the patch, go to the top of the source tree and run `autoconf', then `libtoolize --ltdl' and follow the instruction about aclocal.m4. Then (re)run configure. After that, it's a good move to append the following to the configured libltdl/config.h -- there doesn't seem to be a convenient hook to automate this: #ifndef LT_LAZY_OR_NOW #ifdef RTLD_NOW #define LT_LAZY_OR_NOW RTLD_NOW #else #ifdef DL_NOW #define LT_LAZY_OR_NOW DL_NOW #endif #endif #endif (The effect of this change is to check for unresolved linker symbols referenced by the module when it is loaded. Otherwise, Emacs will crash when calling a function in the module which references an unresolved symbol.) After that, run make to rebuild Emacs as usual. This all may not DTRT if the platform doesn't support dynamic loading; I haven't tried to test that and I don't know if there are any interesting platforms where it won't work. (GNU DLD may help to provide the facility on old systems -- see the Libtool manual.) * Use Modules should be written as normal for the Emacs core, just obeying a couple of conventions. The base name of the source file is the module name, MODULE, and it must export an initialization function init_MODULE. If the module provides a feature, its print name must be the module name. The initialization function does the job of both the init_ and syms_of_ types of functions found in the Emacs source files. (The module name can contains hyphens, in which case, replace them with underscores in the name of the initialization function. The name must be a valid C identifier otherwise.) Thus a module "mod-test" is compiled from mod-test.c, which defines the function `init_mod_test' and may provide the feature `mod-test'. When it's installed somewhere on `load-path', you can `(require 'mod-test)', `(load "mod-test")' or `(load "mod-test.la")'. See node `Writing Emacs Primitives' in the Emacs Lisp manual for rudimentary guidance on writing primitives, but basically you must grok the examples and internal documentation provided by the Emacs source. Note that programming errors in a module will typically crash Emacs and require debugging with GDB &c. (A useful breakpoint for the start of dynamic loading is `lt_dlopen', called from `Fload'.) A module's init function can safely signal a Lisp error, though it won't normally do anything that complicated. There's a simple example, mod-test.c, added to the src directory. You can build it using the installed script as build-emacs-module --dest DEST mod-test.c where DEST is the directory in which you want to install it, defaulting to the current one. See also `build-emacs-module --help' On a system like GNU/Linux which uses the ELF binary format, the installed files are `mod-test.la' (the Libtool description of the module), `mod-test.so' (the module binary) and `mod-test.doc' (the file of documentation strings extracted from mod-test.c in the same way as etc/DOC). On other systems, the .so file may be replaced by one or more others. If you don't specify a destination for the module, other files will be left in the build directory, including a directory `.libs'. The test module defines function `Y1', computing a Bessel function, as C-h f Y1 RET should say after you load it using (load "mod-test.la") or (require 'mod-test) A more realistic use of a module is to provide an interface to a database (e.g. using db, which needs need the relevant library specifying to build-emacs-module and available at load time). Another potential application is language-specific parsing functionality which is too slow in Lisp -- see comments in font-lock.el :-/. The lisp.h and config.h headers, which modules need to include, as well as the libtool script, are installed in the configured archlibdir directory. That's not obviously appropriate but it seems the only reasonable place to put them all. * Caveats There's no provision for versioning information (a la libfoo.so.1.2.3) any more than for Lisp libraries. I'm not sure how it would be useful and robust in practice. A module's init_ function could check it's being loaded into a sufficiently recent Emacs by looking at the value of `emacs-version' and signalling an error if necessary. Obviously incompatibilities could arise if interfaces and types change which are effectively defined by lisp.h. I don't think there's any interaction of dynamic loading and Emacs's conservative stack scanning (c.f. conservative heap scanning). I haven't checked the effect of dumping Emacs with a dynamic module loaded, but that's probably inadvisable. As for etc/DOC, a module's .doc file can't be moved or changed once the module's is loaded, or the help strings will go awry. All the header files that `build-emacs-module' needs may not be installed if you don't build with GCC since any included by the system- and machine-specific files used directly by config.h can't be found. I doubt it's reasonable to try to be more general, but suggestions welcome. If you really need other headers from the Emacs src directory, you'll have to install them by hand, but you should probably be wary of writing modules which depend on them. * Licensing Note well that since such modules are linked with Emacs, they must comply with the Emacs licence viewable with C-h C-c. Specifically, they may not be distributed only in binary form or if they interface to components which are incompatible with that licence. * Whence At the time of writing, this is available from