[Axiom-developer] Crystal: literate programming and "shape shifting"

[Tim Daly]

I'm pondering a new (i.e. I have not seen it anywhere else but
I'm sure it exists somewhere) idea of "shape-shifting documents".
By year end I hope to have all of Axiom fully literate so all
that remains are the books containing the sources. This is still
just the 0-th layer of documentation. Despite the huge pile of
refactoring that remains to be done it is time to ponder how to
present the information in a future-proof way.

I've recently found
http://literate-molly.sourceforge.net
which is an html/literate folding document structure. Since I
already define heading names (e.g. \pagehead) in the latex sources
it would be trivial to name them with HTML-like syntax and map them
to molly-like layouts for folding.

I've used folding editors before and they have some advantages
and disadvantages but I like the idea. Unfortunately, I don't think
the idea is general enough. The Crystal concept wants user-definable
facets that present information in various ways and a folding table
of contents is only one of them.


What I'm striving for, as part of the Crystal idea, is a way to
shape-shift a document. I'd like to be able to view a document
in several different ways including:

 * differing hierarchy
  The idea here is to show the same document with different
  "table of contents" style viewing. I'd like to have a
  meta-document format that allows me to specifiy the way
  that the document is viewed. For instance, I'd like to be
  able to present the Axiom algebra from a
    - "source code view" which shows the source and allows
      you to navigate from source to source
    - "teaching view" which shows the explanations and the
      source structured from a tutorial view
    - "developer view" which pairs the source with the
      structures and the caller/callee tree
    - "problem view" which restructures domains to show how
      to solve particular problems in particular subjects
      like physics, chemistry, robotics, etc.

 * differing faces
  The idea here is to show the same document with different
  navigation tools, such as a set of graphs that lay out the
  contents by code orientation:
    - caller/callee, the usual call chain
    - reference/use, similar to caller/callee but with data
    - index, similar to a book index
    - keyword in context, strings rotated to have keyword first
  or by content orientation:
    - subject, a "concept" index
    - technique, a programming layout
    - references, a literature lookup
    - algorithm, a "how is it implemented" description
  or by performance:
    - time, as measured by running the program on this machine
    - space, to examine the cache and memory footprints
    - resource, to examine the cpu/disk/network footprints
    - comparison, such as determinants vs pseudo-inverse
    - scaling, such as running on 1,2,3,...,n processors
    - roundoff/numerical sensitivity/interval analysis


The architecture of such a system would have to be very clean and
very generic. There needs to be a good separation of content and
structure so anyone can create a graph of any data along with a
definition of the navigation primitives (as function closures?)
and it should "just work".

I'd like the representative structures to be both user definable
by special sections in the literate document (think of a list of
"alternate table of contents" sections) as well as computed sections
such as caller/callee and performance.

The navigation, such as folding, hyperlinking, linear text layout,
graphing, etc should all be available to any data I supply just by
moving from one crystal facet to another. This is similar to a
model-view-controller where the view is chosen by the face, the
controller is supplied by function closures, and the model can be
chosen from maps in the literate text.

I suppose it would be worthwhile to spend time studying Edward Tufte
and classifying the crystal facets by display technique. The idea
would be to figure out an "object model" for each kind of Tufte display.
The model would then have "methods" that would need to be supplied to
generate the display from underlying literate data. These methods
could be supplied as function closures mapping from data to display.


Another facet of the architecture is the tight coupling of the Crystal
structure with the implementation. I'm in the process of a complete
re-factor of Axiom into a form that is both "embedded"
(that is, the Axiom algebra can be viewed and used as ordinary
lisp functions and lisp functions can be viewed and used as ordinary
algebra functions) and "closed" (that is, the functions can be
composed with each other so that the total mapping is well-defined).
Both properties are straight out of Abelson and Sussman's Structure
and Interpretation of Computer Programs.

With these properties and a tight coupling it should be possible to
use algebra in the display closures, making it possible to run real-
time comparisons, timings, etc. It should also be possible to embed
the display closures in algebra functions to enable real-time actions
such as graphing.

Documents in a fully literate system are not just merging text and
code. The game is to steal the lisp idea of a homoiconic language
(program==data). We want to define the concepts so there is a fluid
boundary between the text and code where each can "embed" the other.
If it all compiles down to lisp S-expressions its all mutable.

Ultimately, its all "just lisp", right?

Tim