[Axiom-developer] Axiom Design musings

[daly]

Axiom is at the beginning of another major transition.

There have been several major transitions in Axiom which were the
result of design choices.

There was a transition from the NAG version to the open source version.  
The NAG version required a running Axiom to build Axiom. The open 
source version had to bootstrap from nothing. This involved design 
choices about building from scratch.  These choices were encoded in 
the makefile structure.

There was a transition from CCL to GCL. The CCL version was a highly
portable, byte coded Common Lisp subset. Portions of Axiom were hard
coded for performance. GCL was a full Common Lisp. At IBM Axiom was
rewritten to be portable among about a dozen Common Lisps so it could
be run on many platforms. GCL was hand-optimized internally to support
Axiom with the best performance. Porting from CCL back to Common Lisp
was a design choice to make the Axiom code conform to the Common Lisp
standard. This involved a large, low-level rewrite. The choices were
defined by the standard.

There was a transition from raw code files to pamphlet files. Raw code
files did not provide a suitable medium for communicating ideas from
one developer to another, leading to a loss of information in each
generation of developers. Pamphlet files restructured the system so
that raw code was contained in a document format which could provide
space for explaining (not "documenting") code.  This design choice
swept through all the files, resulting in individual, tiny pamphlet
files.

There was a transition from BOOT code to Common Lisp code.  BOOT was
ad-hoc and spoken by a few people at most. Common Lisp is a standard
and spoken by a large community. This design choice eliminated all
BOOT code by compiling it to Common Lisp. The resulting compiler
output had to be hand rewritten to human format. This transition is
still ongoing and is expected to be completed by the end of this year.

There was a transition to test automation.  The problem was that
testing was ad-hoc. This made it difficult to say what was different
about any instance. The regression test files were automated.  A
regression file format was designed and implemented. In addition, a
large scale Computer Algebra Test Suite (CATS) was built to create
test cases from published outside sources. CATS work is ongoing in
collaboration with Albert Rich, and Series and Products work using
Gradshteyn and Ryzhik.

There was a transition in release cycles. Release cycles were set on a
two-month cycle. A new transition to continuous integration and
continuous deployment is taking place (Jenkins and Docker).  The test
automation work supports this effort, making it less likely that bad
builds will reach the outside world.

There was a transition from Makefiles to Common Lisp build
mechanisms. Makefiles are unable to take advantage of information
embedded in the system. The Common Lisp build mechanism has global
information it can exploit, eliminating the need to keep track of
modifications, such as adding or reorganizing algebra code. This will
eliminate makefiles. This is ongoing and is expected to complete by
next year.

There was a transition from tree-structures of tiny files to
books. Tiny files are a historical artifact of tiny 1970s machine
memories. They splatter source code over a directory tree, depending
on directory and filenames for minimal semantics.  Books provide
higher level organization, allowing a flow of organization and
explanation of the ideas along with the associated code implementing
those ideas. This transition is still ongoing and is expected to last
as long as people find ideas worth explaining. A primary Axiom goal is
to provide a teaching platform for future computational mathematicians. 
It is important to communicate ideas and the book form supports this.

There was a transition to video. Written documents cannot easily
capture "process-like" actions, such as system maintenance and
debugging activities. They are also not good for showing the power
and ease of the user interface. A new set of video documentation
is being created and posted to the website.

There was a transition from C code to a browser. The C code depends on
X11 which is stable but fading. The universal front end of systems
seems to be moving to browsers. The Axiom read-evaluate-print loop has
been moved to the browser. The hyperdoc structure is partially
moved. The graphics code still needs to be rewritten into canvas
form. The Crystal structure needs definition. This work is ongoing.

There was a transition from symbolic to symbolic/numeric support.
Historically Axiom had support for the NAG libraries. This did
not survive the transition to open source. Work is ongoing to
rebuild this and other numeric capabilities as well as work on
support for numerical analysis to allow for facilities such as
explicit error bounding. (book volume 10.5)

There was a transition to literate support in the interpreter
so code and test cases could be automatically tangled from the
books and pamphlets. This support was used to improve the
automated testing (the tangle and regress commands).

There was a transition to full and automated cross referencing,
automated access to published literature, and bibliographic references
underlying the algebra. A repository of bibliographic papers was
created.  An effort is underway to build a geneology tree of the
algorithms in order to show the current Axiom state and suggest work
to bring Axiom up to the latest published state of the art. Work is
ongoing.


Now there is a new transition...

The majority of the code in the system now lives in books.  The first
steps toward organizing these code piles has begun.  The design goal
and the challenge is to make it possible to navigate the system in
some coherent way. The "mental model" is similar to what is found in
the preface of books explaining which chapters can be read and how
they depend on each other. Thought needs to be given to good, clear,
and well-supported navigation pathways and tools based on intent
(Crystal). Ideas need to be presented in hyperlinked graphics.

The current focus is on the Interpreter (book volume 5). There are
many things that need to be done to make it into a useful, readable,
and pleasant document. It represents a real design challenge.

Looking from the "top-down" there should be a seamless transition
from the Jenks book (book volume 0) and the Daly book (book volume
1), which are written at the user level, to the high level structure
of the interpreter. This can be thought of as a flat (user) surface
with underlying layers of support (e.g. the history mechanism)

====================( User Interface )============
    ====      ====     ====     ====    ==== ...
    ====      ====     ====     ====    ====
 (history)   (undo)  (reading) (eval)  (print)

Things like eval and printing are supported by the algebra

====================( User Interface )============
    ====      ====     ====     ====    ==== ...
    ====      ====     ====     ====    ====
 (history)   (undo)  (reading) (eval)  (print)
                                 (algebra)

and that is supported by the databases

====================( User Interface )============
    ====      ====     ====     ====    ==== ...
    ====      ====     ====     ====    ====
 (history)   (undo)  (reading) (eval)  (print)
                                 (algebra)
                                (databases)

But life is so much more complex than that. For instance, some of the
algebra is hard coded in Lisp, such as the coercions from one domain
to another. There is also algebra performance optimizing code written
in Lisp.

====================( User Interface )============
    ====      ====     ====     ====    ====
    ====      ====     ====     ====    ====
 (history)   (undo)  (reading) (eval)  (print)
                                |- (algebra)
                                +-- (databases)
                                |
                              (coercion, performance)

There is shared code used by both the interpreter and the compiler
(book volume 9).

====================( User Interface )============
    ====      ====  |   ====     ====    ====
    ====      ====  |   ====     ====    ====
 (history)   (undo) | (reading) (eval)  (print)
                    |            |- (algebra)
                    |           +-- (databases)
               (compiler)       |
                              (coercion, performance)

There is code to support "external code", notably hyperdoc,
graphics, and the browser

====================( User Interface )================
    ====      ====  |   ====     ====    ====     ====
    ====      ====  |   ====     ====    ====   (socket)
 (history)   (undo) | (reading) (eval)  (print)    +-> hyperdoc
                    |            |- (algebra)      +-> graphics
                    |           +-- (databases)    +-> browser
               (compiler)       |
                              (coercion, performance)

>From a "bottom-up" perspective there is code to support multiple
Common Lisps, infinite precision arithmetic, sockets, performance
macros, multiple semantics (MACLISP vs VMLisp vs Common Lisp). There
are subsystems such as the Burge parser and the Charybdis mathematics
printer which need to be packaged into coherent wholes, organized,
and explained. There is legacy code to purge such as the Saturn
and Aldor interfaces.

Since Axiom is a "typed system" there needs to be an effort to
recognize, name, classify, categorize, and explain the types of the
data structures in the interpreter. Work has started to provide every
Lisp function a "signature line" using these types, just like the
algebra above it.  The lisp signatures need to compile to optimization
declarations for the Lisp compiler which will improve Axiom
performance.  Code needs to be rewritten to be more functional rather
than modifying structured arguments when it makes sense. This will
support the future "proof system" effort.

All of this needs to be thought about and organized in ways that will
support understanding. Developers need to understand the printing
mechanism. Mathematicians need to understand the coercion interpreter
system. Users need to understand the undo mechanism.

This latest transition is slowly taking shape. It will involve
massive writing efforts and massive code refactoring.  It will involve
building things that exist nowhere such as a mechanism to hyperlink
and cross-reference everything to everything (Crystal).

There are future transitions "in plan", such as developing a proof
mechanism (book volume 13). There are tools to provide proofs at
various interfaces (e.g. math to spad (COX), spad to lisp (ACL2), lisp
to C (LLVM tool), C to machine (FX)).

The current "gold standard" for literate software is the Pharr and
Humphrey's book "Physically Based Rendering" which I highly recommend.
If the algebra was written in that style it would make it easier to
understand the algorithms and the underlying mathematics.  But we must
raise our eyes past that level as Axiom is well over a million lines
of code and multiple books.

That "30 Year Horizon" looks mighty close.

Stay tuned....

Tim