Re: [Axiom-developer] sbcl and Axiom

[C Y]

--- root <address@hidden> wrote:

> CY,
> 
> ============================================================
> BUILD ORDER

> Notice the '32' prefix which gives a clue about which make stanza it
> is. This line is output from the Makefile running in the 
>   src/graph/view2D
> subdirectory. If you look at the src/graph/view2D/Makefile.pamphlet
> and search for 'echo 32' you will see the stanza:
> 
> ${MIDOBJ}/write2d.o: ${MIDINT}/write2d.c ${LINC}/write.h ${HEADERS}
>       @ echo 32 making ${MIDOBJ}/write2d.o from ${MIDINT}/write2d.c
>       @ ( cd ${MIDOBJ} ; ${CC} -c ${CFLAGS} ${MIDINT}/write2d.c )
> 
> This stanza says: 
> 
>   if you need to build  ${MIDOBJ}/write2d.o
>   then you must first build
>      ${MIDINT}/write2d.c 
>      ${LINC}/write.h 
>      ${HEADERS}
>   once these have been built it echos the '32' line and then
>   it builds ${MIDOBJ}/write2d.o 

Ah!  Thanks!  That makes sense :).


> ============================================================
> BUILD-TIME DIRECTORY STRUCTURE
> 
> 
> There are 4 top-level directories 
>  ${SRC} -- the source tree (human written)
>  ${INT} -- the intermediate (machine independent, machine generated)
>  ${OBJ} -- the object files (machine dependent, machine generated)
>  ${MNT} -- the final ship tree
> and within each Makefile combinations of these make local variables.
> 
> The system is set up so that it is possible to copy the SRC and INT
> directory to another system and restart the build. Historically this
> was important because it used to take 3 weeks to build a system from
> scratch (well, not scratch but from a previous system).

Yikes.

> The makefiles are designed so you could mount NFS disk space on
> multiple systems and build them all in parallel. Axiom used to build
> on PCs, Symbolics, AIX, Solaris, and VM/370 all from the same 
> SRC and INT. The target system is named by the ${SYS} variable,
> these days usually just linux. The target system is determined from
> the $AXIOM variable so:
> 
> export AXIOM=/tmp/axiom/mnt/linux
>                             ^^^^^
>                              the target
> 
> and you could build others by NFS mounting the source directory and
> setting the AXIOM variable such as:

Is this still a desirable feature, to be able to compile OS targets
from a single copy of the source tree?  I must confess I've never run
into such a situation - each machine I've ever installed on gets it own
copy of the source - but I suppose that is rather inefficient.  Most
lisp programs seem to compile their binary files alongside the source
files, although come to think of it Garnet also moved things around
somewhat.  I'll have to find out if asdf supports moving the output
files into another directory.

> export AXIOM=/tmp/axiom/mnt/solaris
> export AXIOM=/tmp/axiom/mnt/freebsd
> export AXIOM=/tmp/axiom/mnt/symbolics
> export AXIOM=/tmp/axiom/mnt/aix
> etc...
> 
> 
> So the basic idea is that you start from the sources ${SRC},
>   cache a lot of machine independent work in ${INT},
>     use ${OBJ} as a machine specific temp area (.o files, etc),
>       and create a machine-specific "ship" system in ${MNT}.

OK.  So fasl files (the sbcl compiled file) should be targeted to
${OBJ}.  

> Once the machine-specific directory exists (${MNT}/linux, ${MNT}/aix,
> etc) you can throw away everything else, including the sources.

Got it.

> ============================================================
> BUILD-TIME INTERMEDIATE IMAGES
>
> So the key steps that use these images amount to:
> 
> step 0: build noweb

OK.
 
> step 1: build a lisp with our patches ==> lisp
>   if we had a fully patched lisp (or did not depend on any 
>   special features) we could just copy a lisp rather than build one.

I'm hopeful that we won't need to recompile sbcl et. al., but I admit
I'm not nearly expert enough to tell for sure yet.

> step 2: build a lisp with the boot translator ==> bootsys
>   extract the boot translator clisp files from the boot files into
>   INT
>   load the boot translator into the lisp image
>   save it as bootsys

OK.  So step one is accomplished using the lisp code contained in the
bootdir boot.pamphlet files, which allows the lisp to then translate
the bootdir boot code into clisp files?  (e.g. the "bootstrap" step?)
 
> step 3: translate (use bootsys)
>   translate the boot files into clisp files using bootsys
>   after this point bootsys is useless and we abandon the image

Is there any reason not to just use the same image and delete the boot
and boottran packages?  (I suppose from the make standpoint it's six of
one, half a dozen of the other.)

> step 4: build a lisp image with all the macros ==> depsys
>   we use lisp macros which are needed at compile time but not runtime
>   we load those macros into a lisp image and save it as depsys

Is eliminating these macros from the final image done to save memory?

> step 5: compile lisp (use depsys)
>   we use depsys to compile the clisp/lisp/lsp files 
>     clisp is boot-generated code
>     lisp is lisp-original code
>     lsp is GCL generated code

Ah ha!  Thanks.  I hadn't made the connection between lsp and GCL.

> step 6: build a lisp image that contains the interpreter/compiler ==>
> interpsys
>   we load all the compile files into lisp
>   we can preload some algebra code for efficiency
>   and save it as interpsys (eventually copied as AXIOMsys into
> ${MNT}/${SYS}
> 
> step 7: autoloads (use depsys)
>   we have a set of files that are used in special cases but are not
> needed
>   we use depsys to compile these into the autoload subdirectory.  

OK.  It's starting to make sense now.  Since we don't want to hold the
whole Axiom system in memory, we dump the build image and create a
smaller, less hefty image and use the autoload system to get what is
needed.

> step 8: algebra (use interpsys)
>   using interpsys, compile the algebra 

OK.  I take it this part should be the least sensitive to
cross-platform issues, given a properly running interp?
 
> step 9: we may have to do deep debugging ==> debugsys
>   this is purely for my use. sometimes the only way to find a 
>   bug is to run the interpreted lisp code. 

OK.

> Just to add to the pain each of the source files are documents so
> we have to extract the required sources at each and every step.

Indeed.  I set up a convenience feature where I just run notangle in
advance on all pamphlet files in a directory before I start working on
it (so asdf has lisp files to work with), but that's probably not the
best idea.

> ==============================================================
> BOOT HISTORY
> 
> 
> Boot is necessary to translate the boot files to clisp files.
> 
> It is not normally used in a running system. However, boottocl
> is a command line tool that allows you to dynamically translate
> a boot file to a clisp file. This is a way of making dynamic
> changes to the sources (it's lisp, after all). The system used to
> be built on top of an existing system. Our style of working was
> to keep modifying the system while it was running. Thus the boot
> code was changing while we worked. Eventually this led to a system
> that could no longer be built from scratch but required a running
> system to build. Boot is one example of this problem. The boot
> compiler is written in boot. In order to start the process we now
> cache the generated lisp files in the original documents. Hacking the
> boot compiler would require re-generating and re-caching these files.

OK.  I thought so.

> The boot code is slowly disappearing from the sources.  Once that
> happens the boot subdirectory is no longer necessary and boottocl
> will disappear. As you can see from the steps above boot adds
> complexity we don't need.

Amen.

> FUTURE DIRECTION
>
> I'm rewriting and documenting the system internals into straight
> common lisp. At the moment I'm working on the interpreter which
> will eventually become the 5th volume of the axiom series (hence,
> src/interp/bookvol5.pamphlet). This fully documented interpreter
> will mirror the existing system. Once it is documented we can
> modify it to suit future needs.

OK.  So trying to monkey with this level of code at this stage sounds
like a Bad Idea.  I take it when the transition is complete the entire
interp system will be in bookvol5.pamphlet and the others will go away?

> My ultimate aim is to make the system a literate lisp program that
> only needs to be compiled and run in a single pass on any ansi
> standard lisp.

Ah, OK.  Having gone just far enough to begin to appreciate the
gargantuan nature of this task, I salute you.  I'm beginning to suspect
there is little we can do to help you here, but if someone wanted to
try what task(s) would be most useful to your effort that are still
doable by others?  If the first version of bootvol5 will mirror the
current system is it pretty much a strict "clean up and document the
generated Lisp" process?

> The lisp ASDF facility will allow us to remove the
> makefiles and the autoconf/autoload machinery. Boot will disappear
> as will the many lisp images, intermediate directories, etc. Lisp
> will be extended with native notangle/noweave so you can just
> load or compile literate files.

OK, cool.  

> The next version of axiom will consist of a set of volumes that are
> literate documents, fully explaining the code. The compiler will be
> its own volume and can be replaced by a different compiler.  The
> interpreter will be its own volume and can also be replaced (possibly
> by an aldor standalone version).  The algebra will be organized (in
> some fashion) as volumes that fully document that algebra. Axiom will
> sit on top of an ansi common lisp literate base (or aldor standalone 
> base if we go there).

Personally I would hate to see us move completely away from lisp
(although I guess I should wait to see how aldor looks).  IIRC, the
Aldor compiler is written in C?  (arrgh.)

> The build process should involve just a drag-and-drop of the volumes
> and any literate papers containing algorithms you might want to use.

Sounds good!  If there is anything I can do which will help you with
this Tim, please don't hesitate to ask.  

Cheers,
Cliff

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com