RE: [Axiom-developer] out of source builds

[Weiss, Juergen]

My 5 cents worth of advice:

When I had a look at the build system of Scratchpad maybe
15 years ago, I got the clear impression, that it did
not follow best practice at that time -- it certainly
does not follow best practice today. So we should be
glad, that some people took the challenge to rebuild
it according to methods (autoconf et al.) which are
best practice today. We should move to the new build
system as soon as possible. 

Regards

Juergen Weiss

P.S.: We should adopt best practice in other areas as well, namely
the way source code control systems are used. Use one (SVN) in
a way, that the hassle of other contributors is not 
maximized, but minimized. And we should certainly do
EVERYTHING to keep developers like Gaby and Waldek. I value
their contribution higher than all the LP stuff of the
last 5 or so years.

Juergen Weiss     | Universitaet Mainz, Zentrum fuer Datenverarbeitung,
address@hidden| 55099 Mainz, Tel: +49(6131)39-26361, FAX:
+49(6131)39-26407
 

> -----Original Message-----
> From: address@hidden 
> [mailto:address@hidden
>  On Behalf Of address@hidden
> Sent: Sunday, July 01, 2007 9:13 PM
> To: address@hidden
> Subject: [Axiom-developer] out of source builds
> 
> I'm not sure why out of source builds are particularly important. But
> there are two points to discuss.
> 
> The first point, mentioned in a prior email, is that there is a 
> general class of requirements:
>   (b) doing it because that's the way everybody does it
> and this seems to be of that character. If there is a different
> need anyone can point to as a good argument why this is important? I 
> understand the CD argument as I put Axiom on the ISSAC CDs.
> However this is a rare situation for distributing any open source
> software these days. I cannot remember getting sources for an open
> source project on CD. 
> 
> (Caveat: I had to recreate the whole build subsystem from scratch
> so some of these features are not fully tested but they are there
> by design. In particular, the newly added literate facilities are
> not correct yet (e.g. Makefiles are extracted to "src" instead of
> "obj/sys", which is a bug that needs to be fixed))
> 
> The second point is about the claim that the current build system
> cannot build "out of source".  By design the "src" tree should never
> be touched by Make so you should be able to mount the source read-only
> and type Make. Any failures are a bug.
> 
> The current build system IS designed to do out of source builds. To do
> this you need to notice two things. First, the top level Makefile
> contains the lines:
>   SRC=${SPD}/src
>   INT=${SPD}/int
>   OBJ=${SPD}/obj
>   MNT=${SPD}/mnt
> 
> Second, notice that all of the file manipulations in the Makefiles
> are prefixed by paths built from these 4 variables. Thus, to do
> out of source builds it should suffice to just change these:
>   SRC=${SPD}/src
>   INT=${MYHOME}/int
>   OBJ=${MYHOME}/obj
>   MNT=${MYHOME}/mnt
> 
> All other Makefiles create IN, MID, OUT, DOC, etc from these. For
> example see src/interp/Makefile. At the top you'll see:
>   IN=${SRC}/interp
>   MID=${INT}/interp
>   OUT=${OBJ}/${SYS}/interp
>   DOC=${MNT}/${SYS}/doc/src/interp
> and the SYS variable is gotten from `basename $AXIOM` which is
> a name to distinguish the type of target to build.
> 
> 
> 
> 
> 
> 
> There were other issues back in the scratchpad days. Since a build 
> could take weeks it was vital to architect the build system so the
>   (1) builds need to be done minimizing the re-work.
>   (2) builds could be done in parallel, out of source, 
> multiple targets
>   (3) builds only built the request parts
> The current build system is designed to do all three.
> 
> MINIMIZE REWORK
> 
> Requirement (1), that builds need to be done minimizing the re-work,
> is accomplished by the "int" subdirectory. Because Axiom uses boot
> there is a whole translation stage needed to extract the source code.
> The "src" source tree gets extracted into the "int" source tree. Many
> steps, such as the Boot->Lisp translation are system independent and
> only need ever be done once. Unless the boot code is changed there
> is no reason to redo the work and waste the time, which is the point
> of using Makefiles at all.
> 
> The "int" directory represents an Axiom build that considerably
> speeds up the build process. The "int" directory
>   (1) caches the Boot->Lisp translation (removes the need for 
>       running bootsys, thus removing a whole build "pass".)
>   (2) caches the .fn files for code optimization (allows optimal
>       code to be generated on second compiles which improves the
>       execution time of the final system)
>   (3) caches the algebra system-independent translations
>       (the .nrlibs contain several files including .lsp files
>        which are post-algebra compiles. since algebra translation
>        is an expensive step this is a huge savings)
>   (4) caches the pamphlet weave output and tangle output (removes
>       the need to run noweb at all)
>   (5) caches the latex aux files so latex need only run once rather
>       than twice for every file. (halfs latex time)
> 
> In theory you could ship the "int" directory and do a complete build.
> With proper caching this could reduce the whole build time 
> considerably,
> probably to less than half an hour.
> 
> There has been much discussion about how long builds take and this is
> a real concern. So careful attention to the system-independent cache
> would create a huge savings of time, reducing single-file changes to
> just a minute or so.
> 
> Make needs to pay close attention to doing rework only if a source
> file changes. Otherwise Make can use the cached information in "int"
> and reduce the build times by large factors (e.g. no algebra 
> compiles).
> 
> I doubt anyone notices this feature because they probably do
>    make clean
> which destroys the cache. However Axiom is designed to do incremental
> changes. You should be able to modify any source file, type 'make'
> and have the system only rebuild the parts that are affected. It used
> to do this and is designed to do this. Thus the issue about long
> system builds should only happen on new systems. If you can't minimize
> rework what is the point of using Makefiles?
> 
> After the first time you should almost never need to rebuild Axiom
> from scratch, at least by design of the current build system. We
> rarely did in the scratchpad project because it took weeks.
> 
> 
> 
> 
> 
> 
> PARALLEL BUILDS, OUT OF SOURCE, MULTIPLE TARGETS
> 
> Requirement (2), that builds be done in parallel, was needed so that
> multiple machines could work on the same source tree. Since Scratchpad
> was build on wildly different architectures (Symbolics, IBM/RS6000,
> IBM PC, SUN, VM/370) and wildly different lisps (Zetalisp, CMUCL,
> AKCL, Golden Common Lisp, etc) there was a lot of build time involved.
> 
> The solution to this problem was the "obj" directory and NFS. By
> design, if you do a build on any one system you will do 2 things:
>  (1) you will cache all the rework into the "int" directory
>  (2) you will create a new directory (e.g. obj/SUN, obj/RIOS)
>      for that particular target system.
> 
> Once you have a valid system build life is good. All you have to
> do on a new target system, e.g. PC386, is:
>   make a new directory, 
>       mkdir AX
>   use NFS to network mount the old src and int directories readonly
>       mount -readonly NFS:/home/daly/AX/src AX/src
>       mount -readonly NFS:/home/daly/AX/int AX/int
>   set the AXIOM variable
>       export AXIOM=AX/mnt/PC386
>   and run make
>       make
> 
> Since the build system knows about int and the int files are newer
> than src the build on the new system, PC386, only needs to compile
> the lisp files for the interpreter and algebra. It only needs to 
> compile the C files for graphics and hyperdoc. All of this work went
> into the "obj/sys" directories. If the master source changed all you
> had to do on the other systems was simple rebuilds.
> 
> This would be very useful if and when we have a compile-farm setup
> with many different target architectures and different lisp systems.
> All you need to do is a network mount and everybody builds from the
> same src/int tree.
> 
> 
> 
> 
> 
> BUILD ONLY REQUESTED PARTS
> 
> The requirement (3) build only the request parts is partially
> re-implemented but not fully. Look at the top level Makefile for
> the lines:
>    PART= cprogs
>    SUBPART= everything
> 
> This information is passed in the ENV variable. Using this the
> lower level Makefiles can skip whole subdirectories. You used to
> be able to specify that you only wanted to build hyperdoc with:
>    make PART=hyperdoc
> and you could specify subparts, e.g. the hyperdoc pages
>    make PART=hyperdoc SUBPART=pages
> 
> 
> 
> 
> 
> 
> When I recreated the build system (post NAG) I was not exceptionally
> careful to ensure that the "int" directory and "obj" directories
> where especially "pure" but that is a trivial fix. It only requires
> prefixing the output filenames in the makefiles with the right paths.
> This is maybe a day's worth of work. Recovering the "PART" ability
> is also reasonably simple since it only requires checks in Makefiles
> before calling Make on subdirectories.
> 
> Axiom has been built for years and no-one has mentioned out of source
> builds before. If this is a real issue then the current system is
> designed to do it (modulo Makefile mistakes).
> 
> Tim
> 
> 
> 
> 
> 
> _______________________________________________
> Axiom-developer mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/axiom-developer
>