[Axiom-developer] Re: noweb vs. leo

[Norman Ramsey]

 > On Tuesday, May 09, 2006 4:38 PM Norman Ramsey wrote:
 > > ... 
 > > I'm afraid you can have 'simple and understandable' or
 > > you can have 'precise'. You can't have both.  The man
 > > page represents the usual muddled compromise that provides
 > > neither.
 > 
 > I don't mean for the following commments to be interpreted
 > as critical but the above statement from someone who has
 > devoted a lot of effort to literate programming concerns me
 > greatly. Perhaps even, this illustrates the one major "fatal
 > flaw" in the whole concept of literate programming...

I wish now I'd made it clear that this statement is not intended to
cover all documentation for all programs but rather the narrow issue
of noweb's lexical analysis.  The fact is that a great deal of
complexity was introduced into the algorithm in order to enable a
great many common cases to work without requiring users to think about
noweb's markup, and while still enabling noweb to issue error messages
on inputs that are likely to be in error.  It is the complexity of the
algorithm itself that makes 'simple and understandable' incompatible
with 'precise'.  And the complexity of the algorithm in turn arises
from the complexity of the problem, namely to provide human-editable
markup that will serve the common cases well while still making it
possible to express any input.


 > The reason that this methodology is not in wide use so many years
 > after its invention:
 > 
 > Writing *good* documentation is still hard work - harder
 > even than the programming itself - and the first generation
 > of literate programming tools (of which noweb is a prime
 > example) seem to do nothing at all to make this easier.

I have said for years that if you want better documentation, don't
look to your tools---look to your *process*.  The good news is that
almost any kind kind of process will work: you just need multiple eyes
to review the documents, and you need accountability to be sure that
revisions are done and checked in.  It may well be that tools can help
with this, but they will be the same kinds of tools that are used to
support many other sorts of 'computer-supported cooperative work'.

 > And as Ralf pointed out recently that we have a similar
 > situation right now in the Axiom literate program source
 > files - the information might be there (e.g. in the
 > makefile.pamphlet) but it is not very accessible because
 > it lacks a simple and easily understood overview.

Always the first thing to go wrong with any large literate program. 
The effort involved in providing such overviews---and keeping them up
to date---is *enormous*.

 > The only literate programming tool that I am aware of that
 > does attempt to address this problem - at least in some
 > limited degree - is Leo [with its outlining facility].

We have found that for a moderately large project (say over 10,000
lines of code) an outline (or even a set of outlines) is not enough.
We rely heavily on a LaTeX table of contents (our best analog to a Leo
outline), but we also rely on diagrams and overview chapters.
This is all very expensive in time and effort, and I would say that in
my research group we seem to be able to afford this effort only
roughly every 5 to 10 years.  Of course we are a very small shop; your
mileage may vary.

 > I think this is very similar to what Tim Daly has written
 > about the need to view large complex literate programs like
 > Axiom from the point of view of a "crystal" with multiple
 > facets.

Also quite interesting.  And expensive, as every facet must be polished.

 > Organizing a project to take best advantage of all this involves
 > some considerable effort. I am continuing to try to work up the
 > energy to tackle Axiom with this tool.

Too true, and good luck :-)


Discussions of these sorts of questions used to enliven
comp.programming.literate.  May I have your permission to post this mail?


Norman