Re: [Axiom-developer] Terms of Surrender (was: BAD tim)

[William Sit]

root wrote:

> sorry, but you are reasoning by wrong analogy.
> 
Let me try another analogy. Ever use MS Word to generate an html page and look
at how inefficient the generated html source is?  Well, I hate that every time I
see such html code, and I myself never used that: I hand code html pages using
only html tags. The result is a very clean, readable code that I can maintain.
Those pages arn't fancy, but they are not too plain either (see
http://www.sci.ccny.cuny.edu/~ksda/ for a seminar site I helped design). Why?
Because I don't like Word, I don't know the fancier programs like FrontPage, I
don't use Frames, etc. So, I am LIMITED in my ways to construct only utilitarian
pages. However, I do not argue that hand coding html pages is the BEST way. That
depends. Hand coding requires a good knowledge of html (an example: relatively
few people know that html can do superscripts and subscripts, and do most of the
Greek alphabets without loading a language module). But nowadays, one can use
high level tools like MS Word or Mathematica to generate equations, or FrontPage
or a host of other Web tools to create much more complicated pages and while it
is possible to boil them down to a clean efficient handcoded html page with the
same features, it is very likely the amount of effort is not worth it. An artist
now can design a fancy layout by concentrating on the design rather than the
technicalities of font tags, table tags, picture formats, etc. That is why
almost anyone can design a home page without much training in html at all. Don't
we want the same for Axiom, that anyone who wants to use symbolic computation
can use Axiom without training in lisp?

> the interpreter needs to be:
>    simple to build,
>    simple to maintain,
>    simple to extend,
>    simple to port.
> 
> writing boot in boot forces the system build to be more complex. [BOOT evils 
> snipped]

Let's say I agree with your assessment for the evils of Boot (I don't have the
qualification to verify your claims, but I am fairly convinced by Bill Page's
rebuttal in his message of Sun, 6 Nov 2005 10:36:03 -0500; but let's put this
aside for the moment). Can you please lay out your vision of how these evils
will be eliminated by giving us an overall picture of what your ultimate
architecture is for Axiom? What will be the remaining components and how do they
interact? What is the booting sequence? What would be required of developers to
maintain the system (besides cloned GOOD TIM)?

You quoted from Bill Page:
> > Now about this language war ...
> > [snipped]
> > So, *if* I was thinking of such a compromise then there are
> > some "terms of surrender" that might make this easier:
> >
> > 1) Before any more "lispifying" goes on, *first* move Axiom
> >    over to ANSI lisp. This will make it possible to write lisp
> >    that conforms to modern practices and as high level as
> >    possible. It will also make Axiom less dependent on a
> >    particular implementation of Lisp and more easily portable.
and said: 
> so you're suggesting that we modify the boot compiler to generate ansi
> lisp code? yet during the "lispifying" rewrite i AM rewriting to ansi
> lisp.

I don't see any conflict in modifying the boot compiler to generate ansi lisp
(or any other type of lisp based on some switch) and rewriting some parts using
ansi lisp, as long as there are good reasons to do so. I agree with your goal to
write for portability, but at some point, portability has to be of lower
priority, especially before the Axiom system become stable, well documented, and
the bugs corrected. What is the value of new users trying out Axiom on multiple
platforms and finding it to be unreliable computationally or difficult to
understand the outputs (there are many such very simple examples on
IssueTracker). Many will simply abandon Axiom and go back to the M's.

> >
> > 2) No more just cutting and pasting the generated lisp output
> >    of Boot into the literate document! This seems pointless
> >    and a net loss in intelligibility even if it is accompanied
> >    by a verbal explanation of the code. The original Boot code
> >    is, as Tim admitted above, significantly more readable as
> >    "pseudo-code" then what the compiler outputs.
> 
> the cut and paste has to happen. it's the only way to make sure the
> code does what it originally did. successive passes of the code have
> been complete rewrites. you'll see this at the next code release.
> there are a LOT of subtle issues that require careful thought and
> having working lisp code, even clipped code, is a necessary guide.
> 
> believe me, i find the boot compiler output code even more painful
> than anyone else. but it is important to move the code from a
> correct state to a correct state during the rewrite and i'm not
> man enough to do it all in one go.

Your care to maintain the correct state is admirable and for your purpose, I can
understand why you put the boot compiler output code in the pamphlets. For
proper balance, as well as documentation, before you jettison Boot altogether, I
think the boot code generated from Spad should also be included (if available,
since I don't know if Spad compiles to Boot first or not).

However, I think this miniproject of yours is exactly like someone wanting to
streamline html code generated by say MS Word. I have tried this myself, and I
started with the generated source and then gradually pairing down the redundant
or worse, irrelevant code. But it is tedious, error prone, and in the end, other
than getting a file that is much smaller, I wasted hours of my time. I don't do
that any more: hard disk space is cheap and CPU's are fast. I don't notice a
difference other than I still hate the generated code mess. But I understand
that whoever designed that html page did not start from a blank page with html
code and will not maintain it by reading the generated code. If they need to
make changes, they will be using the high level tools because they trust (and
don't care about) the generated code. The problem to streamline generated code
can be solved by optimization (even though often that is not done or tools are
not available). Perhaps you will consider writing an optimizer for the boot
compiler instead? 

> (btw, as a side-effect of the rewrite i've already found and
> corrected two bugs).

In a production environment, language compiler bugs should be rare.
Besides, are you sure it is the rewrite that helps you find the bugs and not the
understanding of the Boot code before rewriting? Did you discover the bugs
before or after the rewrite? If before, did you try fixing it in the boot code?
If not, why not? If it is AFTER, did you also try if it can be fixed in the boot
code? In this case, I also seriously worry that your rewrite may not be
faithfully rendered (assuming your translation is exactly logically equivalent
as far as Lisp goes, but somehow the Lisp compiler handles them differently and
the object codes differ). This time, you may be fixing two bugs, but next time,
it could be introducing two bugs. Are you willing to dig into the Lisp compiler
to figure out the differences?

> if you tried to maintain the system as it is now you'll find that
> the boot compiler output is all over the source tree. try this:
> 
>   cd axiom/src
>   fgrep -r DEFUN *
> 
> the boot compiler generates an uppercase DEFUN. look at how
> pervasive the compiled lisp code has to be to actually get the
> system to bootstrap. in every file where the DEFUN occurs
> the BOOT code is just PSUEDO-CODE.
> 
> in particular,
> 
>   cd axiom/src/boot
>   fgrep DEFUN *
> 
> and you'll see how many of the boot compiler files aren't REALLY
> written in boot.

I am not following your line of thought. Isn't the role of the boot compiler to
compile Boot into Lisp code? So if the Boot code can use SOME embedded Lisp as a
subset of the language, what is the problem? Are you suggesting that if one only
uses Lisp (and get rid of Boot) that there will be fewer DEFUN's?

> believe me, it took a long time with a lot of effort to get the
> system to build without a running axiom system.

I don't for any moment forget what you put into the project to make Axiom alive
again. To that we all thank you. But we now have a running axiom system and why
must we continue to build without a running axiom system? Can't we ship Axiom
just like NAG shipped it (minus whatever is not open-sourced)?

>  and it ain't pretty. but it should be.

Beauty is in the eyes of the beholder.

[snipped]

Quoting from William Sit:

> > These terms should not be negotiable! Actually, I would put these as
> > requests to whoever does the work) I would add at least
> 
> ummm, i'm doing the work :-)

Hope that you are not the only one forever.
 
> and i REALLY don't want to rewrite the boot compiler to make it do
> things like ansi common lisp generation. especially if, after having
> achieved the 3 *terms* i'd would then have agreement that i could
> elide boot from the system. what would be the point?

But so would be documenting the Boot code. It is kind of "on the path"; like a
rocket fuel tank that has to be jettisoned after launch. That does not mean the
fuel tank is pointless.

> > 4) The error messages outputted by the compiler should NOT refer to any 
> > Lisp.
> > Nothing like the current $m, $x, $f stuff. All errors should refer as close 
> > to
> > the offending source code as possible. (I believe such output was due to
> > "laziness" or viewing more intelligible messages as an insult to the 
> > developers.
> > But that might be why Axiom failed to capture market shares in the 1990's.)
> 
> the output uses the variables like $m because that's all the developers
> need to know to debug. $m is the target mode information that the
> compiler is trying to achieve.

No, unless the developers also know Boot and Lisp. Often the values for $m, etc
have no relation to where the bug is in the source.  While breaking, I got into
Boot, and the data structure (usually raw) is presented by such a long string
that I quickly got lost. It is necessary to know deeply how the source is
translated into Lisp. On the other hand, understanding the logic at the source
and often the type system in Axiom was what saved the day. For engineers,
scientists or mathematicians who are not also computer scientists, this is a
huge hurdle to overcome.
 
> i agree that the compiler needs to be rewritten to do things like
> generate better error messages.
>   to do THAT level of rewrite requires a deep understanding of the compiler.
>     to achieve THAT level of understanding we need to deeply document how
>     the compiler works.
>       to do THAT level of documentation we need to review each function
>       in detail.
>         to do THAT detailed level of documentation is my current goal.
>         as a side-effect of achieving that level of understanding the
>         code will be in lisp (which is a language i deeply understand.)

How does error messages from the Aldor compiler compare? If it  has already
achieved THAT level, may be that's the way.

> > 5) The )trace function should likewise return high level state of the 
> > algebra
> > variables, not lisp code or representation of them.
> 
> admirable goal. BIG changes to the interpreter and compiler data
> structures (since the code is COMPILED into machine language) to
> maintain state information that trace can use.
> 
> and it needs to be done without impacting efficiency, accumulating
> large amounts of possibly unneeded state, etc.
> 
> this is like asking C to give you only algorithmicly related error
> messages.

What we need is a development kit with tools to set breaks and examine Axiom
variables, displayed in mathematical notations, not Lisp data structures down to
the nitty gritty. The translation tools are definitely there already (in coerce
to OutputForm) and it is a matter of whether those tools can be called from
within Lisp or Boot.  Since Axiom algebra code is highly structured, it should
be possible to dig (zoom) into substructures for debugging purposes, but
substructures need not be automatically revealed. Such tools are routinely
available for other common languages.

> > 6) The interpreted language should adhere to the compiler language to 
> > facilitate
> > prototyping (in the interpreter) and production code (using the compiler) 
> > and
> > any deviations (due to the smartness of the interpreter) should be logged to
> > help the transfer (maybe it is even possible to translate a working 
> > interpreted
> > set of routines automatically into compilable code?).
> 
> we need to "reify" (name) the interpreter type-translation, guessing
> and manipulation mechanisms. we need to separate them from the
> interpreter itself so we can understand, add or modify them.
> 
> ( to do THAT level of rewrite requires a deep understanding of the interpreter
>   ( to achieve THAT level of understanding we need to deeply document how
>     the interpreter works.
>      ( to do THAT level of documentation we need to review each function
>        in detail.
>        ( to do THAT detailed level of documentation is my current goal.
>          as a side-effect of achieving that level of understanding the
>          code will be in lisp (which is a language i deeply understand.)))))

I am not convinced that deep understanding of Lisp is sufficient or necessary,
much like a deep understanding of assembler languange alone is neither
sufficient nor necessary to debug Lisp code. At each level of translation, the
logic and design of the program need to be understood and well documented to
assist debugging. What is necessary and sufficient for working in Axiom,
especially with the compiler, is a deep understanding of the algebra Spad code
and the mathematics behind it. A better error message response on the algebra
Spad level rather than the Lisp or Boot level, together with a compatible
interpreter/compiler environment will go a long way to help the writing of
robust algebra code -- the goal of Axiom. It will remove at least 50% of the
frustrations in developing algebra code and correspondingly doubles the
retention rate of new users (empirical projections, of course).

> read the book when it comes out.
> the story turns out ok in the end.
> BAD Tim

Can we change that to GOOD TIM?

William