Re: [Axiom-developer] Literate Programming example

[daly]

On Sat, 2011-11-19 at 14:35 +0100, Laurent PETIT wrote:
> Hello,
> 
> 2011/11/19 TimDaly <address@hidden>
>         On Fri, 2011-11-18 at 20:02 -0800, Daniel Jomphe wrote:
>         > On Friday, November 18, 2011 7:17:08 AM UTC-5, TimDaly
>         wrote:
>         >         Many of you asked me to show an example of a
>         literate
>         >         program and demonstrate the use of the tangle
>         function.
>         >
>         >         I usually use Latex for literate work but I've done
>         >         this example using HTML and <pre id="foo"> tags.
>         >
>         >         I've written a self-referential literate program
>         that
>         >         explains the details of the tangle function in
>         literate
>         >         form. You can find the web page at
>         >
>         >
>         http://daly.literatesoftware.com/lithtml/litprog.html
>         >
>         > I have read your literate program, and must recognize that I
>         know how
>         > tangle works even though I didn't want to really read the
>         source code.
>         > I read your prose all the way through. I still haven't read
>         the source
>         > code;
>         
>         
>         In fact, that's the whole point. You don't read the equations
>         in a
>         calculus textbook either. You read the words. The equations
>         are icons.
>         If you understood the text and "spoke" mathematics you could
>         probably
>         write the equations.
>         
>         In programming we can reach the same level of literacy.
>         Reading just
>         the words in the literate version it should be possible to
>         recreate
>         the program in your favorite language. Note that you would be
>         creating
>         a different program with different design decisions but the
>         same
>         functionality.
>         
>         > I didn't feel the need to read it. Were I to maintain your
>         program,
>         > I'd have more than enough confidence to start hacking the
>         code right
>         > now.
>         
>         
>         One thing worth trying would be to code the same program in
>         Clojure.
>         
>         The tangle program is conceptually very simple but there are a
>         lot
>         of low level design decisions that I would make differently.
>         For
>         example, there are loops in the C program which would go away.
>         
>         Would you map read or would you slurp? Mapping a read function
>         allows
>         transforming "& lt;" to < at read time. This does not matter
>         in the C
>         program because the buffer is mutable but it would matter in
>         Clojure.
>         
>         Would you use the Clojure pattern language to find the <pre>
>         tags?
>         Would you be able to parse out the string from the id? C
>         encourages
>         character-level hacking but Clojure would be much more
>         powerful.
>         
>         >
>         >
>         > I think this speaks very positively about literate
>         programming. What
>         > remains to be seen is how much (or not) I'm going to
>         practice it in
>         > the future.
>         
>         
>         If you do try to rewrite it in Clojure please post the
>         program. I
>         would be very interested to see how Clojure's concise syntax
>         and
>         semantics get reflected in your design decisions.
>         
>         The tangle program in Clojure might turn out to be a single
>         s-expression of only a few lines. The code density would be a
>         huge win but a literate version would still have to have the
>         vitals of the story. Remember that the key test for a literate
>         program is the "independence test". Someone can read it
>         without
>         talking to you, understand how it works, and be able to change
>         it.
>         
>         >
>         >
>         > What do you think of marginalia? It's a bit the reverse of
>         tangle; it
>         > assembles all those 70's files together into this book you
>         might want
>         > to read. Is it sound or not? Have your thoughts changed from
>         what you
>         > wrote in [1]?
>         >
>         >
>         > [1] http://goo.gl/cXWzF
>         
>         
>         Literate programming is a mindset, not a tool. You can write a
>         literate program in anything, including marginalia.
>         
>         That said, I have yet to see a Clojure program that lays out a
>         story
>         so I can sit and read it. For a real challenge, if you try to
>         write
>         tangle in Clojure, try writing the story in marginalia. I'm
>         sure
>         Fogus would welcome the feedback.
>         
>         
>         The readability aspect is a real feature. Heck, you could even
>         give
>         your programs to a company so they could read them BEFORE the
>         job
>         interview. I would strongly favor hiring someone who could
>         communicate, who cared about code quality, and who could
>         improve
>         the company's maintenance headache in the long term. A
>         Literate
>         Clojure programmer would be a real gotta-hire person.
>         Companies
>         use many programming languages but all programmers really do
>         need
>         good communication skills.
>         
>         
>         In the long view, it would be sweet if the Clojure reader knew
>         how
>         to read a literate program. Just call "(literate-load file
>         chunk)"
>         and you get the same effect as if you had tangled the program
>         to
>         a file.
>         
>         With literate-load available you would be able to write all of
>         your Clojure code in a literate style, making Clojure much
>         easier to understand, maintain, and modify.
> 
> 
> I'd be interested in studies proving that programs written in a
> litterate style are easier to maintain.
> 
> 
> So far, there has been this metaphore between a "story" and a
> "program". How far can this metaphor be pushed ?
> 
> 
> Is it easy to write a book with more than a couple people ? To
> maintain it ? To modify it ?
> 
> 
> I can easily understand the appeal: if I were to discover a program
> for the first time, I would certainly be interested in discovering it
> via a literate style, because I would have some guarantee that some
> effort has been pushed into providing for the mainteners: contexts,
> argumentations, debates concerning design choices, etc.
> 
> 
> But past this first "introduction" to the program, I don't know what
> it would cost to maintain it without adding to the entropy of its
> literate part.
> 
> 
> I mean: to be honest, it's not as if, already, all programs written
> were correctly documented and we were facing a wall: "we can not
> improve our documentation further, we need to find another way to
> document our programs" ;-).
> 
> 
> Sorry to have to play the role of the devil's advocate, but I think
> there are currently holes in the argumentation in favor of literate
> programming, at least wrt to the material I've been exposed to so
> far ...
> 

YOU signed a contract with the Devil too? Hmmm. I thought I had an
exclusive contract. Perhaps he lied to me. :-)

I have experience what most programmers never see. I got my own code
back after many years because I volunteered to open source the Axiom
code. I was one of the original authors. 

I write dirt simple lisp code. I know exactly what it does. However,
I don't know why I wrote it. I don't know how it fits into the whole.
I was clearly solving a problem but the problem definition is gone
even though the code is there. I know I need the code because the
system crashes if I delete it.

I'm left with the problem of reverse-engineering the code. All I
can do is invent probable reasons. Nobody wrote down the design
decisions. It is very difficult to maintain and modify 1 million
lines of lisp code.

If you want to see the difference a literate program would make I
suggest taking one of Rich's ideas, such as software transactional
memory, and try to explain the code that implements it. I have
tried that in my Clojure in Small Pieces book and it is a LOT
harder than it looks.

Literate programming is not for you. In fact, it is probably not
worthwhile for most programs. But if you're going to make a program
that "lives" beyond you then it really matters. You need to communicate
to people you will never talk to and who cannot ask you questions.
Some of the authors of Axiom are already dead. They don't seem to
be answering my emails.

As a lifelong lisper I've decided that Clojure is interesting to me.
I'm willing to devote a lot of time and attention to it. In particular,
I want it to outlive Rich Hickey's involvement in it. Rich will
eventually retire into management somewhere and the code will begin
to rot. I am hoping that a literate form of Clojure will make it
possible for new people to maintain, modify, and extend it.

To that end, I'd like a literate form of Clojure that passes the
"independence test". That is, I'd like a new person to be able to
take the literate Clojure Core off to an island for a couple weeks
of reading and come back as a well informed, productive developer
capable of contributing correctly without breaking existing ideas.

Thus, literate programming is an investment in the future.

However, as Knuth points out and as I've already experienced, writing
a program in literate form vastly reduces the errors. There are two
causes I can find. 

First, if I have to write an explanation then I have to justify my
poor code hacks. Sometimes I find that I rewrite the code because the
very act of explaining the bad code made me realize that the code is
bad. I find I have to talk about memory allocations so I catch leaks.
I have to talk about argument handling so I end up proving that the
arguments are correct, or I end up checking for possible failures.

Second, because other people can see the reasoning at the code review,
they can chastise me for failing to explain, or explaining things that
the code does not do, or just plain failing to implement a correct
solution. 

I have found that literate programming makes me a better programmer.

Convincing people to use literate programming is like convincing
them to use lisp. Before the AH-HA moment occurs they can't imagine
why anyone would code this way. After the AH-HA moment they can't
imagine why anyone would code any other way. I have clearly crossed
that line.

Tim Daly