[Gnu-arch-users] Re: [OT] Architectural renovation [was: funding free software R&D]

[Tom Lord]

    > From: "Stephen J. Turnbull" <address@hidden>

    > The problem for investment is cash flow (which will fix itself,
    > [...]) and poor allocation decisions by management (which is
    > what we should discuss how to fix).

Ok.


    > If you want to talk about how to employ more programmers, that's fine,
    > but I thought we were talking about restructuring the way the industry
    > works.  

Ok, let's not get caught up on the employment question.   I'm not
talking about a private "workfare" program here.  However, let's
not forget:

   +----------------------------------------------------------------+
   |       The Software Architecture Principles for Business        |
   |                                                                |
   | The software architecture of a large computing system          |
   | determines:                                                    |
   |                                                                |
   | 1. The engineering roles (the set of jobs) needed to           |
   |    implement, maintain, and extend the system.                 |
   |    This is similar to how the design of an automobile          |
   |    factory determines the jobs of line workers.                |
   |                                                                |
   | 2. The degrees of freedom concerning how the system can be     |
   |    extended and customized.   This is similar to how the       |
   |    design of an automobile factory determines how much         |
   |    variation there can be in next year's car models, and       |
   |    the years' after that.                                      |
   |                                                                |
   | 3. Important constraints on business models concerning         |
   |    the system.   This is a direct consequence of (1) and       |
   |    (2).  It is similar to how the auto factory is a large      |
   |    constraint on the prices and varieties of next year's       |
   |    models, on what parts suppliers can offer, and on how       |
   |    fuel stations and repair services can support next          |
   |    year's models.                                              |
   |                                                                |
   +----------------------------------------------------------------+

The architecture of software systems is very much about who can
participate, in what ways, and what they can (practically) do.   Take
out those considerations, and you're left with empty and purely
theoretical computer science (if you even want to dignify it by
calling it so).

The architecture of software systems is also, therefore, about what
prospects the industry has to grow.  It's about how facile the
"owners" of a given architecture are in competition with their
industry peers.   It's about the economic "ecosystem" that grows up
around a given line of products.

To talk about how to restructure investment _without_ talking about
jobs is naive.

Surely "The Principles" have analogs in every industry.  A corporation
buys stuff, transforms it, and sells it -- labor being one of the
largest and most important things they buy and transform.  What's
specific to our industry is the realization that the software
architecture of our systems is what shapes how labor may be deployed.


    > What needs to be done is to take those CTOs and MIS VPs and show them
    > there's a better architecture.  [....]

    >     Tom> Same old, same old.  Some software architectures give hackers
    >     Tom> more inspiration to participate and generate value than
    >     Tom> others.

    > Tell me more.  I don't see it.  XEmacs _today_ has the "Emacs
    > architecture," supports internal graphics, embedded toolkit widgets,
    > sound, and we're working on smell :), supports DSOs, supports drug and
    > drip.  It provides a reasonably robust and simple to maintain package
    > manager.  We move another submodule out of C and into Lisp every few
    > weeks.  The architecture is there.  It needs some of the burls hacked
    > off and sanded down (not to mention polished up).

The Emacsen are interesting if we look at history rather than their
current state.  A lot of good (for the time) stuff did get written for
them and the developer "ecology" that produced that stuff is
interesting.  But in the here and now:

It's functionally anemic, by many critical metrics.

1) Emacs lisp is slow by design.   Dynamic binding combined with
   buffer-local variables is the largest culprit.

2) Emacs lisp is slower than it needs to be because of how it was
   first implemented.   This is hard to change without doing as much
   work as a fresh implementation would take.

3) Emacs is too text-centric.  The "Emacs architecture" needs
   thoughtful extension for other media types.  Background rasters,
   embedded images, icons, buttons, toolbars, and so forth are very
   cool hacks -- but where are apps like:

         http://freshmeat.net/screenshots/30893/
         http://freshmeat.net/screenshots/2969/
         http://freshmeat.net/screenshots/64/
         http://freshmeat.net/screenshots/2203/

   and, _more_importantly_, how do the necessary capabilities for
   such apps fit coherently into the "Emacs architecture" so that
   such applications can be written, extended, and documented with
   an ease comperable to Gnus or c-mode?   (E.g., how does the interact
   loop generalize?   Does `(point)' generalize?  Does
   `(current-buffer)' generalize?  Overall, does the "program and user
   have isomorphic dynamic state" principle generalize?)

4) Emacs' text handling is too slow.   Regexps are one large culprit;
   markers another; properties another.

5) The `(interactive ...)' mechanism is too tightly linked to its
   "mode line on a tty" origins.   How does it generalize for more
   GUI-ish interaction, including support for multiple media types?

6) Flow of control is poorly managed.  Busy programs freeze
   interaction.  Worse, the critical "program and user have isomorphic
   dynamic state" model is _not_ carried through in the area of
   recursive interact loops (where concurrent loops are far more
   desirable).

7) Window and frame configuration support is poor -- compare it with 
   the "layout widgets" of competing systems.

8) The lisp library is a crufty old mess.

9) Subprocess support and systems programming support generally is
   frustratingly minimalist -- it's incomplete.   Where's SCSH?
   Can I set-up a complex pipeline without relying on the shell?
   Do I have a variety of efficient options for I/O processing both
   with files and with subprocesses?

Etc.

Two of those points, Emacs lisp speed (by design) and flow of control
issues, make me think that it the current implementation can not be
rescued without, in essence, rewriting it.

Other points, Emacs lisp speed (by implementation), text-centric, text
speed, `interactive', and the lisp library, reinforce my belief that 
rewriting it is the easier path.

The remaining points, window and frame configs, cruftyness in ./lisp, 
subprocess and systems programming support, are areas where I think
the current Emacsen could certainly be improved -- but personally I
don't feel very motivated to work on it given the other problems:  the
payoff wouldn't be large enough.



    > Where are the hackers?  More at GNU Emacs (which of the above has only
    > the internal graphics support, and a dedicated GNU-camp third-party
    > maintainer tells me it still sucks) than at XEmacs, and GNU Emacs is
    > way understaffed for its role IMO.  Evolution probably has as many as
    > the Emacs projects combined, at least guesstimating from list traffic.

I think that, for the most part, Emacs hackers are there to support
people who use the current Emacs to do what the current Emacs does
best -- not so much to take it in radical new directions.

Regarding Evolution's number of hackers: when the "kids these days"
walked into the room, what did they see?  They saw a big, flashing
videotronic display over the Gnome booth saying "Welcome to The
Future" and on the desks at the Emacs booths they saw some xeroxed
handouts detailing the cat-fight between RMS and JWZ, a demo of Emacs
running in an xterm and in an X window (looking about the same), and a
small contingent of grey-haired protestors wearing coffee-stained
"VI-FOREVER" t-shirts.

Yes, I suppose you're right (elsewhere) that when Lucid
fall-down-go-boom, especially in combination with the "Why it is
impossible to work with RMS" flames, shallow-thinking execs had seen
all they cared to about the difference between these architectures.
But if you read your RPG, it's seems pretty clear that Emacs had
little to nothing to do with Lucid's fate, other than to _extend_ it's
life by a few years.


    > I'm not discounting your _intuition_; I respect that highly.  But the
    > _words you say_ make me think "XEmacs" (ok, ok, to a guy with a hammer
    > everything looks like a thumb, or whatever), and I just don't see the
    > interest.  So something else is needed.

Quality and the overcoming of executive superstition.  There's an
investment hump there for the emacs architecture.   I'm sure that much
more has been spent pushing the tree-of-widgets architectures up their
particular oft-climbed hill.   I'm sure that at least one quite
promsing "New Emacs" project was shat upon and shut down a decade ago
for No Good Reason Whatsoever.   Overcoming executive superstition and
then paying to produce a quality foundation.  That's what's needed.


    > Even once we've got that "something else", there's another hurdle.
    > That is the belief that integrated architectures enhance market power
    > (Microsoft taught us that).  So we _also_ need a business model that
    > turns that on its head, somehow.

This is too long already.  Let's continue on that topic if you get
this far and don't sidetrack on minor irrelevancies too much :-)

Very briefly, the emacs (programming environment) architecture has
incredible expressive power and, importantly here, makes distributed,
decoupled development of applications for niche situations dirt cheap
and (in the areas well supported by the basis set)
_produces_better_results_.   So, we can talk about how to deploy labor
to effectively flood-fill niches as fast as they arise, with both very
fine and very gross granularity.


    >     Tom> Yes, it will take a lot of work to make a new one with enough
    >     Tom> performance and eye-candy, and graphics features to have
    >     Tom> impact

    > Starting from today's XEmacs, three man-years.  

This is another line we could get into in detail.    I'd rather do it
in a room than on a list and with some money on the table, though.

I'm really skeptical that starting with the XEmacs source base is the
best course -- but I fully acknowledge that it's a decision to make
rather than a no-brainer.

    > The dedicated hackers are not personally interested in going in that
    > direction.  They already have their hacking platform.  Casual hackers
    > would rather work in C# (or SWIG private libraries and call that an
    > API) than harangue us to make existing features more accessible.  

Cause that's what the sexy rich kids do.

    > Python, Perl, Ruby are feasible, even superior, alternatives for
    > non-text-intensive applications.  I've never much liked Perl or Ruby,
    > so I can't give examples for them.  

It'd be a mistake to reduce this to a language war.   We would need to 
start by talking about other aspects of the "Emacs architecture" first.
Overcoming superstition.


    > And although it has nothing to do with calibration of secondary
    > gender-specific characteristics, even you turned to C to reimplement
    > Arch.  Why not Python, or XEmacs Lisp (yes, we would very likely
    > swallow a Lisp binding that linked to libarch as an option)?  Or Guile
    > or systas scheme or scsh or clisp or Perl or Ruby?  Something deep is
    > going on here.

These are interesting questions but I think that language wars are OOT
in this thread.

-t