Re: gcl, shared libraries, was [Axiom-developer] Lisp portability

[Waldek Hebisch]

Camm Maguire wrote:
> Greetings, and thanks for including me in this thread.  Unfortunately,
> axiom-developer mail is still not getting through my ISP for some
> reason. 
> 
> 
> > In my private tree Axiom is nicely working on top of sbcl and clisp.
> > The remaining known problems are sockets and writeablep function.
> 
> Great!  It is always nice to have more options.  But I do have a
> general question here.  Work in this direction obviously makes sense
> if 1) GCL is currently deficient for axiom use in some regard and 2)
> we have difficulties addressing said deficiencies via changes to GCL.
> If both of these are true, I would greatly appreciate you and/or
> others letting me know the details, as it is and has been a goal of
> mine to ensure that gcl remains optimal for axiom use.  As in the
> past, I pledge that if axiom needs anything from GCL which is
> currently unavailable, I will make sure said functionality works in
> GCL.  What I would like to avoid is chasing every random package which
> no one has any serious intention of deploying in a real shipped
> program. 
> 
> If either 1) or 2) above are not true, may I suggest that significant
> work in this regard is a distraction from the real goals of the axiom
> project -- to deliver the best symbolic algebra package in the open
> source world.  In our company, we manage approx. $70 billion dollars
> using open source software and code we've written on top of it.  We
> would get nowhere if we spent any time at all making sure our code
> works both on gcc and icc, etc.  Rather we pick a platform that
> appears flexible, stick to it, and get the results we want out of it
> quickly.  We do almost everything using gcc, gdb and emacs, not
> because they are the slickest or hottest or even fastest, but because
> they have good portable performance, they can be changed at the source
> level if necessary, and they will be available forever, making our
> payoff period for the learning curve essentially last a whole
> lifetime.
> 
> I'd really love to help the axiom project by offering a similar setup
> for use by the developers.  Even one better -- axiom has its own
> personal compiler developer to tailor the compiler to its needs if
> necessary.  This only pays off if my work on GCL frees axiom developer
> hours for work on the core system.
> 

That is very generous offer and I appreciate it.  But I must disagree
with some of your assuptions.  Namely, sticking to a single
platform/toolchain is perfectly good business decision.  But,
especially for open source project, there are good reasons to
to seek portablity to other Lisps.  Let me mention few reasons:

1)  Code readability: for Ansi constructs it is reasonable to say
    that behaviour is described by HyperSpec.  But if we use non
    Ansi things or depend on GCL specific aspects then this is
    no longer the case.  So it is good to separate Ansi part from
    GCL specific ones.

    Some people belive that such separation should be done via
    extensive audit/review.  I take pragmatic position here:
    compiling with other Lisp implementations identifies 
    various dubious constructs much more cheaply than audit.

2)  Sbcl port helped to make Axiom compatible with Ansi GCL.  Together
    with clisp port it significantly reduced chance for future
    Ansi changes in GCL to break Axiom.  

3)  I have some personal intrest in other Lisp implementations.  

4)  Lisp developers seem to be attached to the implementation they
    use.  There is good chance that little effort spent on keeping
    Axiom portable will be offset by extra contibutions possible
    thanks to portability.

5)  Ability to use already installed Lisp for Axiom development/
    testing.

There are also some problems that I had with GCL based Axiom:

- Axiom is slow.  To identify reasons I need a profiler.  But I was
  unable to use GCL profiler.  I tried to follow sparse instruction
  given in doc/profile, but I got no output.

- From time to time Axiom signals more or less random errors or
  crashes.  The exact reasons are hard to determine because 
  by default Axiom is build with safety 0 and error messages
  are rather uninformative.
  I have (few times) build Axiom using safety 3, but the build than
  took more than 16 hours.  Also, in one case error was reproducible
  in safety 0 build, but went away with safety 1 (or safety 3)
  
  I would appreciate some hint how to debug safety 0 programs and
  how to find a good compromise between debug abilities and build
  time.  Let me remark that at safety 3 sbcl build took me about
  3.5 hours compared to 2.5 hours building with gcl at safety 0
  (or more than 16 building with gcl at safety 3).

Let me also mention that Axiom currently have only old Windows binaries
and no Mac binaries (we have reports of successfull builds, but the
resulting binaries did not work on other machines).  For Windows I
seriously consider cross-compilation (creating binaries on Linux).
Another possibility is to build Axiom using Wine -- I do not know
if gcl works in Wine.  Yet another posibility is dosemu, in the
past dosemu run djgpp programs very well, but it seems that now
djgpp is deteriorating.

> > Also, currently for clisp I need cl-fad library, which I consider
> > problematic.  Namely, cl-fad does not work with gcl, so for gcl
> > we need separate code.  We need only few functions from cl-fad,
> > to work around clisp weirdness (clisp makes strong distinction
> > between paths to files and paths to directories and refuses
> > to perform file operations on directories).  So my current plan
> > is to eliminate use of cl-fad and provide the needed functions
> > directly.  Related problem is performing operations on
> 
> I think this is a wise choice, but needless to say, if you need
> cl-fad, I'll make sure it works.
> 

Currently I eliminated use of cl-fad.

> I love lisp, obviously, and primarily for the magnitude and quality of
> AI-type code which has been developed and has recently entered into
> the open source world.  But lisp is not going to compete with perl,
> python, java, or C, IMHO.  The primary goal should be in fulfilling

I belive that currently choosing language/implementation one is faced
with choice between speed, safety and convenience.  Native perl
(and probably also python, ruby, tcl etc.) is convenient but slow
and in many cases unsafe (due to reliance on parsing and various
"magic" convertions).  I belive that it is possible to create
a language that makes better balace.  In fact, I belive that
already just as a language Lisp is better (but current implementations
lack many real life qualities).  OTOH I am not sure if Lisp is
the best option.  And of course real life survival is much
more complex than "language quality".


> Thankfully, the ansi-standard is written down, and with all its warts,
> will last > 30 years, IMHO.  If we need items outside the standard, we
> should rely on implementations in C shared libraries, as these will
> have a longer lifetime and in general will be technically superior.
> We should then pick one again non-standard glue between lisp and C,
> and stick to it, and call all the shared library routines from within
> lisp.  These of course are just my opinions.
> 
> It is good you brought this up, as gclcvs has followed clisp-style
> directory usage, at least at present.  We can undo this if required.
> Pathnames with spaces are also fully supported in gclcvs.  These can
> be backported to 2.6.8pre if necessary.
>

I personaly think that clisp choices are misguided.  But for Axiom
I have a workaround so this is no longer a problem.
 
> 
> > Concerning sockets, we need Unix domain sockets and select.  It
> > seems that clisp provide both, but to get Unix domain sockets
> > one needs version including rawsock module, which is not included
> > in default clisp configuration.  
> 
> > sbcl offers sb-bsd-sockets which seem to have basic functions,
> > but I do not see select.
> 
> > gcl documentation suggest that Unix domain sockets are unsupported.
> > Also, I see no traces of select.
> 
> gclcvs has a select/fork based parallel processing system:
> 
> SYSTEM>(walker::macroexpand-all '(p-let ((x (foo)) (y (bar))) (+ x y)))
> 

I do not understand details of macroexpansion, but I belive that it
is doing something different.

> This can be adapted to whatever other interface one may desire.
> 
> Do you mean dgram (UDP) sockets?  If you specify the interface, we'll
> put them in.
> 

Unix domain sockets have interface very similar to network sockets,
but use filenames as address.  In effect Unix domain sockets are
very similar to named pipes, but have some extra capabilities

Concerning select:  conceptually you have a list of possible events
and want to check if some of them happened.  Select allows blocking
wait on multiple events.  It is important that select is _not_
limited to some "structured" constructs, users of select should be
able to specify exactly the set of "interesting" events.  On
Unix it is currently only possible to associate events with file
descriptors, Windows has more general WaitForMultipleObject system
call.

An interesting variation of select system call is given by Linux epoll
system call: you register a set of descriptors with kernel and than
you ask if something changed.  One advantage of epoll interface is
that it avoid repeatedly copying and scanning potentially long
list.  epoll is important for high performance web servers.

For Axiom use any non-crippled version of select will do.  Currently
Axiom selects between two Unix domain sockets and terminal, so
select must allow mixing sockets with other file descriptors,
but performance requirements are quite modest.

> 
> > There is "portable" cl-sockets library but the manual says it supports
> > Allegro CL, sbcl and cmucl.  The manual does not say anything about
> > Unix domain sockets or select.  The manual says that cl-sockets requires
> > UFFI, so presumably cl-sockets works on top of "portable" C library.
> 
> > In short my finding is that portable Lisp sockets are a myth: all
> > implementations provide different interface and frequently miss
> > some essential services.  People who want portablity between Lisp
> > implementations interface to C. 
> 
> Agreed!
> 
> I've been working on a better interface to external shared C
> libraries for GCL.  The issue, as you may recall, is that GCL-compiled
> code referring to external shared function addresses are statically
> relocated to the value of said address at the time the .o file is
> loaded. If one then saves the image and re-executes, the function is
> likely in a different place.  
> 
> I've figured out a persistent solution using dlopen.  Still working on
> the precise interface, but you can see the idea here.  Comments are
> marked with ***:
> 

That looks very interesting.  I notice that you automatically use
C library names and C functions names.  I hope that this is only
a default, because otherwise it would be very hard to resolve name
clashes.



> > Given that it seems that most reasonable way for Axiom is to use
> > existiong C code.  There is are drawbacks: we need to interface to
> > C and typical Lisp implementation can only interface to shared
> > libraries via dlopen.  So we need to handle issues related to making
> > shared library.
> 
> Hopefully, the above will be useful.  It appears that libdl is quite
> portable, but may require cygwin.
>

I do not know about libdl, but native Windows interface is reasonably
simple.  I think that main complication on Windows is supporting
multiple calling conventions.
 

-- 
                              Waldek Hebisch
address@hidden