Re: mod_lisp for guile

[Alan Grover]

Neil Jerram wrote:
(much elision)
> It seems to me, though, that there is nothing especially lispy about
> the module (mod_lisp) which runs inside Apache.  Presumably the Apache
> side of the protocol could be implemented in any language, or could be
> provided "native" by Apache itself?

It's in C. Nothing lispy here. It could be native, but, since it is not
essential to an HTTP server, it is left as a module.

> Is the form of the protocol itself lispy?  (I.e. parenthesised and
> readable using read.)

The protocol is not lispy. It sends each header as:
        header-name <new-line>
        header-value <new-line>
with an "end" as the last header-value.

Several "daemon-side" implementations have been written in other
languages. I think it is only called mod_lisp because it was originally
written for use with Common Lisp.

> How would using threads reduce resource usage?  Otherwise this all
> makes sense.

A thread doesn't require an entire process (like a fork does), in
particular, memory is shared. I think you could limit a thread's
resource requirements to just an execution "thread" (PC, registers,
stack), otherwise sharing everything within the one process. Being
light-weight/cheap is one of the major motivations for threads.

Though I read something recently that claimed some thread
(mis-)implementations are more costly than a fork, and often more costly
than you'd hope. Apparently, the Linux 2.6 thread implementation is in
user-space, which means it isn't very thrifty (and causes some other
interesting issues).

> So you mean that if someone has already written an HTTP request
> handler procedure for use with www-guile, it ought to be possible to
> reuse exactly that same procedure with your mod-lisp also?  Sounds
> good to me.

Also, I don't want to rewrite all the logic from that module (e.g.
query-parsing, etc. etc.).

My goal is to allow no change between CGI and mod_lisp (for certain www
library modules). Perl's CGI module transparently works under CGI or
mod_perl (an embedded Perl interpreter in the Apache process).


>>The interface in the first release is a function that loops, calling
>>your function for each request. So, it kind of takes over. It occurred
>>to me that there might be more polite ways of working. At the very
>>least, there should be mechanisms that impose the least policy.
> 
> 
> Yes, indeed.  This problem isn't specific to web requests, of course.
> It applies to pretty much any socket-driven or GUI (event-driven)
> library.  (For example, the (ossau gds-client) module in my
> guile-debugging package.)
> 
> It would be nice to have a SRFI which standardised the way of
> specifying handler procedures for such events, and of determining when
> each such event source had work to process, along with standard
> procedures for doing blocking and non-blocking selects on all the
> events, and so on.

I look forward to your work on the issue. :) I kind of like the
lazy-list solution, but I've been infected by a pure-and-lazy
programming language (Clean). I think some people call them generators.

I'm liking the lazy-list idea more and more. It seems to work well with
the tcp-socket listener pattern.

The old way:
        ; off in somebodies library module
        ; It "takes over"
        (define (daemon-loop handler)
        set-up listener
        Repeat forever: 
                block on accept
                spawn thread with accepted socket and handler
        )

        ; In your code
        (define (my-handler ...)
                ; called in a new-thread
                do-something)
        ; we will now loop forever on accept/spawn
        (daemon-loop my-handler)

With a lazy-list, it would be something like
        ; In your code
        (define mod_lisp-request
                (make-lazy-mod-lisp with-appropriate-tcp-code))
        ; Whatever "daemon-loop" you want:
        (for-each
                spawn-my-thread
                mod_lisp-request)

>>So, assuming I provided lazy-mod-lisp:
>>      (define lazy-results (lazy-mod-lisp socket-port))
>>
>>      (pair? lazy-results) ; -> #t. maybe not, I'd have to check
>>      
>>       ; actually does the work to figure out first mod-lisp request. could
>>block!
>>      (define first (car lazy-results)) ; might need "force"
> 
> Yes, and blocking is the problem.  The application usually needs to
> know when it can call this without blocking.

An interesting problem given the apparent PThreads implementations (any
blocking-call blocks all threads). In my prototype code, I "soft" block
(just block the one thread), using 'select.

In this case, it is probably OK (and expected) that the 'car would
"soft" block (until the next request comes in), since this is the
daemon-loop. You could push this loop into its own thread, and leave
your "main" thread unblocked.

Under PThreads, I am lead to believe that each "read", if it blocks,
blocks the entire process (all threads). But, under reasonable
circumstances, the blocking should be very short. However, if the Apache
side of mod_lisp is slow in writing, it would cause problems.

> I see.  So combining my thoughts and yours, I think this would have
> three levels to it.
> 
> - Generic API for specifying sources of input and how to handle them.

Well, I'm only addressing mod_lisp at the moment. Though I intend to let
the user (programmer) supply the socket-listener (or whatever they want
that results in a read/write port that appears to talk mod_lisp).

I'd be happy to work with you to extract the (to be written) lazy-list
stuff for general use.

You might want to look at Perl's Net modules. They are layered something
like you suggest: a generic TCP layer, a generic header-value/body
protocol, specific protocols like HTTP or SMTP.

> - The particular case of this API for an HTTP request handler.
> 
> - Your ideas for presenting the HTTP content in various useful ways.

I have plans for several more layers for web-applications. E.g. a
web-application skeleton (get request, decode request, change state,
decide on html, construct html), and layers of policy to fill it with
(e.g. a particular html-template system).

Though I've heard about continuation-passing web-frameworks, I'm not
currently working on them. Also, since I use Guile, there is the problem
that you can't use a continuation in a different thread.

> I hope
> it's of some interest though.

I have been interested. And, I thank you for taking the time.

-- Alan Grover