RE: [Axiom-developer] Website down?

[Page, Bill]

On Monday, November 28, 2005 1:22 PM I wrote:
> 
> On November 28, 2005 12:05 PM C Y asked:
> > 
> > Anybody else getting a HTTP 500 - Internal server error?
> > 
> 
> It's working for me right now. I have not re-set and or changed
> anything since last night (about 10:00 pm Eastern Time. I have
> noticed that occasionally I get very poor response times (20 or
> 30 seconds per page). I am beginning to think that there might be
> intermittent problems at the ISP where we run axiom-developer.org.

Looking at the process statistics on the server on Monday and
Tuesday, is seemed that we were getting only about 10% to 15% of
the total CPU capacity of this dual-processor shared virtual
server. The system was spending a lot of time waiting for disk
access and also "idle" but I am not sure exactly how to interpret
these statistics in the case of a shared server - the total
utilization is 100% for each CPU including the idle time. But
perhaps "idle" means that it is serving some other virtual server
on the same machine? In any case performance was definitely
much less that we have been used to in the past.

> 
> The other possibility is that the once in a while the access
> rates to the server exceeds it's capacity to respond in time.
> This could especially be the case for Zope application server
> which is known to require significant system recourses compared
> to more conventional web server software. There are some things
> that can sometimes be done to "tune" Zope performance and I will
> look into that some more later today.
> 

Since this took some significant effort to do, I thought I would
take the time to explain a little even though this is rather off-
topic for the axiom-developer email list... :) None of what follows
is about Axiom or mathematics, if you are not interested in web
site performance, you can stop reading now.

--------

"Out-of-the-box" Zope runs as a combined database and web server.
The configuration on axiom-developer.org looks something like
this:

                      You
                      /|\
                       |
                    Internet
                      /|\
                       |
               Web cache server (squid?) <----> memory
                      /|\
                       |
               Front-end server (apache) <----- files
                      /|\                  (3)
                       |    (1)
axiom <--->  Application/db server (zope)  <----> memory
                      /|\                   (2)             
                       |
                 Zope File System

(1) My first step in optimization was to ensure that all static
web pages and parts of web pages (e.g. style files) that did not
change are marked as cacheable by the zope. If they are properly
marked then the Web cache server (which is actually part of the
ISP's local network and not on the axiom-developer server itself)
will keep a copy of these in memory for a fixed period of time
(usual 1 hour) and send them directly to the requester without
bothering our server. This is new.

(2) It is also possible to specify that certain frequently used
page templates and scripts are kept in memory for fast access.
In fact it only really affects one file but it is an important
one since it's contents occurs on every web page generated by
this system. This is also new.

(3) Some files such as the image.png files that represent LaTeX
equations in the browser can be accessed directly by the apache
web server. This is controlled by url re-write rules in the
apache configuration file. This is old. No changes here, we
have been doing this since the site was first made public over
a year ago. I just mention it for completeness.

Although these changes improved things some (maybe about a 5%
increase in CPU utilization), on Tuesday the server seemed
overloaded several times during the day. So I decided to make
one final big change to the configuration:

                    ...
                    /|\
                     |   (1)
axiom <--->  Application server (zope)  <----> memory
                    /|\                  (2)             
                     |
           (4) Database server (zeo)
                    /|\
                     |
               Zope File System

(4) The zeo server is an optional component of zope that is
usually only used in higher-performance cluster configurations
that involve more that one physical zope server. Zeo runs as
a separate process and in most cases on a separate physical
machine. It takes a bit of planning to set this up properly
and the documentation is a bit uneven in quality. Some notes
claim that this configuration is in fact more robust and easier
to administer than the simpler setup.

This configuration change is relevant for us even though we
have only one server because the axiom-developer.org server is
actually a dual symmetric multi-processor machine. This means
it can run two processes completely in parallel. By splitting
Zope into two processes: one that does the web page formatting
(application) and one that accesses the underlying database,
these two critical processes can run in parallel and as a
result make better use of the server resources.

The end result was about another 10% increase in CPU utilization.
It's now running between 30% to 35%. And so far this seems good
enough to keep ahead of the aggressive search engines that are
always scanning us plus those people who actually are trying to
run Axiom etc. But I still get the impression that we are running
near the limit of the capacity of this machine.

If anyone has any questions or suggestions I would be happy to
receive them. And of course if you continue to get server error
messages such the one above, please let me know.

Regards,
Bill Page.