[Axiom-developer] RE: address@hidden: [RHH #FKL-86268-984]: move Axiom to shared dedicated server]

[Bill Page]

Tim,

On December 1, 2005 3:17 AM you wrote:
> 
> Axiom-developer has been moved from a shared-virtual server 
> (basically vmware-ish) to a shared dedicated server. the
> shared dedicated server is a dual-processor hyper-threaded
> machine with only 4 customers so we basically get a full
> processor and a gig of memory to ourselves.
> 
> Hopefully this fixes the problem. Let me know.
> 

Thanks. So far this is great!

The machine is actual a quad-processor and even a single
threaded operation (using only one processor) seems to be at
least twice as fast on the old server.

With a quad processor, there is one more thing that I can do
to improve performance - it is possible to run more than one
zope instance, accessing the same zope database. This has some
advantages over just allowing zope to run more threads because
of a limitation in the way threads are handled in Python. It
might take me a day or so to setup an 2nd zope instance, so
I will wait on that for now and just see how it goes.

Some yesterday when I was having a lot of trouble with the
old server I changed the robots.txt file to block all search
engines and spiders.

User-agent: *
Disallow: /

These robots have become particularly aggressive lately -
you just have to look at the web statistics to see where a
lot of the cpu cycles where going.

http://page.axiom-developer.org/usage

Recently their access rate was frequently overrunning the
server's capacity. With the modified robots.txt that was
slowed down a lot (but not stopped since there are some
robots out there that do not play nicely).

This morning when I had a spare moment and after reading
your email about the new server, I temporarily opened up
access to the search robots again. I was surprised to see
them all suddenly "jump" onto the new server and it wasn't
long before they had overrun the zope process again! So even
with this machine is still have to be a little careful.
Adding another zope instance would help prevent this.

Today I did a couple of quick things that might have improved
the zope performance a little: 1) I did a re-pack of the zope
database. Normally this is done once a week on Monday morning
around 00:05 AM. 2) I re-built the catalogue indexes for both
the wiki and the portal. This second one might have been more
beneficial than I originally thought.

I have also cut-back the robots.txt to be more selective
about what we allow the search engines to do.

User-agent: MSNbot
Disallow: /

User-agent: *
Disallow: /SandBox
Disallow: /zope/mathaction/SandBox
Disallow: /bibliography
Disallow: /zope/Plone/bibliography
Disallow: /refs/axiombib
Disallow: /zope/Plone/refs/axiombib
Disallow: /refs/redbib
Disallow: /zope/Plone/refs/redbib

--------

Some explanations about the contents of robots.txt:

I did not see much point in allowing the Microsoft Network
MSNbot in (which is one of the robots that has been quite
aggressive on our site) since we are already be aggressively
scanned by Googlebot.

The /SandBox pages are supposed to be just drafts anyway
so I don't think we want this indexed externally in any case.

The /bibliography pages involve a lot of cpu processing
each time they are accessed and I have not found anyway to
make this more efficient yet, so for now I have blocked
robot access to them.

Right now this seems enough keep things in check but as time
goes on, I might have to add more items to the disallow
list. (It's a bit awkward that the robots.txt standard only
allows Disallow commands and not Allow commands ... )

If anyone has any suggestions about how to optimize the
robots.txt file, I would be very glad to receive them.

Finally, Tim. I am still concerned about the fact that you
(personally) seem to be bearing all the cost of running the
axiom-developer.org server. I expect that this new configuration
is going to cost significantly more since it is dedicated
server. Let's talk about what we can do to make this more
equitable.

Thanks again for setting this up.

Regards,
Bill Page.

> 
> ------- Start of forwarded message -------
> Date: Wed, 30 Nov 2005 23:26:03 -0600
> From: "RH Helpdesk" <address@hidden>
> Subject: [RHH #FKL-86268-984]: move Axiom to shared dedicated server
> Reply-To: "RH Helpdesk" <address@hidden>
> 
> 
> This has been completed.
> 
> Admin
> RoseHosting.com
> 
> 
> On Wed, 30 Nov 2005 18:01:11 -0600, address@hidden wrote:
> > IP: 72.23.23.52
> > 
> > Please move the axiom-developer.org
> > domain (209.135.140.38) from a shared
> > virtual server to a shared dedicated 
> > server and send me mail when the
> > process completes.
> > 
> > Tim Daly
> > Axiom Lead Developer
> 
> - ------- End of forwarded message -------
> 
>