[Sks-devel] web timeouts even with a proxy

[Jonathon Weiss]

Pgp.mit.edu has been running with Apache as a proxy for port 11371 for a
long time.  Despite that I've seen a number of cases where sksd sits for
a minute and then reports a timeout with the web client (presumably the
Apache proxy) that it was trying to talk to.  Usually when it does this
once, it does it many times.  Restarting sksd will generally get it out
of this broken state.  I was wondering if anyone else had seen this or
had theories about what was causing it?  I have my own theory (though
I'm not sure that it's correct) that I'll detail below, but I'd be
interested in whether there are any other ideas out there.



My theory: I think this is a cycle.  I don't know what starts it, but
once it gets started, it needs a long period of quiescence to clear on
its own.  Once we have an initial failure (and one-minute timeout)
someone in the queue will have hit "reload" or otherwise dropped their
initial connection.  I surmise that at this point Apache forgets about
the request, but that the proxied connection is still in sksd's queue.
It then tries to read the query from that connection, sits there for a
minute and times out, at which point more people will have hit "reload".
This could also be exacerbated by Apache's two-minute timeout on proxied
connections, since after you get a couple of failures chained together,
that could cause things to go downhill quickly.  I don't have enough
knowledge of Apache's behavior to be sure that it would do the things
I've described above, but it certainly seems plausible.  There are also
a couple of details about my theory that I'm not 100% happy with, which
is part of why I'm asking for suggestions.  I'm currently testing my
theory by reducing the web-client timeout from 60 seconds to 10 (if
Apache can't get sksd the request or collect the answer from sksd in 10
seconds, I figure I have bigger problems), and preliminary indications
are good, but not perfect.

I should note that our Apache logs suggest we average around 450K
queries to sksd per week or one every one to two seconds, though traffic
is bursty.

        Jonathon

        Jonathon Weiss <address@hidden>
        MIT/IS&T/OIS  Server Operations