Re: [Sks-devel] 3 million keys & and community help requested

[Jeffrey Johnson]

On Nov 3, 2011, at 2:38 AM, David Benfell wrote:

> On Wed, 02 Nov 2011, Robert J. Hansen wrote:
> 
>> My position is 'no,' and for that reason I think asking for
>> justification for Berkeley DB is kind of pointless.  If there's nothing
>> obviously better, then why are we interested in his decisionmaking process?
> 
> Here I disagree. Berkeley DB does not seem to be as reliable as
> MySQL nor does it seem to have the quality of documentation
> available. When something goes wrong, unless you are a BDB geek, you
> will need to wipe the BDB and start from scratch.
> 

Reliable is *entirely* in the eye of the beholder.

The core problem is extrinsic (as in dependent on the amount
of data) interdependent tunables: yhje "default" values for
the no.of locks aren't anywhere near being useful for a store
of 3M keys.

The no. of locks needed is inverse to the pagesize because
Berkeley DB does per-page locking.

Then there's the PTree store which does a tricky recursion. If the
pagesize is too large, then the recursion self deadlocks. If the
pagesize is too small, then there is additional locking overhead.
To make matters worse: the recursion is deemed "CPU intensive"
and not worth tuning.

This all leads to a rather steep learning curve instantiating
an SKS keyserver.

The experience is unforgettable, largely due to a perception
rather than any reality related reliability. And its just way
too much fun to blame Berkeley DB "corruption" in bug reports
than it is to actually look at root causes of failures.

> Admittedly, the cost of this with SKS is not really that high. But
> it has substituted redundancy for reliability. I would argue for
> having both and the only reason I'm seeing for not having both is
> an appeal to tradition.
> 

Appeal to tradition? Nah, SKS+OCAML+BDB is a really really complex
piece of code to change.

I'm sure the ancient chant:
        Patches cheerfully accepted.
applies here to SKS key servers as well.

73 de Jeff