Re: [Sks-devel] 3 million keys & and community help requested

[Jeffrey Johnson]

On Nov 3, 2011, at 10:50 AM, Robert J. Hansen wrote:

>> Reliable is *entirely* in the eye of the beholder.
> 
> Agreed.
> 
>> The core problem is extrinsic (as in dependent on the amount of
>> data) interdependent tunables...
> 
> While I agree with your discussion of the external factors, I'd like to
> point out that any modern database -- regardless of whether it's MySQL,
> MariaDB, Postgres, BDB, or what-have-you -- will have approximately the
> same amount of external factors.
> 

The point isn't so much that extrinsic exists, but rather that tuning
is needed. Berkeley DB is quite tunable *but you have to do that*.

Meanwhile BDB tuning is a bit more obscure than most databases. That
is by no means a fault of the documentation, which is quite wonderfully
complete, just not an easy bed-time storybook read ...

> As far as I can see no vendor has yet managed to field a database that
> yields good results on real-world datasets without some sort of tuning.
> 
> This is of course not in any way a disagreement with your statement:
> just further emphasis.

;-) I'm sure I'm preaching to the choir …
… meanwhile SKS most definitely could use some better tuning examples,
and some database repair scripts.

There's also automated deadlock detection that COULD be run on
the PTree store which is tricky QA because of the recursion, and
is also sensitive to the actual access patter, with a failure
of self deadlock.

Short answer:
        Its not enough to test statistically by watching/waiting.
        Specific self deadlocking recursions may exist in spite
        of attempting to test.

See the claim that 4K -> 8K for page size was needed to avoid self dead locking.

I would expect the opposite behavior with per-page locks: larger pages are
more likely to self-deadlock in general recursions with pages containing
multiple items.

hth

73 de Jeff