Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited)

[Jeff Johnson]

On Feb 26, 2010, at 4:58 PM, Kim Minh Kaplan wrote:


>> Meanwhile, db_recover is only as good as the logs that are present,
>> often only back to the last hourly checkpoint. If logs are being
>> automatically removed, damage to data in the secondary indices earlier
>> than the last checkpoint cannot be repaired without db_dump/db_load
>> afaict.
> 
> This is not my understanding of the phrase "All committed transactions
> are guaranteed to appear after db_recover has run, and all uncommitted
> transactions will be completely undone"[1].  As I understand it, after a
> db_recover the database should contain *everything* that has been
> committed.  Regarding automatic removal of log files, I hope you are
> refering to "log files that are no longer in use"[2] in which case they
> are not needed for normal db_recover operation (i.e. to recover from a
> crash of SKS).  They are only needed when catastrophic failure occur but
> I do not think this what we are talking about.

I am talking about catastophic recovery, particularly
in the sense of hardening, as in not having to reload
an entire database, for certain types of failures.

Examine the size of your logs, and the size of the tables, in KDB/*.

The logs should be approx. the same size (key material is rather uncompressible)
as the tables in order to guarantee *everything* can be recreated.

My logs (particularly after running
        db_checkpoint -1
        db_archive -dv
are not sufficient to recreate the database in its entirety, just by
looking at the size of the files involved.

The definition for catastrophic recovery depends on the size of the logs that 
are kept.

>> If the secondary indices get out of sync with the primary key store, there's
>> a class of lookup failure "weirdness" that force recreating the key database 
>> from a dump.
>> 
>> With DB->associate
> 
> SKS does not use secondary indices in the sense of DB->associate.
> 

What is the schema in use for the KDB tables? I'm looking
for the {key,data} definitions for put operations performed
on the tables in KDB in particular.

73 de Jeff