Re: [Sks-devel] disk full, keys.niif.hu crashed

[Phil Pennock]

On 2018-06-15 at 09:40 +0200, André Keller wrote:
> On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote:
> > Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons
> > of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem
> > got fulfilled.
> > Deleting files and restarting processes did not help:
> 
> keys.communityrack.org shares the same fate. Trying to get it online
> again...

sks-peer.spodhuis.org saw a spike at the same time, AWS CloudWatch
metrics show that the dedicated EBS volume used for /var/sks hit 175,000
write operations per minute, when it's usually around 22,000 peaking
around 56,000.

The write _bytes_ is peaking around the same as normal, so throughput is
probably capping out.  I actually used some of the burst credits I had.

I'm in the middle of migrating OS-view metrics monitoring, in part to
handle having moved SKS into AWS, and don't currently have graphs
showing change in used capacity.  I'm currently at 30GB in use.

I see no change in rate of new keys or updated keys.  I do see 21GiB in
use for the DB directory.

-Phil