Re: [Sks-devel] Slow syncing (was: Re: Memory Leak in recon server?)

[Ryan]

I had the same issue develop last year, never figured out why but when the 
server was unexpectedly rebooted it came back and started working fine.  I 
recompiled/rebuilt the db, everything and nothing worked, it'd get stuck on a 
loop requesting same keys over and over again never getting anywhere.

I retired the server shortly after this reboot and I have not had any issues 
w/my current setup.

Cheers,
-Ryan


On Feb 2, 2010, at 6:28 PM, Arnold wrote:

> On 02-02-10 16:05, Phil Pennock wrote:
>> On 2010-02-02 at 01:19 +0100, Arnold wrote:
>>> If it is not in the number of peers, then what can I do further to make it
>>> sync more quickly?
>> 
>> Do the recon logs contain anything to suggest a cause for not fetching
>> all keys?  What about if you bump up the debug level?
> 
> As far as I can see / understand, it is spending most of the time examining
> blocks of 100 keys (the hashes I assume) and find they are the same.
> 
> Current log level is 4 (can set it higher tomorrow if necessary). Below some
> lines from my log with update activity. Half an hour before it also found a
> difference... The db.log tells me that I've no e-mail peers, which is indeed
> the case.
> 
> One thing I now noted is that most lines of the form
> Requesting 100 missing keys from <ADDR_INET ...:11371>, starting with
> 1ED038D719CE90EEF95B6F185D191EA7
> start with 1D..... or 1E..... Is that OK, or does it mean it is looping
> somewhere?
> 
> If I miss or misinterpret something, please let me know!
> 
> 
> Teun suggested the capacity of my server would be insufficient. Looking at
> the memory use and CPU load (see below), I guess it will do fine. The
> average network traffic over the past 9 days is about 2% of my capacity,
> both upload and download. The statistics at
>       http://www.pool.ntp.org/scores/82.95.191.161
> show that the server is accurate in time keeping, synchronising over the
> net. So I've no network problems either.
> 
> 
> Jens suggested off-list to add only his server and set
>       http_fetch_size: 500
> until my server has caught up. I will try that tomorrow (going to bed now).
> For now I have added three more servers. However, the log shows some time
> peer A, then some time peer B, etc. So, more peers may indeed not do the
> trick, but only spread the traffic across the peers.
> 
> 
> Well, thank you all!
>   Arnold
> 
> 
> 
> $ free
>             total       used       free     shared    buffers     cached
> Mem:       2068328    2016324      52004          0     154700    1445292
> -/+ buffers/cache:     416332    1651996
> Swap:       497972        816     497156
> 
> $ top
> top - 01:50:08 up 9 days,  2:31,  3 users,  load average: 0.41, 0.36, 0.46
> Tasks: 153 total,   2 running, 151 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:   2068328k total,  2016760k used,    51568k free,   154816k buffers
> Swap:   497972k total,      816k used,   497156k free,  1445292k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 
> 32131 debian-s  20   0 35856  31m  28m S  1.3  1.5   0:53.81 sks
> 
> 
> 13462 list      20   0 10840 7832 2564 S  0.3  0.4   1:37.98 python
> 
> 
> 13524 mysql     20   0  126m  24m 5140 S  0.3  1.2   5:09.82 mysqld
> 
> 
>    1 root      20   0  2100  684  588 S  0.0  0.0   0:08.26 init
> 
> 
> 
> 
> In recon.log:
> 
> 2010-02-02 06:54:54 Requesting 100 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 1ECDEB960744508BCA9D77259DAFB81D
> 2010-02-02 06:54:56 100 keys received
> 2010-02-02 06:54:57 Requesting 100 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 1ED038D719CE90EEF95B6F185D191EA7
> 2010-02-02 06:54:58 100 keys received
> 2010-02-02 06:55:00 Requesting 83 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 1ED274B269B1A93E7434DD098FA12319
> 2010-02-02 06:55:01 83 keys received
> 2010-02-02 06:55:02 Added 3 hash-updates. Caught up to 1265090101.513311
> 2010-02-02 06:55:39 Recon partner: <ADDR_INET 76.184.64.189:11370>
> 2010-02-02 06:55:40 Initiating reconciliation
> 2010-02-02 06:55:43 Reconciliation complete
> 2010-02-02 06:55:43 10182 hashes recovered from <ADDR_INET 
> 76.184.64.189:11371>
> 2010-02-02 06:55:46 Requesting 100 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 00C0983B4B12FB65DFE4FB5992282753
> 2010-02-02 06:55:47 100 keys received
> 2010-02-02 06:55:49 Added 1 hash-updates. Caught up to 1265090147.860824
> 2010-02-02 06:55:49 Requesting 100 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 1DE771CA7BDC27F9BF991D9DDD3E3A4A
> 2010-02-02 06:55:50 100 keys received
> 2010-02-02 06:55:50 Requesting 100 missing keys from <ADDR_INET
> 76.184.64.189:11371>, starting with 1DE9463535E7F871D4CCE03BD31B61CE
> 2010-02-02 06:55:51 100 keys received
> 
> 
> In db.log:
> 
> 2010-02-02 06:54:53 Applying 0 changes
> 2010-02-02 06:54:56 Applying 0 changes
> 2010-02-02 06:54:57 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:54:58 Applying 0 changes
> 2010-02-02 06:55:01 0 potential merges found for keyid 79576865
> 2010-02-02 06:55:01 1 updates found before filtering
> 2010-02-02 06:55:01 0 potential merges found for keyid F871868C
> 2010-02-02 06:55:01 1 updates found before filtering
> 2010-02-02 06:55:01 0 potential merges found for keyid F30EFBE2
> 2010-02-02 06:55:01 1 updates found before filtering
> 2010-02-02 06:55:01 Applying 3 changes
> 2010-02-02 06:55:01 Adding hash 9BB6A361CD79739D3F2B4259D51CC582
> 2010-02-02 06:55:01 Adding hash E318F86F4FBF9BED718AFADADCD10E81
> 2010-02-02 06:55:01 Adding hash FD1CD68BE7B2B5DBAA7A772C562108B8
> 2010-02-02 06:55:02 Sending LogResp size 3
> 2010-02-02 06:55:07 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:55:17 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:55:27 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:55:37 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:55:47 <mail transmit keys> error in callback.: Failure("No
> partners specified")
> 2010-02-02 06:55:47 0 potential merges found for keyid FC7D2E6B
> 2010-02-02 06:55:47 1 updates found before filtering
> 2010-02-02 06:55:47 Applying 1 changes
> 2010-02-02 06:55:47 Adding hash 00C0983B4B12FB65DFE4FB5992282753
> 2010-02-02 06:55:49 Sending LogResp size 1
> 2010-02-02 06:55:50 Applying 0 changes
> 2010-02-02 06:55:51 Applying 0 changes
> 
> 
> _______________________________________________
> Sks-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/sks-devel

smime.p7s
Description: S/MIME cryptographic signature

PGP.sig
Description: This is a digitally signed message part