[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Savannah-hackers-public] rsync process load on vcs
|
From: |
Karl Berry |
|
Subject: |
Re: [Savannah-hackers-public] rsync process load on vcs |
|
Date: |
Wed, 25 Sep 2013 17:25:59 GMT |
Regarding sv load, as mentioned, /var/tmp/toplog on vcs has been
accumulating top snapshots every 20 minutes for a while now. (BTW,
there is also /var/log/atop.log*, set up by someone else long ago, but I
didn't research how to do anything with that.)
I just looked at it for the last week. The load average has been over
10 times, with a clear, though not universal predilection for times
around 07-08 and 13-14.
Ward, are you using --bwlimit with the rsync backup job? If not, could
you please institute that? My impression is that a lot of the problem
now is disk contention ...
Aside from that, some incarnation of cvs was in the top jobs every
single time. Python 16 times, Perl 10 times, git 10 times. Many of the
processes have been niced, but not all.
Unfortunately top does not show the full command line (argh), so it
is not simple to say what they are all doing, but I believe it is mostly
about web browsing of the source repositories, presumably by web
crawlers. I wonder about researching the Apache logs and seeing if we
can rate-limit access to the vc browsing, and/or nice everything that
gets started once and for all.
Below is a cut-down version of the data for perusal.
Thanks,
karl
Wed Sep 18 07:03:02 UTC 2013 top - 07:03:03 up 418 days, 23:40, 3 users, load
average: 15.48, 13.06, 8.46
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9427 jave 20 0 5040 3316 1680 D 6 0.1 0:00.03 /usr/bin/perl /usr/
8689 nobody 30 10 5880 1772 832 D 2 0.0 0:00.11 /usr/bin/cvs -f --a
9141 root 20 0 9500 3048 2420 S 2 0.0 0:00.02 sshd: karl [priv]
9391 tzz 20 0 15968 10m 2824 S 2 0.2 0:00.19 /usr/bin/python /us
--
Wed Sep 18 07:21:28 UTC 2013 top - 07:21:29 up 418 days, 23:58, 3 users, load
average: 10.37, 6.46, 6.99
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9784 nobody 20 0 185m 181m 3120 D 29 3.0 7:39.33 /usr/bin/python /us
13323 www-data 20 0 7932 6288 1724 S 17 0.1 0:00.09 /usr/bin/perl /usr/
6881 nobody 30 10 477m 379m 262m D 2 6.3 0:32.33 git pack-objects --
13313 gawkrsyn 20 0 2632 1100 956 S 2 0.0 0:00.01 /bin/sh /usr/local/
--
Wed Sep 18 08:01:38 UTC 2013 top - 08:01:40 up 419 days, 39 min, 3 users,
load average: 17.09, 10.67, 7.72
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21195 nobody 20 0 52336 48m 3092 R 86 0.8 0:08.44 /usr/bin/python /us
21493 tzz 20 0 22648 17m 3020 S 12 0.3 0:00.58 /usr/bin/python /us
20176 www-data 20 0 8360 6808 1772 S 2 0.1 0:00.56 /usr/bin/perl /usr/
21486 tzz 20 0 9636 2456 1680 S 2 0.0 0:00.01 sshd:
address@hidden
--
Wed Sep 18 13:02:05 UTC 2013 top - 13:02:07 up 419 days, 5:39, 3 users, load
average: 16.49, 11.48, 6.96
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17221 nobody 30 10 5960 2024 836 D 4 0.0 0:00.97 /usr/bin/cvs -f --a
4456 root 20 0 4144 336 236 S 2 0.0 20:13.86 /usr/sbin/sshd
15509 gnun 20 0 5616 1504 744 D 2 0.0 0:00.28 cvs server
17217 nobody 30 10 5148 1976 1588 S 2 0.0 0:00.48 /usr/bin/cvs -f --a
--
Wed Sep 18 14:01:16 UTC 2013 top - 14:01:22 up 419 days, 6:38, 3 users, load
average: 13.72, 9.83, 5.74
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28555 nobody 30 10 7216 3212 772 D 2 0.1 0:04.52 /usr/bin/cvs -f --a
30685 nobody 30 10 17200 2064 1816 S 2 0.0 0:00.04 git-upload-pack --s
30742 root 20 0 5172 2016 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
1 root 20 0 2044 500 472 S 0 0.0 8:42.24 init [2]
--
Thu Sep 19 06:41:01 UTC 2013 top - 06:41:04 up 419 days, 23:18, 3 users, load
average: 16.64, 7.92, 4.58
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14409 www-data 20 0 343m 132m 80m D 4 2.2 0:05.83 /var/www/git/cgit/c
14503 nobody 30 10 8448 4468 872 D 4 0.1 0:02.22 /usr/bin/cvs -f --a
14872 root 20 0 5164 2004 1484 R 4 0.0 0:00.02 /usr/bin/top -bcn 1
14866 www-data 20 0 4196 1328 924 D 2 0.0 0:00.02 /var/www/git/cgit/c
--
Thu Sep 19 07:01:15 UTC 2013 top - 07:01:16 up 419 days, 23:38, 3 users, load
average: 11.40, 9.08, 6.48
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19495 www-data 20 0 7812 6292 1724 S 19 0.1 0:00.10 /usr/bin/perl /usr/
17658 nobody 30 10 9484 5520 772 D 6 0.1 0:07.84 /usr/bin/cvs -f --a
19484 nobody 30 10 5272 1304 872 D 6 0.0 0:00.14 /usr/bin/cvs -f --a
18344 root 20 0 11604 8468 2732 D 2 0.1 0:02.12 /usr/bin/perl /usr/
--
Thu Sep 19 07:41:01 UTC 2013 top - 07:41:04 up 420 days, 18 min, 3 users,
load average: 13.43, 10.94, 8.87
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
70 root 20 0 0 0 0 S 2 0.0 312:53.01 [kswapd0]
1 root 20 0 2044 500 472 S 0 0.0 8:43.16 init [2]
2 root 20 0 0 0 0 S 0 0.0 0:00.00 [kthreadd]
3 root RT 0 0 0 0 S 0 0.0 14:53.29 [migration/0]
--
Thu Sep 19 13:41:42 UTC 2013 top - 13:41:43 up 420 days, 6:19, 3 users, load
average: 12.77, 12.09, 9.25
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17869 nobody 30 10 53872 19m 1884 S 10 0.3 0:03.79 /usr/bin/svnserve -
15028 nobody 30 10 9484 5520 772 D 2 0.1 0:06.54 /usr/bin/cvs -f --a
15164 root 39 19 39424 13m 676 D 2 0.2 0:12.92 rsync --server --se
18709 root 20 0 5204 2004 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Thu Sep 19 14:02:19 UTC 2013 top - 14:02:22 up 420 days, 6:39, 3 users, load
average: 16.14, 16.69, 13.91
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4 root 20 0 0 0 0 S 2 0.0 52:23.77 [ksoftirqd/0]
23545 root 20 0 5172 2024 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
1 root 20 0 2044 500 472 S 0 0.0 8:43.58 init [2]
2 root 20 0 0 0 0 S 0 0.0 0:00.00 [kthreadd]
--
Thu Sep 19 14:21:05 UTC 2013 top - 14:21:06 up 420 days, 6:58, 3 users, load
average: 10.05, 9.68, 11.42
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27127 www-data 39 19 9580 6204 2500 D 13 0.1 0:00.08 /usr/bin/python /va
26992 nobody 30 10 7696 3708 872 D 8 0.1 0:02.94 /usr/bin/cvs -f --a
15161 root 20 0 10772 4468 2572 S 2 0.1 0:09.09 sshd:
address@hidden
15164 root 39 19 61952 32m 680 S 2 0.5 0:45.94 rsync --server --se
--
Thu Sep 19 14:41:02 UTC 2013 top - 14:41:03 up 420 days, 7:18, 3 users, load
average: 21.57, 17.46, 14.67
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28895 www-data 20 0 462m 188m 4540 S 21 3.1 15:39.32 /usr/sbin/apache2 -
47 root 20 0 0 0 0 S 2 0.0 51:03.66 [kblockd/0]
70 root 20 0 0 0 0 S 2 0.0 313:08.12 [kswapd0]
29248 nobody 30 10 6992 2720 744 D 2 0.0 0:00.60 /usr/bin/cvs -f --a
--
Thu Sep 19 16:21:51 UTC 2013 top - 16:21:52 up 420 days, 8:59, 3 users, load
average: 13.65, 8.05, 5.37
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24478 root 20 0 5188 2028 1484 R 4 0.0 0:00.03 /usr/bin/top -bcn 1
70 root 20 0 0 0 0 S 2 0.0 313:16.34 [kswapd0]
14409 nobody 39 19 76376 34m 1780 S 2 0.6 0:21.82 rsync --daemon
23815 rmottola 20 0 5260 1968 1564 D 2 0.0 0:00.29 cvs server
--
Fri Sep 20 07:01:45 UTC 2013 top - 07:01:46 up 420 days, 23:39, 3 users, load
average: 15.74, 9.58, 5.87
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32260 nobody 30 10 8416 4568 872 S 5 0.1 0:07.77 /usr/bin/cvs -f --a
979 nobody 30 10 5272 1308 872 D 3 0.0 0:00.20 /usr/bin/cvs -f --a
32240 nobody 30 10 9016 5852 1620 S 3 0.1 0:03.55 /usr/bin/cvs -f --a
978 nobody 30 10 5148 1984 1588 S 2 0.0 0:00.04 /usr/bin/cvs -f --a
--
Fri Sep 20 07:41:03 UTC 2013 top - 07:41:04 up 421 days, 18 min, 3 users,
load average: 10.66, 11.62, 9.43
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5954 www-data 20 0 461m 280m 140m S 2 4.7 0:30.89 git pack-objects --
8269 www-data 20 0 0 0 0 D 2 0.0 0:00.02 [cgit.cgi]
8298 root 20 0 5164 2008 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
21513 www-data 20 0 556m 259m 4488 S 2 4.3 21:48.51 /usr/sbin/apache2 -
--
Fri Sep 20 08:01:56 UTC 2013 top - 08:01:57 up 421 days, 39 min, 3 users,
load average: 15.43, 14.06, 13.23
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12448 nobody 20 0 53664 49m 3092 R 101 0.8 0:24.57 /usr/bin/python /us
12692 nobody 20 0 23404 19m 3044 S 19 0.3 0:00.84 /usr/bin/python /us
21514 www-data 20 0 395m 135m 4544 S 4 2.3 17:08.09 /usr/sbin/apache2 -
12696 root 20 0 5148 2000 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Fri Sep 20 13:03:17 UTC 2013 top - 13:03:18 up 421 days, 5:40, 3 users, load
average: 16.39, 15.47, 9.25
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9857 nobody 30 10 5560 1608 872 D 4 0.0 0:04.68 /usr/bin/cvs -f --a
70 root 20 0 0 0 0 S 2 0.0 313:53.58 [kswapd0]
6142 root 20 0 12756 6528 2572 S 2 0.1 0:07.38 sshd:
address@hidden
11757 root 20 0 5168 2036 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Fri Sep 20 14:01:06 UTC 2013 top - 14:01:07 up 421 days, 6:38, 3 users, load
average: 11.81, 9.96, 6.70
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23472 www-data 20 0 8624 7204 1832 S 10 0.1 0:00.53 /usr/bin/perl /usr/
21500 nobody 30 10 9484 5520 772 D 2 0.1 0:06.09 /usr/bin/cvs -f --a
23450 www-data 20 0 8624 7192 1824 S 2 0.1 0:00.19 /usr/bin/perl /usr/
1 root 20 0 2044 500 472 S 0 0.0 8:44.92 init [2]
--
Sat Sep 21 13:02:07 UTC 2013 top - 13:02:07 up 422 days, 5:39, 3 users, load
average: 12.11, 9.29, 5.45
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15022 www-data 39 19 2177m 1.6g 2696 S 98 26.6 13040:04 python /usr/src/log
662 root 20 0 13492 7224 2572 S 2 0.1 0:08.53 sshd:
address@hidden
4043 barbier 20 0 5740 2140 1140 D 2 0.0 0:00.57 cvs server
5197 root 20 0 5160 2004 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Sat Sep 21 14:01:03 UTC 2013 top - 14:01:04 up 422 days, 6:38, 3 users, load
average: 11.30, 8.02, 6.79
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15022 www-data 39 19 2427m 2.1g 2696 S 98 36.0 13073:06 python /usr/src/log
16560 nobody 30 10 10600 7408 1604 S 3 0.1 0:00.11 /usr/bin/cvs -f --a
16584 www-data 20 0 4484 1164 964 D 2 0.0 0:00.01 /usr/bin/git --git-
1 root 20 0 2044 500 472 S 0 0.0 8:46.12 init [2]
--
Sun Sep 22 07:01:52 UTC 2013 top - 07:02:08 up 422 days, 23:39, 3 users, load
average: 23.28, 12.21, 7.39
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27532 root 20 0 5172 2024 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
1 root 20 0 2044 500 472 S 0 0.0 8:47.24 init [2]
2 root 20 0 0 0 0 S 0 0.0 0:00.00 [kthreadd]
3 root RT 0 0 0 0 S 0 0.0 15:08.86 [migration/0]
--
Mon Sep 23 07:01:16 UTC 2013 top - 07:01:17 up 423 days, 23:38, 5 users, load
average: 11.52, 8.23, 5.80
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15022 www-data 39 19 2427m 2.1g 2624 S 91 34.9 13403:28 python /usr/src/log
7647 nobody 30 10 9268 5424 872 D 4 0.1 0:06.64 /usr/bin/cvs -f --a
8564 nobody 30 10 5408 1480 876 D 4 0.0 0:02.36 /usr/bin/cvs -f --a
8327 nobody 30 10 5148 1984 1588 S 2 0.0 0:00.45 /usr/bin/cvs -f --a
--
Mon Sep 23 07:41:02 UTC 2013 top - 07:41:06 up 424 days, 18 min, 5 users,
load average: 16.90, 12.06, 9.29
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17295 root 20 0 7300 2844 1104 S 2 0.0 0:00.15 /usr/sbin/munin-nod
17623 www-data 20 0 4196 1320 924 D 2 0.0 0:00.02 /var/www/git/cgit/c
17667 root 20 0 5188 2012 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
17765 nobody 20 0 3884 2308 1528 R 2 0.0 0:00.01 /usr/bin/perl -w /e
--
Mon Sep 23 08:02:10 UTC 2013 top - 08:02:12 up 424 days, 39 min, 5 users,
load average: 17.53, 17.05, 14.74
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23403 nobody 20 0 56736 52m 3092 S 17 0.9 0:30.66 /usr/bin/python /us
21179 nobody 30 10 5540 2328 1616 S 2 0.0 0:00.90 /usr/bin/cvs -f --a
21442 nobody 30 10 7216 3216 772 D 2 0.1 0:03.30 /usr/bin/cvs -f --a
23423 nobody 30 10 12752 6592 1892 D 2 0.1 0:00.84 /usr/bin/svnserve -
--
Tue Sep 24 07:01:00 UTC 2013 top - 07:01:02 up 424 days, 23:38, 0 users, load
average: 10.71, 6.65, 4.88
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24 root 20 0 0 0 0 S 8 0.0 7:33.76 [events/0]
773 www-data 20 0 6376 1072 884 D 8 0.0 0:00.05 /usr/bin/git --git-
181 root 20 0 0 0 0 S 2 0.0 91:58.66 [flush-202:2]
575 www-data 20 0 8624 7196 1824 S 2 0.1 0:00.19 /usr/bin/perl /usr/
--
Tue Sep 24 08:01:01 UTC 2013 top - 08:01:02 up 425 days, 38 min, 0 users,
load average: 14.11, 14.47, 12.80
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17041 nobody 20 0 219m 216m 3120 S 15 3.6 5:41.63 /usr/bin/python /us
15193 nobody 20 0 54500 50m 3092 D 9 0.8 0:12.38 /usr/bin/python /us
13897 nobody 30 10 6464 2424 772 D 2 0.0 0:02.05 /usr/bin/cvs -f --a
16066 root 20 0 5040 1996 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Tue Sep 24 11:01:06 UTC 2013 top - 11:01:18 up 425 days, 3:38, 0 users, load
average: 18.35, 15.11, 8.44
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25314 root 39 19 48612 32m 680 D 6 0.5 0:41.72 rsync --server --se
29719 root 20 0 5196 1996 1484 R 4 0.0 0:00.02 /usr/bin/top -bcn 1
4 root 20 0 0 0 0 S 2 0.0 53:02.01 [ksoftirqd/0]
27968 nobody 30 10 5536 2328 1616 S 2 0.0 0:00.87 /usr/bin/cvs -f --a
--
Tue Sep 24 11:21:05 UTC 2013 top - 11:21:06 up 425 days, 3:58, 0 users, load
average: 26.79, 19.63, 14.49
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2093 www-data 20 0 8624 7132 1824 S 3 0.1 0:00.14 /usr/bin/perl /usr/
2066 nobody 30 10 7536 3688 872 D 2 0.1 0:00.27 /usr/bin/cvs -f --a
2108 root 20 0 5204 2012 1484 R 2 0.0 0:00.01 /usr/bin/top -bcn 1
25314 root 39 19 103m 45m 680 D 2 0.8 1:01.32 rsync --server --se
--
Tue Sep 24 15:01:08 UTC 2013 top - 15:01:10 up 425 days, 7:38, 0 users, load
average: 12.26, 7.22, 3.72
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16340 nobody 30 10 530m 213m 33m D 2 3.5 4:26.22 git pack-objects --
22778 nobody 20 0 35088 31m 2728 S 2 0.5 0:05.41 /usr/bin/python /us
29529 www-data 39 19 9584 6204 2500 D 2 0.1 0:00.21 /usr/bin/python /va
29601 root 20 0 5200 2004 1484 R 2 0.0 0:00.02 /usr/bin/top -bcn 1
--
Tue Sep 24 17:01:08 UTC 2013 top - 17:01:11 up 425 days, 9:38, 0 users, load
average: 11.75, 6.83, 3.51
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27902 gnun 20 0 5988 2548 1504 D 6 0.0 0:00.46 cvs server
26282 nobody 20 0 57920 53m 3124 D 2 0.9 0:07.82 /usr/bin/python /us
27829 gnun 20 0 12044 4548 1308 S 2 0.1 0:00.13 sshd:
address@hidden
27832 gnun 20 0 2512 1012 788 S 2 0.0 0:00.23 cvs server
- [Savannah-hackers-public] rsync process load on vcs, Karl Berry, 2013/09/02
- Re: [Savannah-hackers-public] rsync process load on vcs, Ward Vandewege, 2013/09/02
- Re: [Savannah-hackers-public] rsync process load on vcs, Ineiev, 2013/09/04
- Re: [Savannah-hackers-public] rsync process load on vcs, Karl Berry, 2013/09/05
- Re: [Savannah-hackers-public] rsync process load on vcs, Bob Proulx, 2013/09/22
- Re: [Savannah-hackers-public] rsync process load on vcs, Bob Proulx, 2013/09/23
- Re: [Savannah-hackers-public] rsync process load on vcs,
Karl Berry <=
- Re: [Savannah-hackers-public] rsync process load on vcs, James Cloos, 2013/09/27
- Re: [Savannah-hackers-public] rsync process load on vcs, James Cloos, 2013/09/28
- Re: [Savannah-hackers-public] rsync process load on vcs, Karl Berry, 2013/09/28
- Re: [Savannah-hackers-public] rsync process load on vcs, James Cloos, 2013/09/28