Server virtual memory exhausted

[Yaron Yogev]

Hello,

We have a very large repository, and one of the developers reported he
got the "Could not map memory to RCS archive" error towards the end of
a checkout of a module including most of this repository.

I tried this myself, and the same thing happened with my checkout. We
use a client-server setup using pserver, so I used 'top' to look at
the checkout process on the server, and saw that it was using so much
virtual memory that in the end it got near 3GB big.

I first looked at file sizes, and there were no RCS files bigger than
250MB in that repository.
I than ran CVS with valgrind, and it produced the following output for
checkout of a ~15 MB big CVS tree with 1539 files:

yarony@cvs yarony$ valgrind --show-reachable=yes --leak-check=full cvs
-d /arch/cvs co -d test project/tools
==12305== Memcheck, a memory error detector for x86-
linux.
==12305== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et
al.
==12305== Using valgrind-2.4.0, a program supervision framework for
x86-linux.
==12305== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et
al.
==12305== For more details, rerun with: -
v
==12305==
cvs checkout: Updating
test
U
test/.cvsignore
U test/
Makefile
<CVS checkout output removed>
==12305==
==12305== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from
1)
==12305== malloc/free: in use at exit: 5660375 bytes in 45589
blocks.
==12305== malloc/free: 439481 allocs, 393892 frees, 17755021 bytes
allocated.
==12305== For counts of detected errors, rerun with: -
v
==12305== searching for pointers to 45589 not-freed
blocks.
==12305== checked 5942472
bytes.
==12305==
==12305== 17 bytes in 1 blocks are still reachable in loss record 1 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1B9D408F: strdup (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x8077E60: main (main.c:
823)
==12305==
==12305==
==12305== 27 bytes in 2 blocks are still reachable in loss record 2 of
13
==12305==    at 0x1B90506F: realloc (vg_replace_malloc.c:
196)
==12305==    by 0x1BA271BE: getcwd (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x80CAB0A: xgetcwd (xgetcwd.c:
39)
==12305==    by 0x8078596: main (main.c:
943)
==12305==
==12305==
==12305== 34 bytes in 1 blocks are still reachable in loss record 3 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1B9C4DCA: vasprintf (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x80A087E: Xasprintf (subr.c:
1858)
==12305==    by 0x806ED79: set_readlock_name (lock.c:
502)
==12305==    by 0x806EE14: Reader_Lock (lock.c:
535)
==12305==    by 0x808F36E: do_recursion (recurse.c:
822)
==12305==    by 0x808F996: do_dir_proc (recurse.c:
1232)
==12305==    by 0x806815C: walklist (hash.c:
419)
==12305==    by 0x808F325: do_recursion (recurse.c:
877)
==12305==    by 0x808EB97: start_recursion (recurse.c:
455)
==12305==    by 0x80A4DA3: do_update (update.c:
539)
==12305==    by 0x80526E6: checkout_proc (checkout.c:
1145)
==12305==
==12305==
==12305== 34 bytes in 1 blocks are still reachable in loss record 4 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1B904FFA: realloc (vg_replace_malloc.c:
188)
==12305==    by 0x80CA91A: xrealloc (xmalloc.c:
74)
==12305==    by 0x80B68CE: xgethostname (xgethostname.c:
62)
==12305==    by 0x8077F93: main (main.c:
881)
==12305==
==12305==
==12305== 48 bytes in 6 blocks are still reachable in loss record 5 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x80B6661: SIG_register (sighandle.c:
292)
==12305==    by 0x806439B: signals_register (exithandle.c:
21)
==12305==    by 0x8078054: main (main.c:
957)
==12305==
==12305==
==12305== 156 (36 direct, 120 indirect) bytes in 1 blocks are
definitely lost in loss record 6 of 13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1BA43EE6: (within /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1BA43788: __nss_database_lookup (in /lib/tls/
libc-2.3.2.so)
==12305==    by
0x1BBABAFB: ???
==12305==    by 0x1BA04D4B: getpwuid_r (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1BA04590: getpwuid (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x809EF92: getcaller (subr.c:
325)
==12305==    by 0x806938D: history_write (history.c:
792)
==12305==    by 0x80528EA: checkout_proc (checkout.c:
1085)
==12305==    by 0x807A854: my_module (modules.c:
305)
==12305==    by 0x807AC0C: do_module (modules.c:
757)
==12305==    by 0x8051944: checkout (checkout.c:
386)
==12305==
==12305==
==12305== 40 bytes in 5 blocks are indirectly lost in loss record 7 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1BA43AAD: __nss_lookup_function (in /lib/tls/
libc-2.3.2.so)
==12305==    by
0x1BBABB21: ???
==12305==    by 0x1BA04D4B: getpwuid_r (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1BA04590: getpwuid (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x809EF92: getcaller (subr.c:
325)
==12305==    by 0x806938D: history_write (history.c:
792)
==12305==    by 0x80528EA: checkout_proc (checkout.c:
1085)
==12305==    by 0x807A854: my_module (modules.c:
305)
==12305==    by 0x807AC0C: do_module (modules.c:
757)
==12305==    by 0x8051944: checkout (checkout.c:
386)
==12305==    by 0x80781A4: main (main.c:
1162)
==12305==
==12305==
==12305== 80 bytes in 5 blocks are indirectly lost in loss record 8 of
13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1BA32143: tsearch (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1BA43A6E: __nss_lookup_function (in /lib/tls/
libc-2.3.2.so)
==12305==    by
0x1BBABB21: ???
==12305==    by 0x1BA04D4B: getpwuid_r (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1BA04590: getpwuid (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x809EF92: getcaller (subr.c:
325)
==12305==    by 0x806938D: history_write (history.c:
792)
==12305==    by 0x80528EA: checkout_proc (checkout.c:
1085)
==12305==    by 0x807A854: my_module (modules.c:
305)
==12305==    by 0x807AC0C: do_module (modules.c:
757)
==12305==    by 0x8051944: checkout (checkout.c:
386)
==12305==
==12305==
==12305== 260 bytes in 1 blocks are still reachable in loss record 9
of 13
==12305==    at 0x1B904F75: calloc (vg_replace_malloc.c:
175)
==12305==    by 0x80B64BA: SIG_init (sighandle.c:
129)
==12305==    by 0x80B65E6: SIG_register (sighandle.c:
238)
==12305==    by 0x806439B: signals_register (exithandle.c:
21)
==12305==    by 0x8078054: main (main.c:
957)
==12305==
==12305==
==12305== 352 bytes in 1 blocks are still reachable in loss record 10
of 13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x1B9BD10E: (within /lib/tls/
libc-2.3.2.so)
==12305==    by 0x1B9BF94B: fopen64 (in /lib/tls/
libc-2.3.2.so)
==12305==    by 0x80814CD: rcsbuf_cache_open (rcs.c:
2017)
==12305==    by 0x8086CC7: RCS_checkout (rcs.c:
5373)
==12305==    by 0x80A6000: checkout_file (update.c:
1319)
==12305==    by 0x80A4F23: update_fileproc (update.c:
666)
==12305==    by 0x808F653: do_file_proc (recurse.c:
959)
==12305==    by 0x806815C: walklist (hash.c:
419)
==12305==    by 0x808F214: do_recursion (recurse.c:
847)
==12305==    by 0x808F996: do_dir_proc (recurse.c:
1232)
==12305==    by 0x806815C: walklist (hash.c:
419)
==12305==
==12305==
==12305== 516 bytes in 2 blocks are still reachable in loss record 11
of 13
==12305==    at 0x1B90506F: realloc (vg_replace_malloc.c:
196)
==12305==    by 0x80CA8E2: xnrealloc (xmalloc.c:
74)
==12305==    by 0x80AB6C9: wrap_add_entry (wrapper.c:
455)
==12305==    by 0x80AB466: wrap_add (wrapper.c:
445)
==12305==    by 0x80AB0FE: wrap_add_file (wrapper.c:
267)
==12305==    by 0x80AAE8D: wrap_setup (wrapper.c:
105)
==12305==    by 0x8051511: checkout (checkout.c:
154)
==12305==    by 0x80781A4: main (main.c:
1162)
==12305==
==12305==
==12305== 9100 bytes in 1 blocks are still reachable in loss record 12
of 13
==12305==    at 0x1B904F75: calloc (vg_replace_malloc.c:
175)
==12305==    by 0x80B64D2: SIG_init (sighandle.c:
112)
==12305==    by 0x80B65E6: SIG_register (sighandle.c:
238)
==12305==    by 0x806439B: signals_register (exithandle.c:
21)
==12305==    by 0x8078054: main (main.c:
957)
==12305==
==12305==
==12305== 5649831 bytes in 45562 blocks are still reachable in loss
record 13 of 13
==12305==    at 0x1B90459D: malloc (vg_replace_malloc.c:
130)
==12305==    by 0x80CA893: xmalloc (xmalloc.c:
49)
==12305==    by 0x80CAAA3: xmemdup (xmalloc.c:
232)
==12305==    by 0x80A0856: Xstrdup (subr.c:
1844)
==12305==    by 0x80778C7: main (main.c:
553)
==12305==
==12305== LEAK
SUMMARY:
==12305==    definitely lost: 36 bytes in 1
blocks.
==12305==    indirectly lost: 120 bytes in 10
blocks.
==12305==      possibly lost: 0 bytes in 0
blocks.
==12305==    still reachable: 5660219 bytes in 45578
blocks.
==12305==         suppressed: 0 bytes in 0
blocks.

The one problem that stands out here is the last one: >5MB allocated,
still reachable, but in a place that seems to make no sense. It is in
main(), in the allocation of program_path.

Test environment:
CVS 1.12.13
Linux 2.6.18.3, Debian sarge distribution
Hardware: Pentium 4 3Ghz, 4GB Ram

Is this a known problem ?
Can anyone who understands the program better than me take a look at
this ?

Thanks in advance,
Yaron Yogev
IT Software Developer