Re: flist -- "Killed" -- oom (*not* 1.8 related)

[Greg Minshall]

hi, Ken,

many thanks for the reply, and the sparkling idea at the end!

> Is there a long delay when you run flist?

yes, there is a long delay, as the system does some N^m processing.  the
system gets sluggish as always when there is a huge memory crunch.

>  Do you have a lot of folders?  Like a huge number?

well, huge is in the eye of the beholder (or, the malloc'r?):
----
bash archlinux: {49487} find ~/Mail -type d | wc
   2710    2710  113826
----

i run mh-e with emacs.  *most* of the folders come from synthetic
folders created when searching (using mairix, in my case) my e-mail
archives.

if i move those folders out, i have fewer
----
bash archlinux: {49489} find ~/Mail -type d | grep -v mhe-index | wc
   1577    1577   61380
----
and, flist works just fine, thank you very much.

i guess flist doesn't have a way of specifying folders *not* to search.
so, i will just "garbage-collect" most of these search-related folders
(by removing them).

>  I see that there are arrays allocated based on the number of folders
> you have.  I am just trying to figure out if there is a number of
> small allocations or large ones.  You could also disable OOM
> completely; I suspect flist will just segfault when it hits the limit.

> Oh, wait, I see that using limit/ulimit and setting the "datasize"
> limit should cause a SIGSEGV when it hits that limit.  So if you set
> that below the OOM limit that should make it easier to debug things.

ah, great!  yes, that works.  and, yes, to my ignorant eye, it appears
that the call from `folder_read()` to `mh_xmalloc()` is where we are
going south.

----
(gdb) run -all
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/minshall/src/import/nmh/git/nmh/uip/flist -all
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
flist: malloc failed, size wanted: 42269452928

Breakpoint 1, __GI_exit (status=1) at exit.c:137
137     {
(gdb) where
#0  __GI_exit (status=1) at exit.c:137
#1  0x000055555555a339 in adios (what=0x0, fmt=0x555555565178 "malloc failed, 
size wanted: %zu") at sbr/error.c:56
#2  0x000055555555f898 in mh_xmalloc (size=42269452928) at sbr/utils.c:38
#3  0x000055555555acf6 in folder_read (name=0x5555555d5400 
"/home/minshall/Mail/mhe-index", lockflag=0) at sbr/folder_read.c:138
#4  0x0000555555558440 in AddFolder (name=0x7fffffffa7a0 "mhe-index", force=1) 
at uip/flist.c:452
#5  0x0000555555558362 in BuildFolderListRecurse (dirName=0x555555564176 ".", 
s=0x7fffffffb7d0, searchdepth=0) at uip/flist.c:428
#6  0x000055555555805e in BuildFolderList (dirName=0x555555564176 ".", 
searchdepth=0) at uip/flist.c:352
#7  0x0000555555557fab in ScanFolders () at uip/flist.c:323
#8  0x0000555555557cdd in main (argc=2, argv=0x7fffffffda18) at uip/flist.c:244
----

presumably, "fixing this" isn't a priority, especially as i guess it
would require considerable changes to the code.

but, i'm glad to know what it is.  (and, i hope i remember it!)

cheers, Greg