[Monotone-devel] Re: [Botan-devel] memory leak in Botan::Pipe?

[Jack Lloyd]

Hi Nathaniel,

Sorry, your mail was filtered into my Monotone mailbox instead of my
Botan one, and so got to spend some time in email purgatory. The fix
you're looking for was made in 1.4.12, the specific patch being:

--- Botan-1.4.10/src/out_buf.cpp        2005-12-18 17:19:29.000000000 -0500
+++ Botan-1.4.12/src/out_buf.cpp        2006-01-15 23:13:54.000000000 -0500
@@ -1,6 +1,6 @@
 /*************************************************
 * Pipe Output Buffer Source file                 *
-* (C) 1999-2005 The Botan Project                *
+* (C) 1999-2006 The Botan Project                *
 *************************************************/
 
 #include <botan/out_buf.h>
@@ -65,6 +65,7 @@
       {
       if(buffers[0] == 0 || buffers[0]->size() == 0)
          {
+         delete buffers[0];
          buffers.pop_front();
          offset++;
          }

-Jack


On Sat, Sep 02, 2006 at 08:26:32AM -0700, Nathaniel Smith wrote:
> We were noticing some serious resource usage problems in 'mtn
> checkout' on large trees, and the evidence seems to be pointing to
> some sort of problem in Botan.  On a checkout of h:net.venge.monotone,
> a medium-sized tree, valgrind reports ~5 megabytes of leaked data, all
> of data allocated at line 189 of Botan's pipe.cpp:
> 
> void Pipe::find_endpoints(Filter* f)
>    {
>    for(u32bit j = 0; j != f->total_ports(); ++j)
>       if(f->next[j] && !dynamic_cast<SecureQueue*>(f->next[j]))
>          find_endpoints(f->next[j]);
>       else
>          {
>          SecureQueue* q = new SecureQueue;       // <---- right here
>          f->next[j] = q;
>          outputs->add(q);
>          }
>    }
> 
> The application code that (eventually) calls find_endpoints and
> triggers the leaks is (with comments and asserts removed):
> 
>   {
>     ofstream file(tmp.as_external().c_str(),
>                   ios_base::out | ios_base::trunc | ios_base::binary);
>     Botan::Pipe pipe(new Botan::DataSink_Stream(file));
>     pipe.process_msg(dat());
>   }
> 
> This doesn't _look_ like we're doing anything wrong to me... I don't
> understand the memory handling strategy that Pipe is using, though, so
> I don't know where the actual bug is.
> 
> This problem was initially encountered when doing tests on a giant
> repository (100,000+ files, 1+ gigabytes), where monotone mysterously
> ended with hundreds of megabytes of resident memory, presumably
> because of this leak; so this is a relatively serious problem.
> 
> (This massive leak also triggered pathological behavior in Botan's
> memory management routines -- somewhere over 97% of cpu time was
> being spent in Pooling_Allocator::deallocate.  Glibc's default
> heap routines were able to scale to this load easily, though, when I
> tried swapping in a trivial malloc-based allocator -- is there really
> any reason to use Pooling_Allocator for unlocked memory?)
> 
> -- Nathaniel
> 
> -- 
> "But in Middle-earth, the distinct accusative case disappeared from
> the speech of the Noldor (such things happen when you are busy
> fighting Orcs, Balrogs, and Dragons)."
> _______________________________________________
> botan-devel mailing list
> address@hidden
> http://www.randombit.net/mailman/listinfo/botan-devel