Re: strcache scaling issue

[Paul Smith]

On Sun, 2011-01-23 at 09:59 +0100, Ralf Wildenhues wrote:
> * Paul Smith wrote on Sat, Jan 22, 2011 at 10:27:01PM CET:
> > Thanks for your investigations here; that's very helpful.  I had also
> > had some thoughts about improving the strcache in various ways.  For
> > example, I was thinking maybe about splitting it into two: one for
> > filenames and one for variable names, since these strings very rarely
> > overlap.
> 
> But a hash doesn't degrade in quality just because you put two different
> kinds of things in it.  I wouldn't go for more complexity here.  On the
> contrary, if you want an example where one big hash table is used for
> several different kinds of objects very successfully, look at git.

Hi Ralf.  I promoted a rework of strcache.c.  For your simple scaling
test on my system with debug compiled I get these results:
        
                                   Build Info
                                      1000
                                      2000
                                      4000
                                    3.82 -g
                                     2.61s
                                     8.85s
                                     33.52s
                                    3.82 -O2
                                     1.90s
                                     7.62s
                                     27.82s
                                     New -g
                                     (with
                                    asserts)
                                     1.03s
                                     2.31s
                                     5.79s
                                    New -O2
                                      (no
                                    asserts)
                                     0.65s
                                     1.50s
                                     3.52s

Not exactly linear yet but much improved.

However there are other issues I've seen.  First, I've discovered far
too much is being added to the strcache; all the temporary strings found
during implicit rule evaluation, even those which do not match, are
being permanently stored, which is bad!!

Second, related to my comment above on splitting into two caches: I
think I have a reason to want to do this.  On some types of make
compiles, we need to match filenames case-insensitively but we never
want to match variable names insensitively...!

So more work to come in these areas (among others).

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist