Re: [Gnu-arch-users] [PATCH] arch speedups on big trees

[Chris Mason]

On Wed, 2004-01-28 at 15:23, Tom Lord wrote:
>     > From: Chris Mason <address@hidden>
> 
>     > I'll make you a deal, add 15,000 files any project tree that you
>     > work on constantly through the day.  Never change these files
>     > aside from when you add them, and add them inside any directory
>     > structure you like.
> 
>     > Use it for two weeks running basic commands like commit/changes and
>     > pulling in changes from other branches.  At least once pull in 50 or
>     > more changesets from another tree.  At the end of all of that, I'm
>     > pretty confident you'll think arch isn't usable on big trees.  
> 
>     > I'm sure there are better ways to make it faster than my current code,
>     > but I haven't seen a lot of other ideas discussed.
> 
> Were _I_ to do that, I would use tagline tags.
> 
>
> Were I working with such a tree actively, I'd expect to be able to
> expected-case inventory it in a very small number of seconds --
> perhaps even a fraction of a second depending on my hw.
> 
I think tagline have a few issues, please correct me if I've got some
facts wrong here.  They don't work when you don't control the entire
source.  So for mirroring an existing project that isn't using arch,
they are not very practical.

taglines don't reduce the need for inventories.  When applying a
changeset that creates a new id, you still need to inventory the entire
tree to figure out if that id already exists. 

> I would _guess_ that you are mostly using explicit tags and that that
> is a large part of the reason for your dissatisfaction.  That will,
> currently, tend to stress either kernel caches or i/o bandwidth
> noticably.
> 
> So, one of the better ways to make things faster for your situation
> would (most likely) be to extend the inode signature/id-tag
> optimizations so that they work for explicit tags.  This has the
> advantage of also being a much easier change to make.

One of the things that Miles and others on the list convinced me of was
that my reverse mapping was really just an extension of the inode sigs. 
Combined with the partial inventories, a well indexed inode sig would
work.

-chris