[Gnu-arch-users] Re: [PATCH] arch speedups on big trees

[Chris Mason]

On Thu, 2004-01-08 at 20:38, Miles Bader wrote:
> Chris Mason <address@hidden> writes:
> > This either doesn't happen, or happens because arch is doing too many
> > inventories anyway.  Once you take out some of the extra inventories,
> > the inode sigs make less sense.
> 
> Huh?  Inode caches help inventories, but they help changeset-creation even
> more.  On source trees I use, changeset-creation _with_ inode caches is
> pretty reasonable, but _without_ them, it's a disk-killing operation even
> with kernel caching of file contents (because two copies of the tree's
> data is [orig+modified] tree is bigger than the RAM available for caching).

inode sigs do make tree inventory reads faster, but at the cost of
making tree changes much slower.

And the at the end of the commit arch walks the whole tree to redo the
inode sig.  It also walks the whole pristine tree to redo the inode sig
on the pristine tree.  If you're using a revision library instead, it
would have walked the whole revision library to snap the sig at the
start of the commit.

Miles, just for fun, put a big tree into a temporary archive and give my
patch a try.  Run update, replay, commit, changes and star-merge on a
repository with a hard linked pristine tree or revision library.  If it
isn't noticeably faster I'll buy you many beers at the next conference
we both attend ;-)

Fixing inode sigs isn't impossible,  when you build the revision, send
the inventory already taken during build to the inode sig creation
funcs.

While applying a changeset, change the sigs using an inode sig format
that allows for partial updates.  If Tom really doesn't want an indexed
database file, the sig checksums and other important metadata could go
into the files in ++id-mapping in my current patch.

> 
> > 2) Does a reverse mapping safely allow arch_apply_changeset to skip
> > whole tree inventories?  I provided a sample reverse mapping
> > implementation to help argue that it does.  It's fine if you don't
> > like the sample implementation, I'd rather discuss the safety of the
> > concept first.
> 
> Is a reverse-mapping even necessary?

The early versions of my code didn't have the reverse mapping, and after
Tom clued me into some of the issues involved, I took an approach very
similar to what you describe below.  The problem is that for changesets
that add files, a full inventory would be required to makes sure that id
doesn't already exist in the tree.  

The patches that I need to apply frequently add files, so this method
was too slow for my usage.

-chris