[Monotone-devel] Re: monotone CVS import failed.

[Jon Smirl]

On 10/17/06, Markus Schiltknecht <address@hidden> wrote:
> Hi,
>
> Jon Smirl wrote:
> > I have been trying with cvs2svn for three months now and progress
> > isn't happening. You at least seem interested in making things work
> > right for Mozilla.
>
> I don't think so. The graph algorithm is just very new and we still have
> to experiment with it. I don't know about the details why cvs2svn (graph
> based) fails with importing Mozilla, though.
>
> But for sure the cvs2svn people are more experienced with CVS
> repositories and I'm very much basing my work on their experience. To be
> honest, I'm unsure if monotone will give you better results than cvs2svn
> anytime soon (or ever at all). Although, monotone is obviously closer to
> git than svn.

There experience is good and it is definitely worth using.  I
explained the problem with branch/symbol bases in another email.

> >> Jon Smirl wrote:
> >> > You can get the Mozilla CVS repository here:
> >> > rsync -az cvs-mirror.mozilla.org::mozilla ~/mozilla/cvs-mirror
> >> > The import was about 85% through the revs, 1.1M total.
> >>
> >> Why do you think it's 85%? Are you expecting more than 894,279 versions?
> >> How did you count them? I'm sure there's a difference in counting or a
> >> bug in my code ;-)
> >
> > cvs2svn reports somewhere between 1.0 and 1.1M revisions.
>
> Can anybody here quickly explain how cvs2svn counts? Does it count tags,
> branch identifiers?
>
> I'll check how monotone cvs_import counts. I didn't write that portion
> of the code myself. But certainly these numbers should match (or we
> should know why they don't.)

I'm rerunning a pass 1 cvs2svn on mozcvs as I type. I will let you
know the exact counts.

> > Let me know as soon as your code is ready. I know where most of the
> > tricky cases are located in the Mozilla repo. We can work together to
> > get me a good import and you some debugged code.
>
> Sure, but as I told you, the cvs2svn people are probably more
> knowledgeable than me WRT strangeness in CVS.
>
> >> Did you watch memory consumption?
> >
> > Around 1.2GB when it died.
>
> That's good to know. IMHO it's the main difference between my monotone
> cvs_import rewrite and cvs2svn's graph based approach: the 'in-memory'
> vs. 'on-disk' issue.
>
> It convinces me that spilling to disk is not necessary, because it looks
> like the whole mozilla repository with all its blobs and its
> dependencies fits into 1.2 GB of memory (this is of course excluding the
> files and its deltas itself).

I've noticed that most of the memory consumption in programs like this
comes from tracking CVS file names and revision numbers. In my git
importer I write the revision into the git data store the first time
it is parsed out of the ,v. Now I can simply track the sha1 through
all of the sorts, no need to keep CVS file names and rev numbers
around.


> Regards
>
> Markus


--
Jon Smirl
address@hidden