Re: [Monotone-devel] Why not sync to another local db??

[Will Robertson]

Michele,
  I don't know if it's quite what you're after, but there used to be a
feature called "Packet I/O" which seems to have been replaced by the
Netsync alg. for various reasons.  However, it was very handy because it
allowed you to do things like email change-sets, or output only the
change-sets you want to a text file, then apply those to another db.

  A conversation with Clemens Hintze proved very helpful, and I've
copied it here:





From:   Will Robertson <address@hidden>
To:     address@hidden
Subject:        [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Wed, 27 Apr 2005 11:04:18 +0100

This may be a silly question, but I can't seem to find any info. on the
Mailing List about how to use Packet I/O to copy the contents (or
changes since a certain date, etc) into a dump-file, such that you can
take it via sneaker-net to another machine and upload to another
Monotone repository.

This is quite important to me, as I there's no way for me to access a
netsync server from one of the repositories.

-- 
Will Robertson, System Administrator
Technology, Gordian Knot Ltd.


From:   address@hidden
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Wed, 27 Apr 2005 17:44:30 +0200 (CEST)  (16:44 BST)



On 27 Apr, Will Robertson wrote:
> This may be a silly question, but I can't seem to find any info. on

It isn't silly at all ...

> the Mailing List about how to use Packet I/O to copy the contents
> (or changes since a certain date, etc) into a dump-file, such that
> you can take it via sneaker-net to another machine and upload to
> another Monotone repository.

I had exactly the same problem. Unfortunately the packet mechanism is
only rudimentary since introduction of the Netsync protocol.

You can still dumping a single revision, manifest, all certs of a
revision or a certain file using Monotones commands 

   certs fdata fdelta mdata mdelta rdata

> This is quite important to me, as I there's no way for me to access
> a netsync server from one of the repositories.

I have written a small Tcl module for accessing a Monotone repository
and a Tcl script using this module that enable me to dump the full (or
delta) contents using Packet I/O between to arbitrary revisions. I use
this to dump a certain state, sending that packets via email to home
and import it via 'read' into my second repository.

The script is not very sophisticated, but it works for me. Perhaps it
could be of any usage for you? I am currently thinking about wrapping
it into a Starkit (or Starpack) to make a standalone out of it.

Let me know, if you would like to try it out. But please do not expect
any warranty or comprehensive documentation ;-)


Ciao,
Clemens.

PS: Unfortunately, the re-importing repository needs to contain the
    whole history of the revision(s) you are importing via Packet I/O,
    as Monotone does not read-in any version, whose ancestor it
    doesn't already contain ... :-(

-- 
Clemens Hintze  mailto: c.hintze (at) gmx.net


From:   Will Robertson <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Tue, 03 May 2005 16:58:47 +0100

Thanks, Clemens,
  Yes, I would be interested in seeing the TCL script, if only to avoid
having to work it all out myself!!

  Being able to send "patches" this way, that incorporate history, is
one of the main reason for me sticking with darcs (and I don't really
feel like debating any other pros or cons just at the moment);  I'm
afraid having to rely on full network access for sync'ing multiple
repositories is a bit of a show-stopper for me.
  Even Git seems promising in this regard, in-as-much as it's just a
bunch of files, and rsync is just a file-transfer mechanism.


On Wed, 2005-04-27 at 17:44 +0200, address@hidden wrote:
> 
> On 27 Apr, Will Robertson wrote:
> > This may be a silly question, but I can't seem to find any info. on
> 
> It isn't silly at all ...
> 
> > the Mailing List about how to use Packet I/O to copy the contents
> > (or changes since a certain date, etc) into a dump-file, such that
> > you can take it via sneaker-net to another machine and upload to
> > another Monotone repository.
> 
> I had exactly the same problem. Unfortunately the packet mechanism is
> only rudimentary since introduction of the Netsync protocol.
> 
> You can still dumping a single revision, manifest, all certs of a
> revision or a certain file using Monotones commands 
> 
>    certs fdata fdelta mdata mdelta rdata
> 
> > This is quite important to me, as I there's no way for me to access
> > a netsync server from one of the repositories.
> 
> I have written a small Tcl module for accessing a Monotone repository
> and a Tcl script using this module that enable me to dump the full (or
> delta) contents using Packet I/O between to arbitrary revisions. I use
> this to dump a certain state, sending that packets via email to home
> and import it via 'read' into my second repository.
> 
> The script is not very sophisticated, but it works for me. Perhaps it
> could be of any usage for you? I am currently thinking about wrapping
> it into a Starkit (or Starpack) to make a standalone out of it.
> 
> Let me know, if you would like to try it out. But please do not expect
> any warranty or comprehensive documentation ;-)
> 
> 
> Ciao,
> Clemens.
> 
> PS: Unfortunately, the re-importing repository needs to contain the
>     whole history of the revision(s) you are importing via Packet I/O,
>     as Monotone does not read-in any version, whose ancestor it
>     doesn't already contain ... :-(
> 


From:   Nathaniel Smith <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Wed, 4 May 2005 10:39:26 -0700  (18:39 BST)

On Tue, May 03, 2005 at 04:58:47PM +0100, Will Robertson wrote:
> Thanks, Clemens,
>   Yes, I would be interested in seeing the TCL script, if only to
avoid
> having to work it all out myself!!
> 
>   Being able to send "patches" this way, that incorporate history, is
> one of the main reason for me sticking with darcs (and I don't really
> feel like debating any other pros or cons just at the moment);  I'm
> afraid having to rely on full network access for sync'ing multiple
> repositories is a bit of a show-stopper for me.

Hmm, so all you want is the ability to say "take that revision and
that revision and that revision, and put them in a file"?  Yeah, that
should be quite straightforward to script up.  (Basically, for each
revision, use 'rdata' to get the revision packet, then run 'cat
revision' and parse out the new_manifest and old_manifest lines, then
run 'mdelta' to get the delta packet between them, then run 'cat
manifest' on each, scan through file-by-file, and for each file
mentioned on both sides with different hashes, use 'fdelta' on the two
hashes to get a file delta packet, and file files that are new in the
new revision, use 'fdata' to get a full file packet.)

(This will sometimes create full file packets when not strictly
necessary, e.g., when there are renames; but it hardly seems worth the
bother to do better in this case...)

>   Even Git seems promising in this regard, in-as-much as it's just a
> bunch of files, and rsync is just a file-transfer mechanism.

A monotone database is also just-a-file, and if you really want to
rsync it (dunno why you would, it doesn't take any less network
connectivity than netsync, and is less efficient), I bet it works
better than rsync'ing git repos :-).  (Because monotone's db stores
deltas, whereas currently in git, you have to transfer complete copies
of every changed file.)

-- Nathaniel

-- 
When the flush of a new-born sun fell first on Eden's green and gold,
Our father Adam sat under the Tree and scratched with a stick in the
mould;
And the first rude sketch that the world had seen was joy to his mighty
heart,
Till the Devil whispered behind the leaves, "It's pretty, but is it
Art?"
  -- The Conundrum of the Workshops, Rudyard Kipling


From:   Will Robertson <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Thu, 05 May 2005 12:13:35 +0000  (13:13 BST)

(see below)

On Wed, 2005-05-04 at 10:39 -0700, Nathaniel Smith wrote:
> On Tue, May 03, 2005 at 04:58:47PM +0100, Will Robertson wrote:
> > Thanks, Clemens,

[...]

> > afraid having to rely on full network access for sync'ing multiple
> > repositories is a bit of a show-stopper for me.
> 
> Hmm, so all you want is the ability to say "take that revision and
> that revision and that revision, and put them in a file"?  Yeah, that
> should be quite straightforward to script up.  (Basically, for each

[...]

OK, I'll have a go at it.  I couldn't find any docs on how each of the
rdata/mdelta/etc commands related to each other, so my
less-than-competent attempts just made me more confused.  The "old
behaviour" allowed me to ask monotone to just export all revision data
and history since such-and-such a revision/time, which could be applied
against a different repository, and that one would simply ignore any
data it already had.  This is what I do with darcs at the mo'.

> >   Even Git seems promising in this regard, in-as-much as it's just a
> > bunch of files, and rsync is just a file-transfer mechanism.
> 
> A monotone database is also just-a-file, and if you really want to
> rsync it (dunno why you would, it doesn't take any less network
> connectivity than netsync, and is less efficient), I bet it works

Assuming you use rsync across the network.  It's quite good at keeping
multiple local directories in sync (e.g a removable usb-key).

It's nice that git works at a file-system level, which means you can
take advantage of all the file-system tools out there without much
effort.

> better than rsync'ing git repos :-).  (Because monotone's db stores
> deltas, whereas currently in git, you have to transfer complete copies
> of every changed file.)

Yes, I've read a big debate on git vs. Mercurial, where Mercurial is
essentially git using file-deltas and file-history instead of just
collections of files and change-set history.

One of Mercurial's author's big arguments is that even if "disk is
cheap", bandwidth isn't, and rsync'ing a large repository is likely to
hurt at both the client's and the server's end.

-- 
Will.


From:   Nathaniel Smith <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Thu, 5 May 2005 09:36:18 -0700  (17:36 BST)

On Thu, May 05, 2005 at 12:13:35PM +0000, Will Robertson wrote:
> (see below)
> 
> On Wed, 2005-05-04 at 10:39 -0700, Nathaniel Smith wrote:
> > On Tue, May 03, 2005 at 04:58:47PM +0100, Will Robertson wrote:
> > > Thanks, Clemens,
> 
> [...]
> 
> > > afraid having to rely on full network access for sync'ing multiple
> > > repositories is a bit of a show-stopper for me.
> > 
> > Hmm, so all you want is the ability to say "take that revision and
> > that revision and that revision, and put them in a file"?  Yeah,
that
> > should be quite straightforward to script up.  (Basically, for each
> 
> [...]
> 
> OK, I'll have a go at it.  I couldn't find any docs on how each of the
> rdata/mdelta/etc commands related to each other, so my
> less-than-competent attempts just made me more confused.  The "old
> behaviour" allowed me to ask monotone to just export all revision data
> and history since such-and-such a revision/time, which could be
applied
> against a different repository, and that one would simply ignore any
> data it already had.  This is what I do with darcs at the mo'.

Nod.  What is also useful might be to add some automation to say
"everything since last time I ran this".  You can do this by taking
the output of "monotone automate leaves" and saving it to a file
after you run your script; then, when you want to find what's new
since that time, you run "automate leaves" again, and for each
revision it outputs, run "automate ancestry_difference <a new leaf>
<all of the old leaves>", combine the output of all these commands
into a list, remove duplicates, and run the above-described export
script on each.  (See the ciabot script in contrib/ for an example of
this technique.)

What this will miss is if you create certs on existing revisions,
e.g., if you tag one.  For that you'll just have to do something by
hand...

> > >   Even Git seems promising in this regard, in-as-much as it's just
a
> > > bunch of files, and rsync is just a file-transfer mechanism.
> > 
> > A monotone database is also just-a-file, and if you really want to
> > rsync it (dunno why you would, it doesn't take any less network
> > connectivity than netsync, and is less efficient), I bet it works
> 
> Assuming you use rsync across the network.  It's quite good at keeping
> multiple local directories in sync (e.g a removable usb-key).

Well, you can use rsync to make two local monotone databases the same
as well :-).  (Though, again, netsync might be somewhat safer.)

> It's nice that git works at a file-system level, which means you can
> take advantage of all the file-system tools out there without much
> effort.
> 
> > better than rsync'ing git repos :-).  (Because monotone's db stores
> > deltas, whereas currently in git, you have to transfer complete
copies
> > of every changed file.)
> 
> Yes, I've read a big debate on git vs. Mercurial, where Mercurial is
> essentially git using file-deltas and file-history instead of just
> collections of files and change-set history.
> 
> One of Mercurial's author's big arguments is that even if "disk is
> cheap", bandwidth isn't, and rsync'ing a large repository is likely to
> hurt at both the client's and the server's end.

Well, he's right :-).

(If you have some intelligence on the server, of course, you can do
better than rsync on bandwidth usage, and still store whatever you
want on disk.)

-- Nathaniel

-- 
"But in Middle-earth, the distinct accusative case disappeared from
the speech of the Noldor (such things happen when you are busy
fighting Orcs, Balrogs, and Dragons)."



From:   address@hidden
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Tue, 10 May 2005 20:19:41 +0200 (CEST)  (19:19 BST)



On  3 Mai, Will Robertson wrote:
> Thanks, Clemens,

You're welcome,

first let me beg pardon for answering you so late, but I was very
busy!

>   Yes, I would be interested in seeing the TCL script, if only to
> avoid having to work it all out myself!!

Okay, I have attached them to this mail. They are:

  monotone.tcl - The module for accessing the repository via monotone
                 executable, and
  export-tree.tcl - The proggy that is able to export in packet I/O
format.

Both files have to be put together into the same dir (export-tree.tcl
will try to find monotone.tcl in its directory).

Perhaps you have to adapt export-tree.tcl a bit (e.g. name of monotone
executable, path & name of tclsh, ...)

You may use it like this:

  $ export-tree.tcl -delta current af2d > delta.packets
  $ export-tree.tcl 768d > full-dump-of-single-rev.packets

You have to stay in the checked out working directory. 'current' is a
pseudo revision that matches the checked-out revision in that working
directory.

You can dump a revision range (1st example) or a single revision only
(2nd one). Using the switch -delta export delta dumps between
revisions (makes the dump much smaller). Without -delta the dumps are
whole content ones.

Today as Monotone does not support to import single revisions only if
the parent revision does not exists in importing database, the absence
of the -delta switch makes no sense anymore. In the past when Monotone
had no revisions but only manifests, you could import every single
manifest you like. To be sure you could import a full dump of a
manifest and it would be available in the target database regardless
if the ancestor manifest exists or not. But as mentioned, this is
impossible today! So do not use the full dump mechanism, it is
senseless.

Sorry that the scripts are a bit lacking in comments and docu, but I
hope they are still useful for you.

>   Being able to send "patches" this way, that incorporate history,
> is one of the main reason for me sticking with darcs (and I don't
> really feel like debating any other pros or cons just at the
> moment);

Yeah! But with the scripts you get some of the features back. But do
not forget what revision you dumped last to be imported. If you forgot
one of the ancestors and import you packets, they will not be
available to you until the revision chain is complete.

That means: suppose you have that revision chain in your source
repository:

   A -> B -> C -> D -> E -> F

and an empty target database. Then you dumped that chain in three
packets containing:

  1. A -> B -> C
  2. D
  3. D -> E -> F

You could /not/ first importing 2, then 1. But you could import

  a) 1 + 2 + 3
  b) 1 + 3

to get the whole revision chain into your target database. Again,
regardless if those dumps were delta or full one.

>  I'm afraid having to rely on full network access for
> sync'ing multiple repositories is a bit of a show-stopper for me.
>   Even Git seems promising in this regard, in-as-much as it's just a
> bunch of files, and rsync is just a file-transfer mechanism.

Exactly. I hope they will reintroduce that feature again. If not, I am
currently planning to implement some further Tcl scripts, that will
use more logic to produce those packets and remembering what ones
already produces for what targets to avoid unnecessary revisions.

(...)


Best regards,
Clemens.
-- 
Clemens Hintze  mailto: c.hintze (at) gmx.net


From:   Will Robertson <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Thu, 12 May 2005 14:58:00 +0000  (15:58 BST)

On Tue, 2005-05-10 at 20:19 +0200, address@hidden wrote:
> 

[...]

> You're welcome,
> 
> first let me beg pardon for answering you so late, but I was very
> busy!

No problem, better late than never   ;-)
But very, very helpful.

[...]

> Today as Monotone does not support to import single revisions only if
> the parent revision does not exists in importing database, the absence
> of the -delta switch makes no sense anymore. In the past when Monotone
> had no revisions but only manifests, you could import every single
> manifest you like. To be sure you could import a full dump of a
> manifest and it would be available in the target database regardless
> if the ancestor manifest exists or not. But as mentioned, this is
> impossible today! So do not use the full dump mechanism, it is
> senseless.

That's interesting:  I had stopped using Monotone for a while, and
hadn't realised they'd intorduced such requirements -- I s'pose
somewhere around when the NetSync protocol was developed, with it's
smarter sync methods.

[...]

> Yeah! But with the scripts you get some of the features back. But do
> not forget what revision you dumped last to be imported. If you forgot
> one of the ancestors and import you packets, they will not be
> available to you until the revision chain is complete.
> 
> That means: suppose you have that revision chain in your source
> repository:
> 
>    A -> B -> C -> D -> E -> F
> 
> and an empty target database. Then you dumped that chain in three
> packets containing:
> 
>   1. A -> B -> C
>   2. D
>   3. D -> E -> F
> 
> You could /not/ first importing 2, then 1. But you could import
> 
>   a) 1 + 2 + 3
>   b) 1 + 3

So, does that mean if you try to import the same revision(s) more than
once -- i.e if part of an exported packet -- the duplicated revisions
will be ignored/skipped, and only the new revisions in that
revision-chain will be imported?

I mean: when importing, will it fail and stop importing if it hits a
revision which is already in the repository, thereby not importing
revisions that come after that duplicated one, or will it attempt to
apply all the revisions in a packet?

The reason why I ask is that there may be times when I have forgotten
exactly which revisions I've applied to which repository, and it would
be easier to guess and simply create a packet that will have more
revisions than I need (on purpose) in order to be sure that I've not
missed any.

[...]

> Exactly. I hope they will reintroduce that feature again. If not, I am
> currently planning to implement some further Tcl scripts, that will
> use more logic to produce those packets and remembering what ones
> already produces for what targets to avoid unnecessary revisions.
> 
> (...)
> 
> 
> Best regards,
> Clemens.
> -- 
> Clemens Hintze  mailto: c.hintze (at) gmx.net

Thanks again, Clemens.

Do you mind if I bounce/forward your email and scripts back into the
mailing-list so others can see them?
Maybe even persuade some C++ developer to reintroduce something
similar   ;-)

-- 
Will



From:   Nathaniel Smith <address@hidden>
To:     address@hidden
Subject:        Re: [Monotone-devel] Packets or Packet I/O no longer works?
Date:   Thu, 12 May 2005 13:50:55 -0700  (21:50 BST)

On Thu, May 12, 2005 at 02:58:00PM +0000, Will Robertson wrote:
> On Tue, 2005-05-10 at 20:19 +0200, address@hidden wrote:
> > Today as Monotone does not support to import single revisions only
if
> > the parent revision does not exists in importing database, the
absence
> > of the -delta switch makes no sense anymore. In the past when
Monotone
> > had no revisions but only manifests, you could import every single
> > manifest you like. To be sure you could import a full dump of a
> > manifest and it would be available in the target database regardless
> > if the ancestor manifest exists or not. But as mentioned, this is
> > impossible today! So do not use the full dump mechanism, it is
> > senseless.
> 
> That's interesting:  I had stopped using Monotone for a while, and
> hadn't realised they'd intorduced such requirements -- I s'pose
> somewhere around when the NetSync protocol was developed, with it's
> smarter sync methods.

It's because of the addition of revisions and changesets, actually; it
used to be that we didn't actually track history in any strong way, so
if you gave it some disjointed pieces of history, then nothing
broke... any worse than it was normally broken, anyway :-).

Now we have a strong sense of ancestry, so you can't just have some
revisions, without also having their parents.

> So, does that mean if you try to import the same revision(s) more than
> once -- i.e if part of an exported packet -- the duplicated revisions
> will be ignored/skipped, and only the new revisions in that
> revision-chain will be imported?
> 
> I mean: when importing, will it fail and stop importing if it hits a
> revision which is already in the repository, thereby not importing
> revisions that come after that duplicated one, or will it attempt to
> apply all the revisions in a packet?

It should import everything new and ignore everything old.  Most
operations in monotone are idempotent this way.

-- Nathaniel

-- 
"Lull'd in the countless chambers of the brain,
Our thoughts are link'd by many a hidden chain:
Awake but one, and lo! what myriads rise!
Each stamps its image as the other flies"
  -- Ann Ward Radcliffe, The Mysteries of Udolpho