[Monotone-devel] Storing multiple Maildirs in a single monotone branch.

[jack-monotone]

To be filed under tangential-uses-of-monotone...

I have inboxes across several hosts stored in djb's Maildir format. 

One of the things I would like to be able to do is merge/distribute all of
those Maildirs to other hosts. 

Discounting how much simpler this would be to with rsync, I decided to try
monotone for this. 

The attached perl script issues the appropriate monotone add/drop/rename
commands, for new/deleted and modified messages. 

It does this by keeping track of the inodes and filenames in the Maildirs, and
calculating the difference between the last current and last state of the
Maildir. 

This script is run from cron. I'm currently using it to sync a few of my
inboxes of about 40k messages (~300MiB), and about 1000 add/drop/rename
operations a day.  The script typically takes about a minute to run, and a
netsync takes about 2 minutes. 

--Jack 

--
#!/usr/bin/env perl
use warnings;
use strict;

use DB_File;
use File::Find;
use File::Basename;
use Data::Dumper;

my $MAIL_BASE="$ENV{'HOME'}/Mail";
my $INODE_CACHE="${MAIL_BASE}/inode_cache.db";
my @FOLDERS=('inbox', 'sent');

my (@add,@drop,@rename);
my (%new_inodes,%done);

tie my %old_inodes, "DB_File", $INODE_CACHE, O_CREAT|O_RDWR, 0644, $DB_HASH 
        or die "Cannot open file $INODE_CACHE: $!\n"; 

chdir $MAIL_BASE;

print "scanning inodes...\n";

foreach my $folder (@FOLDERS) { 
        print "${MAIL_BASE}/${folder}\n";
        File::Find::find({no_chdir => 1, wanted => \&wanted}, "${folder}");
}

my @keys  = keys %old_inodes;
push @keys, keys %new_inodes;

print "finding differences...\n";

foreach my $k (sort(@keys)) { 
        next if defined $done{$k} ;

        # In old, but not in new -> drop
        if (not defined $new_inodes{$k}) { 
                push @drop, $old_inodes{$k};
                $done{$k}=1;
                next;
        }

        # In new, but not in old -> add
        if (not defined $old_inodes{$k}) { 
                push @add, $new_inodes{$k};
                $done{$k}=1;
                next;
        }

        my $nfile=$new_inodes{$k};
        my $ofile=$old_inodes{$k};

        # inodes same, and files are the same -> do nothing
        if ($nfile eq $ofile) {
                $done{$k}=1;
                next;
        }
        
        $nfile =~ s/:.*//;
        $ofile =~ s/:.*//;
                # strip off the trailing garbage. 

        # inodes same, and the msg basename is the same -> move
        if ( basename($nfile) eq basename($ofile) ) { 
                push @rename, [ $old_inodes{$k}, $new_inodes{$k} ] ; 
                $done{$k}=1;
        } else { 
        # inodes same, and the msg basename different. Reused inode. Drop old,
        # add new
                push @drop, $old_inodes{$k};
                push @add, $new_inodes{$k};
                $done{$k}=1;
        }
}

#XXX: Bleech. There's gotta be a better way to chunk the arrays through to 
monotone. 

my $commit=0;
my $work=0;

print "marking differences...\n";

while ( $#add > $work ) { 
        my @cmd=('monotone', 'add');
        my $chunk=(($#add - $work) > 500) ? 500 : $#add-$work; 
        push @cmd, @add[$work..$work+$chunk];

        system(@cmd) == 0 or die "system @cmd failed: $?"; 
        $work+=$chunk;
        $commit=1;
}
$work=0;
while ( $#drop > $work ) { 
        my @cmd=('monotone', 'drop');
        my $chunk=(($#drop - $work) > 500) ? 500 : $#drop-$work; 
        push @cmd, @drop[$work..$work+$chunk];

        system(@cmd) == 0 or die "system @cmd failed: $?";
        $work+=$chunk;
        $commit=1;
}

# have to iterate through the renames. 
foreach my $e (@rename) { 
        my @cmd=('monotone', 'rename');
        push @cmd, @{$e};
        system(@cmd) == 0 or die "system @cmd failed: $?";
        $commit=1;
}

if ( $commit != 0 ) { 
        my @cmd=('monotone', 'commit', '-m', 'automatic commit');
        system(@cmd) == 0 or die "system @cmd failed: $?";
}

# XXX: Bleech, There's probably a better way of doing this. 

print "updatin inode cache...\n";

foreach my $k (keys %old_inodes) { 
        delete $old_inodes{$k}; 
}

foreach my $k (keys %new_inodes) { 
        $old_inodes{$k} = $new_inodes{$k}; 
}

untie %old_inodes; 

##############################

sub wanted { 
        my $f=$_;
        ! -f $f && return ;
        my ($dev,$ino) = lstat($f);


        $new_inodes{$ino}=$File::Find::name;
}

--

--
Jack (John) Cummings                           http://mudshark.org/
PGP fingerprint: 0774 D073 E386 B70B 6B16  2D2B 1DD8 F8B0 CCF0 FAEE
Now playing on Prime:    Thru' These Architect's Eyes -- David Bowie
Now playing on Remedial: My Shoes -- Spock's Beard

pgpLhg7SDpN7k.pgp
Description: PGP signature