Re: [Ifile-discuss] Re: Maildir format

[William E. Kempf]

Brett Nemeroff said:
> Ooops! That wasn't supposed to be off-list.
>
> Ok, so my script is pretty much done. Use formail to add the header and
> output it to a $messagename.new file of sorts. So I am creating a new
> file, moveing the old one out of the way and then renaming the old file.
> No quesiton about  it, it's an entirely different file.
>
> I don't have too much of a hangup there. I understand that moving the
> messages around can result in data loss, but I think chances are on my
> side. The problem that I DO have is that the file date (of course) is
> reset to when it gets refiled.. So in my message list (in my client) the
> dates and times are all wrong. Does anyone know how to get around this?
> Could I either: 1. edit the original file and somehow retain the edit
> date, or 2. manually set the edit date of the new file to the original
> message date/time. I'm writing this in perl, so any suggestions using
> that would be great...
>
> OKOK, I know I just asked a PERL question. I promise that's the last
> time! :)
>
> Is there anyway to make formail just add the header and not modify the
> date/time??

I'm in a very similar situation.  I use Courier IMAP and SquirrelMail as
well, which means I use scripts that relearn/modify e-mail in Maildir
folders.  So this topic interests me.

First... to the risks of direct modification.  My understanding was that
the Maildir delivery system was mostly designed to work around NFS mount
problems, no?  I mean, short of a system crash during a write, how else
could direct manipulation be problematic?  And for that risk, is it not
identical to the risks associated with mbox manipulations?  Provided my
evaluation is correct, then all of this is pretty much a non-issue if your
Maildir is not residing on an NFS (or similar) mount.  (And there are
reasons to use Maildir other than for NFS safety!)

That said, I would think the Maildir mechanism should be useful for
manipulating files as well.

1) Read the mail message from $Maildir/cur/$msg and write modifications to
$Maildir/tmp/$msg.
2) Remove $Maildir/cur/$msg.
3) Hardlink $Maildir/tmp/$msg to $Maildir/cur/$msg.
4) Remove $Maildir/tmp/$msg.

Any failure should leave you with a valid file in either $Maildir/cur/$msg
or $Maildir/tmp/$msg (depending on when the failure occurred).  No chance
for data loss.  And by employing a Hardlink instead of doing a file move
and/or copy the process should be more efficient with less chance for
failure.

Do I have any of this wrong?

Some other things that need to be paid attention to:

* File names.  See http://cr.yp.to/proto/maildir.html.  Generally, this
process should probably leave the name entirely alone, but I still
wouldn't mess around here with out knowing this information.

* Related to the above... you may want to skip messages that don't have
the S info flag.  It's unlikely you've moved a message that you've not
read, so relearning it is likely wasted time.  However, I don't know how
universally the S flag is used by the various Maildir handling programs.

* File time stamps.  I believe the original file's time stamps should
probably be retained.  I believe Courier uses these for deliver times, and
modifying them can reorder your e-mail when sorted by delivery time. 
Probably not what you want to do, even though you have modified the file.

I'm very intersted in hearing where I have any of this wrong.  I'm also
intersted in anyone who wants to start up an effort to define a full
featured mail delivery/filtering system and would like to collaborate with
me.  Goals I have in mind, and tools I expect to use:

* Classification of spam/ham.  I expect to use ifile for this, though a
reusable system should provide a way to interchange/combine various tools
(bogofilter, spam assasin, etc.) for this.

* Filtering of mailing lists.  I've discussed this on this list in the
past.  I expect to read various (user selectable) headers, munge the
contents in some manner to produce a "unique key" and use this key in a
mechanism similar to ifile classifications to choose folders.  I can give
much more detail on this if needed.

* The above functionalities should be scripted (scripting language TBD),
and callable by such programs as procmail for initial mail delivery.  My
personal interest is only for delivery to Maildir, but a reusable system
would handle the other formats as well.

* Mis-delivered mail should simply need to be moved, by any means, to the
proper location, where a daemon process will relearn how the message
should have been delivered.  This way, you can use any mail client to move
the message, and the system will relearn how to filter and do so correctly
in the future.

The goal is to make it simple to classify, and re-classify/learn mail in a
non-obtrusive manner with the tools of choice for any given user.  I
really need something like this for myself, and am willing to dedicate
some time in developing it, and would like to know if others are
interested in assisting.

-- 
William E. Kempf