[Ifile-discuss] Re: Maildir format

[William E. Kempf]

clemens fischer said:
> "William E. Kempf" <address@hidden>:
>
>> That said, I would think the Maildir mechanism should be useful for
>> manipulating files as well.
>>
>> 1) Read the mail message from $Maildir/cur/$msg and write
>> modifications to $Maildir/tmp/$msg.
>> 2) Remove $Maildir/cur/$msg.
>> 3) Hardlink $Maildir/tmp/$msg to $Maildir/cur/$msg.
>> 4) Remove $Maildir/tmp/$msg.
>
> when moving files (messages) between maildir/ subdirectories, you have
> to make sure that their names are unique.

True, but we're not talking about moving them here.  Messages are moved by
mail clients only (which better know how to do so safely), not by any
scripts.

>  it isn't customary to move
> files back to tmp/:  arriving files are collected there until they're
> safe, then renamed to new/.

I didn't suggest *moving* any messages back to tmp/, I suggested using
tmp/ for *writing* modified files to.  This ensures you don't start to
write over a file and have the write fail midway, destroying the message
entirely.

>  MUAs move them from there to cur/ once
> they've seen them.  it makes more sense to move files from the
> receiving maildir/ to another one, especially since they aren't the same
> messages any more!

No, that doesn't make sense.  I put the message in the Foo folder, because
that's where it belongs.  The fact that it's modified in order to get
ifile to relearn properly is irrelevant.  This is really no different from
the mbox scripts that started this whole discussion.  Logically, the
message is modified in the same mbox (I say logically because, for
technical reasons, the modification goes to a temporary mbox first, to be
copied over the original when done).

>  they will have ifile-user-headers added.

Modified, not added.  The header is added by procmail before being placed
in any folder at all.

>> Any failure should leave you with a valid file in either
>> $Maildir/cur/$msg or $Maildir/tmp/$msg (depending on when the
>> failure occurred).  No chance for data loss.  And by employing a
>> Hardlink instead of doing a file move and/or copy the process should
>> be more efficient with less chance for failure.
>
> it's wrong to assume anything about files unless one can track
> everything that happened to them.  i'd design the operations like
> this:  if a file arrives in new/ (or cur/, for that matter) the
> operation has successfully finished, if not, it has to be repeated or
> failed entirely.

What assumptions have I made?  Who's talking about mail "arriving"?  What
are you talking about?

>> * Filtering of mailing lists.  I've discussed this on this list in the
>> past.  I expect to read various (user selectable) headers, munge the
>> contents in some manner to produce a "unique key" and use this key in
>> a mechanism similar to ifile classifications to choose
>> folders.  I can give much more detail on this if needed.
>
> this interests me.  i have a system where new messages are filtered
> through bogofilter and an awk script that takes care of tagging
> mailinglists, HTML and so on.  the reason for this is that i need
> simple, standardized headers for the various backends i might hand
> messages over to.  currently it's gnus, but it has been procmail+
> mutt, procmail+gnus and i even installed maildrop for its superior
> qualities.  the mailinglists have been the biggest problem:  i want one
> simple header "X-Mailinglist: <name|posting-address>".  i can spot any
> mailinglist by its email-address appearing in either To: or Cc:, but
> there may also be other names and it may appear on continuation lines.
> there's an RFC mentioning which headers a mailinglist should have:

I'm aware of all of this, but the headers inspected should *not* be
limited by this RFC for many reasons.  First, not all mailing lists
properly obey the RFC.  Second, the same script should be able to handle
filing messages from my wife in a specific folder as well.

>> * The above functionalities should be scripted (scripting language
>> TBD), and callable by such programs as procmail for initial mail
>> delivery.  My personal interest is only for delivery to Maildir, but a
>> reusable system would handle the other formats as well.
>
> you could achieve this by careful factoring of tasks.  i currently have:
>
> 1.  SMTP receiver with simple rules (mailfront)
> 2.  pre-queue filtering for spam and reject/accept patterns
> 3.  the queue and local delivery system, which is a black box here 4.
> mail splitting into folders.
>
> i want spam-rejection before enqueing emails in the faint hope that
> spamware will remove my email from their target-lists on receiving 5xx
> SMTP codes, but it seems the majority of spampumps is just that:
> pumps.  it will deliver to dead addresses for years and never note.
>
> you can see that this system is ad-hoc.  the steps you'd have to
> refine is local delivery and splitting, because you'd have to provide
> for autoresponders, virus-scanners and whatever people may dream up. one
> of the most flexible systems i ever learned about is the qmail local
> delivery mechanism, which gives users many of the possibilities normaly
> only admins have.

Even just plain procmail will allow you to do most of what's needed.  The
problem is that it requires a lot of scripting up front, and generally the
scripting done is very specific instead of being generic.  I want a fairly
generic scripting system that lets me define a few parameters and forget
about the system, using nothing but my mail client from that point
forward.

> a good scripting language might be lua.org.  it can easily be embedded
> later into larger chunks of code writen in C.

Right now I'm using Python, because I know it.  I said it was TBD, because
if I get volunteers to work on this, we'll have to pick a language we can
all agree to.

>> * Mis-delivered mail should simply need to be moved, by any means, to
>> the proper location, where a daemon process will relearn how the
>> message should have been delivered.  This way, you can use any mail
>> client to move the message, and the system will relearn how to
>> filter and do so correctly in the future.
>
> then you should generalize this deamon to handle all ifiling as part of
> the delivery process.  this would save you from locking issues and
> tallying ifiles accuracy.

Locking should be the only real issue.  And to work with procmail, I'd
assume it has to already do the proper locking?

-- 
William E. Kempf