[Mailman-Developers] Braindump on new archiver

[Thomas Wouters]

I've been thinking about re-doing the whole archiver thing in Mailman for a
while now, and I need to start writing these things down before I forget
about them. What follows is basically a braindump, for your amusement and
free to comment on.

First off, I have a couple of handicaps. Some are severe. The first one is
that I have very little time, which is one of the reasons I'm writing this
braindump: I don't have time to implement it right now :P The other is that,
for a living, I build and maintain large to very large, high-performance
internet servers (mostly Apache webservers with bells 'n whistles.) This
means I am usually too performance oriented. And lastly: I like the
'Maildir' mailbox format very much. This is a handicap because,
unfortunately, not all mail programs support it. 'Pine', specifically, is
boycotting the Maildir format (don't ask me why, I gave up on them.)

The good thing about the Maildir format is that it's very NFS secure. All of
our customers (except a handful who refuse to upgrade from 'elm' to 'mutt')
have a maildir mailbox, and we have 6 SMTP delivery boxes, 5 POP retrieval
boxes, one IMAP retrieval box (used only for our webmail server, which is
why it's only one right now) and several shell servers that have all kinds
of UNIX mail clients running. The mailspool itself is a dedicated NFS server
(a NetApp Filer.) We have a patched version of Pine that works with maildir
whether it likes it or not :)

Maildir works by making every message a file on its own. A mailbox is a
directory with three subdirectories, 'cur', 'new' and 'tmp'. Messages in
'new' are unread, messages in 'cur' are read, and messages in 'tmp' are in
transit. Message-metadata and state is maintained using the name of the
message-file itself. Delivery is done by creating a new file in
mailbox/tmp/, writing all data to it, and when it's done, moving it to
'new'. Writing data to a file is not trustworthy, over NFS, but renaming and
moving files are.

What I'm thinking for a Mailman archiver (or just any email-webarchiver, in
fact) is something like the following:

- Mail delivery can happen on any host. The Mailman process, after doing its
normal pre-delivery stuff, 'delivers' the message by dropping it in a
Maildir folder (rather than appending it to an 'mbox' mailbox) _and_
cgi-escaping/header-cleaning the message and dropping it in a
monthly/weekly/daily archive directory. The names are generated from uniq
data, like the time of day, the machine name and the process-id, possibly
hashed for obscurity. No locking is necessary. The URL to that _specific_
message is known as soon as Mailman calculates that name, and so it can be
inserted into the message that will be sent out to the list. (A common
feature request :)

- Non-text attachements could be saved as separate files, and replaced by
URLs to them in the archives. This should be possible to do both based on
type and on size, of course.

- The web archives are indexed once every [X] minutes, by looking at all the
new messages and processing them. Processing them consists of creating
indexes based on threads, subjects, author, etc, exactly like now. State
data could be stored in a .db (berkeley db, marshall, pickle, whatever) and
need not even be protected by a lock, if done properly: When started, the
indexing process should compare the mtime of the .db file with the current
time, as well as the last added new message. If the .db file is too new, it
should not run. If the .db file is old, and the newest message is newer, it
should touch the .db file (leaving it exactly as it is, just updating mtime)
and start working, keeping in mind to touch the file every now and then. The
new .db file should be written to a temporary name, and rename()'d when it's
done.

- The individual archive message URL's would actually call a CGI script that
looks up state data from the .db file, grab the cgi-escaped file, prepend
and append standard header and footer containing 'next in thread', 'next in
subject' and the like generated from the .db file, and feed that to the
browser. If state data is absent, but the message itself is there, teh
standard header and footer are added with a note saying that the message is
too new to have links to other messages -- but at least the message will be
visible.

Mail delivery and the Web-interface can be run on multitudes of servers, all
sharing data over NFS, and still be able to communicate in a sane way,
without waiting for eachother's locks forever. The only thing that can go
wrong is for a message to be archived and sent out, and then read and looked
up in the archives before the actual NFS data (the content of the message)
is visible on the web-interface machine(s). Given the average speed of SMTP,
I do not consider this a serious problem :)

One thing should run on a single server, however, and that's the
archive-indexing process. Given that news-feeding and mailing out passwords
should also be done on a single server, I don't see that as a problem either.
And all of the above should work just as fine on a single machine doing
everything on a local disk.

While under the 'the filesystem as the database' spell (hell, I'm still
under that one) I also considered using the filesystem to index messages
based on thread, subject, author, etc: a directory for each category, in
which symlinks point to messages. To go from message 'X' to the next message
in the thread, check where the symlink 'next-threads/X' points. The same for
previous-thread, next-author, etc. I'm not sure if it will managable without
adding a lockfile, nor how efficient it would be to generate overviews from
such data.

Another thing for consideration is the 'downloadable mailbox' link. Should
it download the maildir mailbox, or generate an mbox one on demand ? The
maildir mailbox is *very* useful for incremental updates, since all that
takes is checking which files you've already read. But, like I said, it's
not as widely supported as the crappy ol' mbox format. 

End-of-rant--we-will-now-return-to-your-regular-scheduled-static-ly y'rs,
 ;)
-- 
Thomas Wouters <thomas@xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!