[Mailman-Developers] Re: Requirements for a new archiver

[David Champion]

> exhaustion.  Or you could store everything in an mbox file with a file
> offset index.  Or perhaps store everything to an nntp server (Twisted
> would make a nice platform for this <wink>).
> ...
> Also, I really want the next generation archiver to do everything
> through cgi (or equivalent programmatic interface).  The ability to
> massage the messages on the way out to me outweighs the benefits of
> vending messages directly from the file system.

Well, since you bring this up.... I've been giving this some thought
over the last few weeks, since this latest fit of discussions about
archivers cropped up. I've written up some code to address the problem
to my satisfaction, along with a quick draft manifesto to explain
myself. It's too long to inline here, but I put a copy on the web:

    http://home.uchicago.edu/~dgc/sw/mmimap/

Meanwhile, to cut to the chase: I decided IMAP is the way to handle
this, and I've implemented what I need to provide it for both public and
private lists. There are scripts to extract authentication material from
Mailman, and an IMAP proxy daemon that performs authentication and sets
up an environment to hand off to UW-IMAP.

I've tested on our production server with a restricted set of users.
No complaints, and all the testers approve of the approach. Our server
needs an upgrade before it's powerful enough to do IMAP for 2000 lists
(67,000 subscribers), but it's tentatively the way we plan to go. We
probably won't enable HTML archival after the upgrade. We already have
a webmail product in place, but if we didn't we could just plug that in
on the list server to provide the HTTP access.

I realize that IMAP isn't ideal for all sites or lists, but I think
it should work well for our purposes, where lists are mostly
institutional, and not so public that they need to be Googled.

I'm hoping to get these materials better integrated and documented soon,
maybe once I'm back from LISA. But in case anyone is interested in
working with them, I've put them up on the web, linked from the above
URL. If this were to be a standard solution rather than a local hack,
it would probably need some refactoring for other IMAP daemons, for
newer MM authenticators, etc. I'm sure I haven't done the best as can be
done, and I'd certainly rather see IMAP access to archives be a standard
component of (or interface to) list server software, but it's a pleasing
start.

-- 
 -D.    dgc at uchicago.edu
 University of Chicago > NSIT > VDN > ENSS > ENSA > You are here
 .  .  .  .  .  .  .
 always line up dots