[Mailman-Developers] MailMan modifications to support WebDav & Attachments

[Bill Bumgarner]

In short, it works and I'm a happy coder... been 2.5 years since I have  
written any python and it is a joy to dive back in!  If anyone wants the code,  
let me know and I'll drop a tarball on you-- I'm going to be tweaking it over  
the next few days.  This was developed for a client project, but we [CodeFab]  
convinced them that (a) using MailMan would give us a huge leg up and (b)  
donating the changes back to the community would be the best possible route to  
take....

As such, the implementation was geared towards solving a very specific set of  
requirements for the client.   However, the implementation was done in a  
relative black box-- the implementation was very much done against the question  
"How would one modify MailMan to intelligently support attachments archived to  
web accessible servers?" so as to try and maximize the reusability and value  
to the community [while minimizing the exposure of any proprietary requirements  
or technologies of the client].

Specifically:  MailMan now has the full set of modifications necessary to  
allow a user to post a message with attachments to a MailMan controlled mailing  
list, MailMan will detect the attachments that should be deocded, decode said  
attachments, rewrite the message to indicate that various attachments have been  
decoded, file the decoded message and attachments to a WebDAV server, and send  
the rewritten/urlified resulting message out to  
Usenet/mailing-list/digest/HTTP.

The end result is not pretty-- it works, it is relatively well architected,  
but there is quite a bit of room for optimization/improvement.

The one feature that is truly lacking is that the DAV'd archives are not  
nearly as friendly in structure as are the PiperMail archives.....

Generic feature list:

- ability to decode binary attachments from incoming messages;  configuration  
options to determine which major types should be decoded, where they should be  
filed, how they should be accessed, etc,etc,etc....

- ability to file attachments [via WebDAV] into a repository accessible to the  
end user via HTTP

- ability to rewrite messages to reflect availability of attachments via HTTP

- use of an open protocol [WebDAV] to file/manage attachments

- ability to encapsulate message into an HTTP request made against an  
arbitrary web server / web application

The primary goal of implementation was to render a working solution as quickly  
as possible.  However, a secondary goal of great importance was to render a  
solution that would be palatable to the MailMan / Open Source development  
community and, as such, would be picked up and maintained by them ASAP--  
licensing requires that this piece of the In short, it works!

Specifically:  MailMan now has the full set of modifications necessary to  
allow a user to post a message with attachments to a MailMan controlled mailing  
list, MailMan will detect the attachments that should be deocded, decode said  
attachments, rewrite the message to indicate that various attachments have been  
decoded, file the decoded message and attachments to a WebDAV server, and send  
the rewritten/urlified resulting message out to  
Usenet/mailing-list/digest/HTTP.

The end result is not pretty-- it works, it is relatively well architected,  
but there is quite a bit of room for optimization/improvement.   In particular,  
the archive doesn't have near the friendliness of a pipermail archive-- but  
that shouldn't be *that* hard to add considering how it was originally  
implemented.

---- Summary of implementation ----

Components:

	- apache web server [version 1.3.9 -- latest stable release]

	- mod_dav WebDAV module from Greg Stein  
[http://www.webdav.org/mod_dav/ -- using latest development version from Greg's  
CVS repository.  Can likely move BACK to last stable release-- though a new  
stable release is pending shortly]

	- python [1.5.2 -- latest stable release]

	- pyexpat module [ftp://ftp.cwi.nl/pub/jack/python/ -- version 1.3 --  
includes expat library] -- provides xml parsing toolkit used by both mod_dav  
[expat] and qp_xml/davlib [pyexpat]

	- MailMan [using latest cut from MailMan development cvs server.   
Version 1.2expiremental.  Very close to 1.2 release, so it is stable.  As the  
MailMan community is likely to suck in all the changes that I have made, this  
will be path of least resistance.]

	- qp_xml / davlib / httplib -- various python modules from Greg  
Stein's CVS server.  davlib had to be modified (see below)

	- mimecntl -- [version 1.3 -- Michael P Reilly --  
http://starship.python.net/~arcege/modules/] Slightly modified to deal with  
file handles more cleanly.

	- sendmail -- generic, free, totally unmodified sendmail.  Latest  
version.  Basic configuration;  no spam filtering, but configured to not allow  
relaying.  Configuration will change slightly once in production.

Assembly:

APACHE / mod_dav

	Apache is configured with mod_dav installed and enabled as a  
dynamically loadable module.   Configuration is quite straightforward; the  
directory identified as the root of all WebDAV MailMan operations has DAV  
enabled and requires password authentication to perform any operation.

PYTHON

	Straightforward python 1.5.2 installation.  Dynamic loading of modules  
is enabled [required for loading of expat module]

PYEXPAT

	Straightforward dynamic loading build of pyexpat.   For production, we  
might consider moving to a statically linked version of pyexpat-- this will be  
both more effecient and less prone to failure during upgrades/such-- but will  
require slightly more maintenance during upgrades.

SENDMAIL

	No assembly required.   Configuration details will be documented  
elsewhere [no changes to document now-- it'll all be performance/feature tweaks  
reflective of the production environment].

MAILMAN

	The internal architecture of MailMan was relatively consistent. As  
well, it is clear that the original architects gave a great deal of thought as  
to how to create a very flexible and extendable body of code.   This proved to  
be a huge asset-- while the features required were relatively alien to the  
existing body of code, it was fairly clear how to add functionality while  
following the existing architecture.   The end result is that the architecture  
of MailMan itself is largely untouched.

	The integration of certain required features, however, was not so  
clean or straight forward.  The following specific implementation requirements  
caused a bit of frustration:

	- dynamically manipulatign MIME messages;  it appears that very few  
people have ever given thought to the value of being able to rewrite a MIME  
message by replacing various parts of a message with pieces encoded  
differently.   Thankfully, Michael Reilly had already thought through this  
problem-- the mimecntl.py package proved to be a huge timesaver.

	- WebDAV:  DAV is an immature technology and, as such, the packages  
available to the developer for manipulating DAV accessible repositories tend to  
be extremely primitive.    Largely, this is due to the extremely  
primitive/immature state of the XML marketplace combined with the inherently  
complex nature of a typical XML document.   Similarly, DAV requires a client  
and server that implements HTTP/1.1 -- a version of the protocol that is  
relatively new.   Greg Stein's qp_xml (in conjunction with pyexpat), davlib,  
and httplib saved a huge amount of time.  However, it still required a lot of  
work to generalize access/control of a webdav server from MailMan.

	Synopsis of changes/additions::

	- added a number of configuration options to various different admin  
pages throughout MailMan.   Allow for setting target DAV server, advertised  
Dav'd resource retrieval, whether or not attachments are decoded, which types  
to decode, disabling pre-existing archival functionality, where MailMan->HTTP  
gateway'd messages should be sent, etc....

	- add two additional handlers to the message handling pipeline (see  
HandlerAPI.py in Handlers in MailMan);  Rewrite -- rewrites a mime message,  
decoding attaachments as needed and ToHTTP -- encapsulates a message and sends  
it to the configured web server

	- modified existing post script such that it will instantiate a  
MimeMessage when a particular list is configured to do so.   MimeMessage is a  
new subclass of Message that encapsulates an instance of MIME_recoder from  
mimecntl package.  MIME_recoder allows for random access and recomposition of  
MIME based multipart messages.

	- modified ToArchive handler to be aware of DAV.   Added functionality  
for enumerated messages received by a list and archiving each message into a  
DAV enabled web server.  ToArchive will also automatically validate and create  
any collections [directories] needed on archive server to service a given list  
or message.

	- added a custom subclass of davlib called davxmllib which optimally  
performs various common DAV operations through the use of well defined XML  
structured DAV requests.   davxmllib also presents an error/exception interface  
that is much more desirable to high-level manipulation of a DAV based resource  
server

	- added ability to turn off pipermail [legacy archival system  
previously used in MailMan]