On Mar 26, 2012, at 03:20 PM, David Jeske wrote:

I'm writing to find out the state of and philosophy surrounding pipermail in mailman, to see if there is a productive way to provide some code/development-time to that part of mailman.

That's awesome. Pipermail (in its current state) is ancient, and it shows. It's biggest flaw IMO is the lack of "stable URL" support, meaning if you regenerate the archive from the underlying mbox file, you'll likely break all your links. Pipermail has lots of other problems, but this one for example is why it was ditched for Launchpad.

I think the time is ripe for new archivers, and Mailman 3's philosophy is to provide a framework in which developers can easily experiment with new archiver technology, integrated easily into Mailman.

I personally think it would be best to spend effort on one of the many archiver projects being worked on, and mentioned here, rather than trying to improve Pipermail. One possible connection though would be to consider how a site would migrate from a Pipermail-based archiver to one of the new ones. Do you keep the old URLs alive but just not add anything new to Pipermail? Do you provide a mapping from Pipermail URLs to new-archiver URLs? etc.

I've written code for this a number of times (eGroups, Yahoo Groups, Google Groups). I also released an open-source python/clearsilver/sqlite based archiver with redundant text-eliding, a few different thread views, and search... ( http://www.clearsilver.net/archive/ ) which is hardly used both because I don't try to popularize it, and because many sites just leave the default (pipermail).

Neat. Perhaps you'd like to contribute an implementation of the IArchiver interface in Mailman 3 that would send posted messages to ClearSilver?

Here's the current interface definition:

http://tinyurl.com/cup7amw

and some example implementations:

http://tinyurl.com/cynyfou

a) pipermail is fine... if you want to fix a bug or two submit a patch, but we don't want to improve it

Note that before I ripped Pipermail out of the Mailman 3 trunk branch, I created a project on Launchpad and pushed a semi-sanitized branch:

https://code.launchpad.net/pipermail

I don't personally plan to touch this, but if someone is really motivated to hack on Pipermail specifically, I'd happily give you access.

b) we're ditching pipermail entirely... in the future sites will have to choose an install an external archiver

Yes, but remember, you can choose one or more archivers in Mailman 3. So it would be easy to archive to something local, and The Mail Archive, and Gmane, etc.

c) we'd love pipermail to be improved... but we still want it to be simple, static-html, and dependency free d) we'd love a dynamic-ui replacement for pipermail... as long as it uses the same cgi/templating model as mailman ui

Because Postorius (the official, but not required MM3 web ui) is Django-based, I think it should be pretty flexible for customizing the look and feel. I won't dictate what a new archiver should look like, but some principles that I personally think should be followed include:

• Modern web technologies, with flexible templating, so that sites can customize the look and feel as needed.

• JavaScript not required: an archiver should be navigable with a text-based simple browser, but that also doesn't mean it has to be feature-equivalent if you *do* have JavaScript. I think "progressive enhancement" is the right term? I.e. make it awesome for today's web, but usable in older browsers, screen readers (for accessibility), etc.

• Dynamic generation of pages with caching. I'd love to see an enhanceable approach to the actual HTML generation. Let's say for example, that I suddenly want to recognize and hyperlink "bug numbers" so they point to my tracker. I should be able to drop in some extension to do that. Or maybe the spammers have cracked my email obfuscation algorithm. I should be able to drop in a replacement, invalidate the cache, and all my new page would automatically get the new obfuscation algorithm.

• Separate, dynamic support for take down notices. You posted your personal information and have asked the site's postmasters to take that down (either the full message or the personal information parts of it). It should be really simple for the site admins to do this, while possibly retaining the original message in a publicly inaccessible location for forensic purposes.

• Support for private archivers, so configurable authentication would be necessary.

• Merging of forums, archives, newsgroups, and IMAP.

• A REST API for querying information about the site, its lists, individual archived messages, metrics, etc. Maybe even some control (e.g. take downs) for users with the proper permissions.

• Stable URLs, RFC 5064 + X-Message-ID-Hash. See the above links. If you can implement the IArchiver.permalink() method and ensure that even if completely wiped and regenerated from the underlying raw messages, your URLs will remain stable, I think you will have won. :)

That's about all I can think of right now, but I'll say this: I think there's huge untapped value in a really great archiving framework. I have lots of ideas and it's something I'd like to work on eventually, once we get the core engine stable and released.

Also, as you work on archivers and try to integrate them with mm3, please do provide feedback on the IArchiver API and the integration implementation in particular. What's missing? What's wrong with the API?

Note too that atm, the lp:mailman bzr trunk is a bit ahead of 3.0b1 here. I had to disable some things at the last minute in the ArchiveRunner, and I've since fixed and pushed updates. So you probably want to at least take a look at the bzr branch.

Cheers, -Barry