Mailman 3 October 2003 - Mailman-Developers

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 23:01:14 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Wed, 2003-10-29 at 16:54, J C Lawrence wrote: >> Aye, picking the right interface abstractions is key. > Right on. I'm still debating if I can run down there on the 8th. I'd love to go to EuroQuest, but I also really need to be in Providence on the 7th, and back at work on the 9th. Aaaarrrgh. _IF_ I can make it we must go hit a pub with whiteboards in hand. Sorry for no earlier reply on this BTW, I'm in drowning eyeballs mode. > ...as well as implement a bulk mailer to eliminate the need for an > outgoing mail server. Eeeek! I trust this would be for immediate handoff to a "real" MTA versus handling final delivery directly? Quite the Pandora's box if not. >> Mailman v2.1 has a plugin layer for the membership roster. Its not a >> fully mature interface, but there are LDAP and SQL adaptors in the >> wild. > This interface was largely bolted on, so it's clumsy. Mailman 3 will > be defined by interfaces from the start. <nod> BTW Whatever happened to Michel Pelletier's interfaces PEP? I see the draft, and I see signs that something got done, but not what... >> Oh yeah, and just to keep Nigel Metheringham hopping: >> Mailman just doesn't have enough configuration options. > Heh. That's another issue. Last I heard Nigel was still running screaming into the hills. > I'm sure Mailman 3 will grow many more configuration options. The > trick is making them manageable (and mostly ignorable -- i.e. the > defaults Usually Work out of the box). <nod> > I've been experimenting with ideas for list styles which will make > list admins lives easier I think, without reducing the flexibility for > experts. Aye, that's something the Plone folk have been digging at with some success: a base library of waffle-stomp configuration patterns. I'm not sure for Mailman if we want just a picklist, or a very simple wizard. I suspect something more akin to the very brief Q&A wizard at Creative Commons for choosing a license type may be more effective and interesting than a picklist: http://creativecommons.org/license/ Very simple, very general, covers the basic cases, hides all the ugly stuff and picks sane defaults. It becomes even more interesting if site admins can tailor the configs for the basic cases. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 17:51:27 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Wed, 2003-10-29 at 13:30, J C Lawrence wrote: >> 2) Message IDs are not guaranteed globally unique, but the collision >> rate can be manageable/acceptable in a large number of deployment >> cases. > Ah, which reminds me, elaborating on my strawman, the answers to "when > should Mailman rewrite Message-ID on posts" should be: Never, Only to > resolve duplicates, Always. Does that mean that we keep a database of all Message-IDs that all lists on that host have ever seen? If so, what happens when a single message is CC'ed to multiple lists? NetNews servers require global uniqueness across all newsgroups. I'm rapidly coming to the conclusion that we have to rewrite all Message-IDs whenever the internal archive is enabled. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: Rewriting Message-ID (was Re: [Mailman-Developers] Requirements for a new archiver)
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 17:47:18 -0500 Barry Warsaw <barry(a)python.org> wrote: > In the spirit of RFC 2369 we define a new header called > List-Message-ID, and as in that standard, this field MUST only be > generated by a mailing list, not by end users. Nested lists SHOULD > remove the parent's List-Message-ID and supply its own. > List-Message-ID conforms to the same syntax as for Message-ID in RFC > 2822. Of course, for now read the header as if it had an X- prefix. > When an MLM receives a message, it generates a List-Message-ID header > which is guaranteed to be globally unique. A cooperating archiver > should use this header as its primary key, and must provide a > mechanism whereby the List-Message-ID can be presented and the > archived message can be returned. It may fall back to Message-ID when > there is no List-Message-ID header present. I haven't finished musing on this (busy day, thus slow on other replies as well), but my first thought: What happens when a given a message is sent to several lists on the same host? Does each list do its own munge? Do we do USENET-style crossposting? I want to do crossposting. I don't think we can due to per-list customisations. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 18:41:20 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 11:48 AM -0500 2003/10/29, J C Lawrence wrote: >>> You need a guaranteed unique id to be used as a primary index field. >> "Need" is a strong word. Its very deployment and use-case sensitive. > In the case of a database, it is a hard requirement. A primary index > field must be guaranteed unique. There is absolutely no way around > this issue. Right, and I'm not arguing that. My point is two fold: 1) Using Message ID as a primary key is attractive. 2) Message IDs are not guaranteed globally unique, but the collision rate can be manageable/acceptable in a large number of deployment cases. We don't have to guarantee key uniqueness for all messages BEFORE they are submitted to the message store. The unique property can be assumed from external sources (with all that implies) should the deployment case want that. There are tradeoffs here, and it is not clear to me that there is an instant and obvious global solution. >> "Need"? No. It is a deployment choice with easily understood >> ramifications. > Perhaps for the application, but this is a totally different ballgame > when it comes to a database. Google for "primary index field", and > hopefully you will understand. I'm neither an idiot or a neophyte in this game. Yes, a database needs a primary unique key. That's not in debate. The questions are: Do we know the key before submission to the store? (If we don't the store operation shouldn't be asynchronous) Is the risk of discarded messages due to key collisions acceptable? (Some deployment cases consider such losses acceptable, others can guarantee uniqueness without Mailman's involvement) Rotely assuming that Mailman must guarantee key uniqueness before we hit the message store is not a given, its a choice. Let's at least be on the same page. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Tue, 28 Oct 2003 12:49:51 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Tue, 2003-10-28 at 00:41, J C Lawrence wrote: >> If the URL is predictably based on the Message-ID this is not a >> problem. >> Quite, this is how/why NNTP uses Message-IDs are unique indexing >> qualifiers. > Yep, and we'd have to do the same thing (ensure that Message-IDs are > unique). Yup. Of course this heads directly into that beautiful debate of whether MLMs should rewrite Message IDs. Summarising briefly: If we rewrite all IDs we'll piss off the people who use ID to do dupe detection/deletion for courtesy copies. If we don't do some rewriting some messages won't make it through NNTP and some other people will be pissed off. Two contrasting approaches: 1) We guarantee uniqueness of all Message IDs. The only way to do this is to rewrite all IDs. This will piss off some people. 2) We best-effort guarantee uniqueness by only guaranteeing uniqueness within the last N messages to the list. This could be one by rewriting all IDs, in which case we might as well guarantee total uniqueness, or it could be done by keeping a DB of the last N (cf CDBD) and either discarding or rewriting detected collisions. This of course means that some messages will be discarded by NNTP and we won't know about it. Some may be willing to accept those risks. Premise: Mailman just doesn't have enough configuration options. My temptation would be to add the following configuration options to Mailman: Keep track of the Messages IDs we've seen for the last N messages? (0 disables, value sets N) How to handle messages with IDs we've seen before? (discard/rewrite radio button) Replace the Message IDs for all the messages we rebroadcast with our own unique values? (checkbox) > Note that we sometimes get shit from people who complain about > Mailman's NNTP posting code modifying Message-IDs to adhere to the > stricter NNTP requirements. Aye. It is very easy to criticise. It is a little less easy to define, implement and maintain solutions which can't be criticised. > But Mailman can't rely on the good graces of remote mail tools to > ensure globally unique Message-IDs, so it has to check and munge if it > gets a dup (or, it's within it's right to treat a dup /as/ a dup, > e.g. discarding it). Yeah, thus the above logic. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 16:20:10 -0500 John A Martin <jam(a)jamux.com> wrote: > Out of idle curiosity, why doesn't 'write once read many' indicate a > directory more than a database? 1) The filesystem is a database. 2) Unix filesystems have extremely limited meta-data. 3) A discussed format is putting the mesasges on the filesystem (as a BD), and the meta data in a different DB (primarily due to open(2)/stat(2) expense. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 16:40:53 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > by the way, this statement is in conflict with my previous statemenet > of "use cycbufs". I'm fully aware of that conflict, too. resolving it > will be one of the big challenges. cycbufs implement a filesystem-based heap with pool semantics. (There's a fair bit of literature on that space in the OS and application realm) As such they are specifically tuned for the case where the number of calls to malloc() are of a similar magnitude to the calls to free(). This makes sense in a netnews world where news articles expire regularly, and in general as much data is added to the spool as is removed from it. Does that model really apply to list archives? It doesn't for me. I may be unusual in this regard, but I generally consider list archives as one-way systems: messages go in and never come out. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

5 6

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 13:11:01 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 29, 2003, at 1:05 PM, David Birnbaum wrote: >> 2. third-party add-ons make it that much harder to install. If I >> have to set up a Mysql or Postgres database to use Mailman, it's a >> step that will put off people who don't already have it going. > actually, if you do it right, it's much easier -- because when you > build in those tools, you build in standardized interfaces that third > party add-ons can access, instead of the current case, which are code > hacks that break every time Barry burps at the CVS server... Aye, picking the right interface abstractions is key. There's also a disjoint between the novice SysAdm case who loves the fact of Mailman's all-in-one service, and the more meaty chap who integrates what he needs to. Much of Mailman's appeal at the low end is its all-in-one simple-to-install nature. (Well, ignoring thee GID FAQ...) Mailman v2.1 has a plugin layer for the membership roster. Its not a fully mature interface, but there are LDAP and SQL adaptors in the wild. At some point those adaptors will move into the Mailman core. If we move the archiving components (storage, presentation, index) behind plugin interfaces as well there's a reasonable opportunity for similar third parties to build adaptor layers which then also move into the Mailman core. Oh yeah, and just to keep Nigel Metheringham hopping: Mailman just doesn't have enough configuration options. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

4 14

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 09:15:35 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Thu, 2003-10-30 at 00:08, J C Lawrence wrote: >> Hang-on. Apache isn't the target. Mailman's UI is a CGI app. As >> such it works with any web server that supports CGI-bin, which pretty >> much means any web server with no exceptions. That's a pretty large >> gain, especially in the novice admin or simple deployment case >> territory. > Sure, but I suspect that plumbing Mailman out to http will be just a > proxy rule away from integrating with an existing web server. That's > not without its headaches too, but should be as widely supported. Considerably more web servers support CGI-bin than support proxy rules. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 07:04:19 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 12:40 AM -0500 2003/10/30, J C Lawrence wrote: >> I've already said my bits there and proposed what I see as the cheap, >> easy, incremental improvement course: Twisted's NNTP supports for >> storage, Message IDs for keys, a variant best-effort detection and >> rewriting policy for collisions, and a MeoWWW derivative for HTML >> presentation/posting. > I don't know anything about Twisted or MeoWWW, so I can't say how they > address the subjects above. Twisted is a pythonic library that implements most of the basic network protocols. Among other things it has an RFC conformant NNTP server and client implementations. Creating an NNTP server with a backing message store is, literally, three lines in Python. Of course it doesn't support all the nifties that real netnews servers do ala expires, administrative controls, feeds, etc. Its not intended for that market, and Mailman doesn't need those supports. If deployment sites need that, they're going to be using inn2|[BCD}News|Diablo anyway. MeoWWW is a (very inefficient but fixable) pythonic CGI which supports reading and posting to netnews via NNTP. It has various nice UI points, a decent feature set (more than we have now), and does The Right Thing in almost every aspect I've checked except for performance in the spool reads. > I can say that I'm not sure about an NNTP-based storage solution... We should really start out by splitting that discussion. NNTP is an access protocol. Netnews servers have various storage formats and techniques. Currently NNTP and IMAP are the only standardised wide-deployment protocols for message spool access. I'm not interested in IMAP for the reasons previously discussed. NNTP isn't great, but it is already supported by Mailman for the new gating features and adds a clean abstraction model which allows trivial replacement of Mailman's implementation by inn2|[BCD]news|Diablo|whatever should the deployment site wish. Additionally, again as a standards-etc based protocol, it allows clean abstraction for archive presentation: anything that talks NNTP can now be an effective Mailman archive presenter. Ditto for archive indexing. As a dev I'm interested in arguments about how to handle the store behind the NNTP interface -- I find that stuff fun and intriguing -- but also think they are fairly uninteresting right now for Mailman specifically. The 90% case for Mailman will have less than 200K messages in their site-wide spool, and most of those an order of magnitude less. For me the interesting point is that once we abstract the message storage behind a well-supported standards-based protocol we can incrementally improve our implementation and those really concerned with the larger cases can throw in inn2 or whatever else, like a filter to SQL, instead. ITMT we get the flexibility and time to grow and do it Really Right. Additionally, having adopted such a well defined abstraction model once, moving down the road should something else better appear it should be a comparatively small cost to support that in addition or instead. > ... although certain storage techniques we've recently discussed > borrow a lot from extant NNTP implementations, and I'm not sure how > much sense it would make to rip out just those parts we know we need, > or if we could actually reasonably take the whole thing, > kit-n-caboodle. Which may indeed happen. > I do believe that we need an alternative solution to the message-id > header as it was presented to us in the message, as a stable > guaranteed unique (well, as good as MD-5 or SHA-1 gets) message > identifier that can always be used to refer to the exact same message > no matter what. I'm in split minds here. I see the temptation. I like using Message-IDS, and they are a natural fit to the model semantically, but messing with Message-IDs has unpleasant effects for some other systems. <shrug> > Whether we use this message identifier as a replacement for the > message-id header value as it was presented to us -- I think that's a > more philosophical discussion, and I think we should address it by > allowing both options but deciding which would be a reasonable default > to take. <nod> I'm on the side of rewriting Message-IDs if we do generate our own keys. I don't like it, but it seems the cleanest approach. > Given that the mailman UI is basically completely contained within the > CGI, I'm inclined to leave it there and work on improving it > internally, allowing us to continue to work with most any webserver > the client may have. Agreed. > I don't know how MeoWWW addresses this issue, either by replacing the > webserver, or providing additional tools that may make it easier to > present a good and consistent UI. MeoWWW is a CGI as discussed above. Twisted implements both sides of HTTP in addition to the NNTP discussed above, but I haven't looked at the details. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0