Mailman 3 October 2003 - Mailman-Developers

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 29 Oct '03

29 Oct '03

On Wed, 29 Oct 2003 16:49:14 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 2:30 PM -0600 2003/10/28, David Champion wrote: > You cannot assume a homogenous client mix. Moreover, you can't assume > broad support for less common protocols like IMAP or NNTP. Apparently assumable/desirable protocols enclude: SMTP HTTP XML/RPC (on HTTP) SOAP (on HTTP or SMTP) Mailman currently supports the first two, with the caveat that it has no SMTP retrieval supports and no a priori primary key for HTTP. Moving to a store which supports a Message ID primary key doesn't change the protocol list (tho it may extend it). However moving to such a store allows a simple extension to allow key-based retrieval via SMTP and HTTP, which are the primary protocols. Extending that down the road to XML/RPC and SOAP on any transport wouldn't be difficult, especially in the Python world (I'm beating on SOAPpy's MIME supports as I type this). What form the backing store takes is orthogonal to this aspect of the discussion. The key features are a priori key definition and key-based retrieval. Get those two and the rest become relatively trivial. The exact form of the backing store is irrelevant. Nobody cares. Protocol access and protocol behaviour (API) are the important bits. To date three backing stores have been proposed: Twisted's NNTP implementation, IMAP, and SQL. All could work. All could match the above discussion perfectly. Implementing each would require significantly different levels of effort. I'd posit that Twisted is the cheapest/easiest route simply due to it being pythonic and small. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 29 Oct '03

29 Oct '03

On Wed, 29 Oct 2003 16:31:53 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 12:53 AM -0500 2003/10/28, J C Lawrence wrote: >> For me that came out to inn2, Message-IDs, We:Search, and a mix of a >> MeoWWW derivative (HTTP) and NNTP straight to inn2. The critical >> point is that each bit of that amalgamation is independent and can be >> trivially swapped out for something else that suits a local use case >> better. > One of the hallmarks of mailman is that it provides almost all the > features anyone would be likely to want, out-of-the-box. Maybe > they're not the 100% ideal solution, but they're usually more than > sufficient. You get a decent archive solution provided "for free", as > part of the base package. You're making my point for me again. By picking up the abstraction model I've suggested and tucking Twisted and MeoWWW under Mailman we do get the 80% solution for bundled with Mailman. But we also get an easy abstraction model where any one of the pieces can be trivially replaced by something else, like the inn2 I mention. > If you throw that out and go with a modular approach based on any > number of external third-party add-in tools, then you might as well go > back to Majordomo. Which hasn't been proposed, even slightly. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 29 Oct '03

29 Oct '03

On Tue, 28 Oct 2003 00:44:43 -0500 J C Lawrence <J> wrote: > On Mon, 27 Oct 2003 19:16:35 -0500 Barry Warsaw <barry(a)python.org> > wrote: >> Which does point to an alternative direction -- maybe we don't need >> any direct connection to an html archive. Maybe the archiver should >> just be a separate process that reads messages from the NNTP >> interface a MM3 might export. Just blue-skying here. > +1 More simply I've come to want a clean abstraction in the work cases: storage, indexing, presentation. The three are different, largely unrelated, and I see little reason (non in my case) to even have them done by related or similar tools. As such the questions come out to" What's an efficient way to store messages? What's an efficient way to retrieve messages by a primary index? What's an efficient way to build other indexes to that primary key? How do I want to present the data? For me that came out to inn2, Message-IDs, We:Search, and a mix of a MeoWWW derivative (HTTP) and NNTP straight to inn2. The critical point is that each bit of that amalgamation is independent and can be trivially swapped out for something else that suits a local use case better. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

4 5

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence 29 Oct '03

29 Oct '03

On Mon, 27 Oct 2003 17:28:50 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Mon, 2003-10-27 at 17:08, PieterB wrote: > When Mailman decorates a message for copying to the list, I want to be > able to include a link to the archived message in the footer. The > problem is that there is little or no connection between the process > doing the decoration and the process doing the archiving, and in fact > the message may be posted to the list long before the archiver gets a > crack at it. If the URL is predictably based on the Message-ID this is not a problem. > So I don't want to have to ask the archiver for that url. I want > Mailman to be able to calculate it from something unique in the > message, and have the archiver agree on the algorithm, so that it (or > some other translation layer) can do the mapping back to the archived > article. Or, Mailman should be able to calculate a unique id for the > article and stuff that in a header for the archiver to index on. Quite, this is how/why NNTP uses Message-IDs are unique indexing qualifiers. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

How do I got the email addresses?
by Markus.Zuelch＠materna.de 29 Oct '03

29 Oct '03

Hallo, I need a way to store (automaticaly, on the fly) the incomming new Subscriber email-address into an Oracle Database. Also I need a way to tell Mailman (Version 2.1.3) not to collect the email-adresses from its own database, but rather from the Oracle Database BEFORE Mailman sends the email to the subscribed recipients. Can anyone tell me, which modules are responsible for handling the Subscriber and recipients email-addresses? Thank you very much Markus

1 0

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 28 Oct '03

28 Oct '03

On Tue, 28 Oct 2003 16:46:02 -0500 J C Lawrence <J> wrote: > While I see your concern for spending a whole lot of time investing > and specialising in a netnews core for Mailman, I think the fear is > displaced. Mailman already bidirectionally gates to netnews. The > changes required would be comparatively small for a significant gain > in feature-set for the average non-geek case (better web archives, > posting from web archives, more flexible message store, ability to > have archive URL placed in broadcast messages, etc). I wrote this badly. I'm looking at this in a rather abstract light, and haven't really stated that fact. From here this is all a question of abstraction models. Currently Mailman doesn't have a clean or well defined abstraction model among the list processor, the archives, the message sort, the web presentation layers, or search engine supports. Gaining a clean and well defined abstraction layer that does the Right Things would do a whole bunch of useful things, like allow for far easier and cleaner plugin-style approaches to any of the core components: Message store Search support Web archives. Message access. Right now those four are a somewhat incestuous mess and pulling them apart into a clean model is a bitch for an external integrator. Harking back to the earlier process queues model for Mailman 3.0, the same abstraction and division gains happen again. We can implement (say) Twisted as a trivial message store. Really what we're doing however is stating that Mailman can use any store you want as long as messages can be sent to it via process pipe, SMTP, or NNTP. We can implement a CGI-based web interface to that NNTP store. Really what we're saying however is that Mailman will provide a web-based interface for archive browsing and message posting to its own default message store, or any NNTP-based message store you implement. We can implement a plugin layer for external search engines as we no know (control) the unique primary retrieval key for messages in the default store (and thus the default URL etc). Or, if you wish, what we're really saying is that Mailman's archives can be integrated with any external search engine that can be handed the retrieval key and can index the message appropriately (via web scrape, custom store access, NNTP, whatever). Sure, we can grow up and feature-fluff the search side later. I'm a little less concerned with that at this point. I'm more concerned with gaining a clean abstraction model among the store/retrieve/present components of the archives. Sure, we could also decompose the messages into SQL structures. Having done this, I'll briefly note that it is not a trivial task on several scores (MIME, data partitioning, etc). This isn't to say that its horrible, merely not trivial. Picking up Twisted's store is about as close as you can get to trivial. You don't get all the MIME decomposition etc that you might get with an SQL store, but having adopted the abstraction model it is then much simpler for someone to write an SQL-based message store that Mailman can talk to cleanly In an OpenSource world smaller shorter feedback loops based on simpler iterative steps are almost always the better choice. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 28 Oct '03

28 Oct '03

On Tue, 28 Oct 2003 12:48:46 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 28, 2003, at 12:30 PM, David Champion wrote: >> message-delivery protocols, and tools that know about messages; why >> keep trying to reinvent them over HTTP? > because once you leave the niche of dealing with your fellow geeks, > that's what users are going to want. browser access. NNTP is simply a > non-issue any more, and iMap is fine, but they know how to go to a > URL, don't assume they can reconfigure their mailer. Quite. > Not saying don't do this, but if you write geek tools for geeks, > you'll lose the rest of your audience, the non-technical users. This is where the Twisted+MeoWWW approach seems attractive. If we go for Twisted+MeoWWW we get a flexible message store with a CGI-based web front end which also allows posting via web. The fact that it is netnews based can be, and should be, utterly transparent to the casual user. There's no need in fact for the NNTP interface to be exposed to arbitrary connections. However, should someone wish to setup news access to their lists or archives, that's as trivial as telling Mailman to open the port. If they should wish to use their already configured inn2 or whatever, that's as trivial as telling Mailman to deliver to inn2 instead of the Twisted Netnews server. But in the average case they don't have to, and don't have to care. Mailman just uses the netnews base of Twisted as a message store with a known-ion-advance primary retrieval key. >> Nobody needs web access: what they need is access via a web >> browser. With browsers that understand NNTP and IMAP prevalent, and >> with a wide selection of web-mail and web-news gateways for the cases >> where that doesn't work, this is sufficient. > is it? it seems to me to (frankly) be a real hack with bad navigation, > at least the stuff I've seen. I'd be happy to be proven wrong. That's my interpretation as well. It serves some very narrow cases well, but not the general case. > And for intermittent or one-time access to an archive? won't > bother. And how does it get into google so they know to look at it in > the first place? <nod> That's where I like the MeoWWW approach. Its just another CGI, so it auto-installs as part of the Mailman CGIs without requiring specific SysAdm configuration. As it renders to HTML/HTTP Google will index it happily. If the list/group is configured to allow posting, then MeoWWW will happily provide a web-based way for arbitrary users coming into your archives via (say) Google search to post and respond to items in a list's archives. Of course standard poster/spam controls would/could be applied. While I see your concern for spending a whole lot of time investing and specialising in a netnews core for Mailman, I think the fear is displaced. Mailman already bidirectionally gates to netnews. The changes required would be comparatively small for a significant gain in feature-set for the average non-geek case (better web archives, posting from web archives, more flexible message store, ability to have archive URL placed in broadcast messages, etc). -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Re: Requirements for a new archiver
by J C Lawrence 28 Oct '03

28 Oct '03

On Tue, 28 Oct 2003 14:30:36 -0600 David Champion <dgc(a)uchicago.edu> wrote: > With browsers that understand NNTP and IMAP prevalent, and with a wide > selection of web-mail and web-news gateways for the cases where that > doesn't work, this is sufficient. A minor problem with this is that news: URLs can't specify a server. However there is no UR standard for IMAP folders. > I favor IMAP over NNTP for this: ... > But integrating with both is even better. Setting up either, or in fact any other form of store (eg SQL) is trivial, and can be as simple as a procmail recpipe or other filter hung off a process pipe. There's nothing unique or special about netnews servers or IMAP or POP or SQL, or whatever in this regard. What is special boils down to two points: 1) What is the primary retrieval key for the messages in the archive, and can Mailman know what the key for a given message is before submitting it to the store? 2) How can the messages in the store be otherwise indexed (eg full text search). #1 is the kicker. #2 is easily abstracted into any method or tool you want. IMAP is a message store with a single store-specific primary retrieval key. There is no standard method for knowing what that key is prior to inserting the message. NNTP-backed systems are also a message store, except that they support two primary retrieval keys, one per-store specific (message number) and one per-message specific (Message-ID). The Message-ID is of course known prior to insertion of a message into the store. Both these described characteristics are constant across all RFC-conforming netnews and IMAP systems. Some IMAP systems support in-band searching (cf Cyrus). This can't be relied on. There are however dozens if not scores of indexing systems which will index netnews spools, mail folders, or even HTML representations of new news spools, mail folders, etc. There's no reason to not leverage that wealth of capability. Search isn't and arguably shouldn't be Mailman's space. We can do something here, but it is not a core market or skill for the product. If Mailman had two things we'd seem to be 90% of the way there: 1) A default message store which also had a very trivial search capability. 2) The ability to pass arguments to a process stating the unique primary key of the message Mailman just submitted to the store (default or otherwise) so that the "search engine" could then index it. Twisted can provide a simple default message store based on netnews. Those interested can trivially use the current gating support to use other netnews systems ala inn2, CNews, etc instead should they wish. or nothing at all. MeoWWW provides a reasonably well featured NNTP-based newsgroup browser with full posting-via-web support. As a GPL pythonic CGI it is relatively trivial to incorporate it in Mailman. Search is a bit more of a mess. I'm not aware of any pythonic trivially integrated search tools ripe for the plucking. I like We:Search as it is fast, braindeadly simple, and has almost no dependencies. It is however also optimised for extremely large mail stores, which isn't Mailman's target, but that also doesn't hurt it. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence 28 Oct '03

28 Oct '03

On Tue, 28 Oct 2003 12:42:21 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Tue, 2003-10-28 at 00:38, J C Lawrence wrote: >> It would take little to bolt the NNTP supports from Twisted into >> Mailman and then grab something like MeoWWW for the archive >> presentation. Its quite useful as-is. > I'm google-fu is low today. I'm not finding any usable links to > MeoWWW. I found some reference to it on newsreaders.com, but both > links were dead. Yeah, I remember it being hard to find, but it is out there. The following holds a copy of what I have: ftp://ftp.kanga.nu/pub/users/claw/odd/meowww.tar.gz > I think you're reading my mind. :) Heck, I may yet cash in that beer I owe you. Eurogames is coming up in early November, and I might just be persuaded to head down for the Sunday... http://euroquest.gamesclubofmd.org/ -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence 28 Oct '03

28 Oct '03

On Mon, 27 Oct 2003 15:12:29 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Mon, 2003-10-27 at 15:06, Kevin McCann wrote: > There are a couple of ways I could see slicing things. You could > store one message per file a la MH, with some elaboration to avoid > inode exhaustion. Or you could store everything in an mbox file with > a file offset index. Or perhaps store everything to an nntp server > (Twisted would make a nice platform for this <wink>). It would take little to bolt the NNTP supports from Twisted into Mailman and then grab something like MeoWWW for the archive presentation. Its quite useful as-is. > What would then be in the database would be records providing easy > lookup by message-id (at least) into the on-disk message store. Bingo, plus the option of opening the NNTP interface to external users as another list reading/posting method. Now Mailman can become an all-in-one lightweight archiving news and mailing list system for precious little expense. > Also, I really want the next generation archiver to do everything > through cgi (or equivalent programmatic interface). The ability to > massage the messages on the way out to me outweighs the benefits of > vending messages directly from the file system. Agreed. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1