Mailman 3 March 2012 - Mailman-Developers

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Barry Warsaw March 28, 2012

March 28, 2012

On Mar 27, 2012, at 08:53 AM, David Jeske wrote: >Storm-ORM looks like it allows any primary key so that's good. From a quick >glance it looks very similar to the Clearsilver-odb-orm in syntax and >function. My primary beef with django-orm (last time I looked at it) was it >requiring a unique-id primary key which is pretty poor for multi-row fetch >performance design. Here's my take on ORMs. Early on mm3 went through phases with both SQLObject and SQLAlchemy, which I'd say are the other two most popular Python ORMs that aren't specifically tied to a CMS. It's been a few years now, but both had lots of problems that manifested as unacceptably slow velocity for making changes to Mailman. What I like about Storm is that it's so lightweight. When something goes wrong (e.g. not being able to store flufl.enum.Enums in the database or UTC timezone-aware datetimes into SQLite), I've always found it pretty trivial to dig into Storm, find the problem, maybe fix it, or at least easily work around it. The well-written, thin-ish layer of Python code makes up for the lack of comprehensive documentation. UTSL. Storm is also heavily used within Canonical so I know who to beg with or scream at if I need something :). Seriously though, it powers Launchpad among other tech, so it's pretty well battle-tested. But I'm the first to admit I'm not a database expert, so while I can make stuff work in Storm, it may not be optimal. Two examples: I've gotten donations to make some of the queries more efficient and I'll always welcome those types of contributions. Second, I know that some of the constraints I have expressed in the SQLite schema can't be expressed in a more strict database such as PostgreSQL. It would be nice for someone to dig in and fix those. The other thing to keep in mind is that within the core, I've tried very hard to again, use formal interfaces between components so that it could almost not matter. This hasn't been field tested so maybe I failed, but the idea is that you could potentially rip out the model and database layers and provide alternative implementations of IDatabase and other interfaces, and the rest of Mailman should Just Work. I try to be very careful not to break those abstractions, and I think the Zope Component Architecture is the big win here. >Clearsilver templating has a slightly more aggressive enforcement of 'no >code in template' than django, because it has an explicit intermediate >dataset called HDF (it doesn't allow access to arbitrary python lists, >structs, classes, and methods). This allows it to work from any language >(not just python), and makes the binding from the ORM a bit more explicit. >However, it also means it's a c-module, and I can see the advantage to >keeping c-compile dependencies to a minimum. While I want to keep the Mailman core as pure Python, and so far have seen no reason to think it can't be, we already have some C dependencies in our stack. Some of the libraries that Mailman depends on have C extensions that are either required or improve performance. This already means that to deploy a mm3 server you currently need a C compiler (and the "development" bit of Python, e.g. Python.h). Now, eventually you'll probably get this all from your distro, so won't have to worry about it, but that's the state of affairs atm. >After I finish some quick cleanup and UI changes to CSLA I want to do, I'll >take a look at MM2/3 and see if I have better questions once I'm informed. mm2 and mm3 will differ greatly in the parts you're going to care about. My suggestion is to forget about mm2 and concentrate on mm3. Cheers, -Barry

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Barry Warsaw March 28, 2012

March 28, 2012

On Mar 27, 2012, at 09:32 PM, Stephen J. Turnbull wrote: >Well, for the official poop you'll have to wait for Barry, but AFAICS >archivers aren't restricted to Storm + RESTish (which is what Mailman itself >uses) because they're separate applications. If the archiver/web UI is going >to be distributed *with* Mailman, Barry would probably prefer Storm + Django >because that's what Mailman/Protorius (core and admin web UI, resp.) are >using. But I imagine that's negotiable as long as everything is free >software. > >I don't think the database backend much matters, but Mailman and Django >are both happy to use sqlite, I believe. What Stephen says above is pretty accurate. See also my previous follow ups in this thread. The core engine uses Storm as its ORM, which I think officially supports SQLite, PostgreSQL, and MySQL, though it's probably compatible with others out of the box or easily so. Mailman itself currently only supports SQLite out of the box because that comes with Python, and we have donated support for PostgreSQL. I'd love for someone to donate MySQL support; I think it would be easy for someone with MySQL experience, there isn't much that would need to be added to Mailman, there are good examples now, and I would of course be happy to help. The biggest problem with multiple database support is ensuring we're getting good testing with them. I do almost all my tests with SQLite and we don't have a buildbot/jenkins farm yet to test all combinations (and Python 2.6/2.7 :). >> I'm not a fan of javascript client-side rendering because of the generally >> poor performance, poor mobile compatibility, and lack of benefit for this >> kind of application. > >Not my pidgin, you'll need to talk to Toshio/Barry about that. >>> AIUI, the current philosophy is that As a third party archiver, I can't tell you what to do, but my personal preference would be that JS is fine if it enhances the user experience, but that it be possible to perform basic archiver interactions without it. Cheers, -Barry

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Terri Oda March 28, 2012

March 28, 2012

On 03/26/2012 11:37 PM, David Jeske wrote: > CSLA doesn't currently have any concept server-auth. The only stateful > features it has are view-preferences and read-state, neither of which are > important enough to require a password. It uses a password-less system > which uses cookies for prefs and a 'read state userid' which a user can > manually set if they want. I like it, because it doesn't require login to > get some basic browsing prefs and features. > > Hooking up an auth system would be necessary for some of the editing ideas > in the document I read, or to allow online posting. So Postorius (the webUI) has a sketch of an auth system using BrowserID at the moment. BrowserID is convenient 'cause it proves you have ownership of a given email address, but we should have OpenID working soon once we've got the code to confirm that a given OpenID can be associated with an email address. We should do a little thinking about how to make sure that the archives system can make use of the webui authentication. In theory, you could just use the same browserID/etc. and perform authentication again to provide a single sign on with the same tokens, but we can probably do something nicer by sharing the webui django accounts. Terri

2 1

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Barry Warsaw March 28, 2012

March 28, 2012

On Mar 26, 2012, at 10:37 PM, David Jeske wrote: >Yes. My CSLA code is on the table for MM2 or 3. It's python, and it's BSD. >More than that, if you folks want it, I'm happy to retool/rewrite it as >needed to make it suitable. (it has lots of great features, but i havn't >touched it in a long time and it could be better) I'll put up a demo of the >mailman list in CSLA tomorrow so folks can check it out. > >I also have "some experience" with this space, and by that I mean >mountainous-experience. I coded/managed/architected several versions of >eGroups/Yahoo Groups and Google Groups, including the listprocessor, schema >design, and a few different python mail-to-html formatters. I also wrote my >own personal gmail-like web-mailer in python, which someday I'll get around >to open-sourcing. See, stuff like this is *awesome*. >Can you share something about dependency philosophy (besides licensing) in >Mailman? I really want to promote loose dependencies between major components, with well-defined formal interfaces for them to interoperate. Think "service oriented architecture". Unlike mm2, which tightly bundled the web ui and archiver with the core engine, I think we can split these out into separate components and yet ensure they'll all work together nicely. Then we can decide what might fall under the "GNU Mailman" umbrella, what will be our default services, how we're going to bundle and release them, etc. But others will be able to make their own choices. >Currently CSLA uses Clearsilver templating (BSD c-module) and sqlite >(public domain) storage.. both of which can be included directly as source >to make installation easy, but they are both c-modules. > >I'm quite fond of the Clearsilver orm-to-templates toolchain, which is in >use for lots of big sites. However, I also recognize it's not that popular >in the python commmunity, mostly because we don't promote it, but also >because the template processor c-module (for performance) instead of pure >python. In a way, it's like the Python library ecosystem. Most third party libraries are independently developed and released, but pretty much work with (a specific version of) Python. Occasionally, some are borged into the Python stdlib, which is both a blessing and a curse. So in some sense, CSLA needn't become *the* Mailman archiver, but it should definitely be *a* Mailman archiver. Then you can make all the engineering and design decisions you prefer, but with the confidence that it will Just Work with Mailman 3. (This of course also means you don't have to worry about boring stuff like licensing, FSF copyright assignments, etc. etc.) >Using SMTP for an included archiver would require it be a long running >server instead of merely a handler. > >Are we talking about third-party-site archivers here? In some sense, yes. Eventually, something will percolate to the top, have technological decisions that we're comfortable with maintaining, compatible licensing, etc. and then we can bring it under the Mailman umbrella. But hey, don't wait for that! Make it work with the mm3 engine now and release early and often. (I clarified in a previous message that SMTP into the archiver is not a requirement. The implementation should be flexible enough to handle just about any input mechanism you need.) >CSLA doesn't currently have any concept server-auth. The only stateful >features it has are view-preferences and read-state, neither of which are >important enough to require a password. It uses a password-less system >which uses cookies for prefs and a 'read state userid' which a user can >manually set if they want. I like it, because it doesn't require login to >get some basic browsing prefs and features. > >Hooking up an auth system would be necessary for some of the editing ideas >in the document I read, or to allow online posting. Or to support private archives, which is very definitely a feature goal IMO. Even a 99% public site will likely have a few closed or limited access lists, so I think an archiver will need to support use case. -Barry

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Barry Warsaw March 28, 2012

March 28, 2012

On Mar 27, 2012, at 10:34 AM, Stephen J. Turnbull wrote: >Pipermail is going the way of the dodo, yes, but there will be something >bundled with Mailman, I'm pretty sure. fsvo "bundled" :) >(1) the communication from Mailman to the archivers will be via >LMTP/SMTP, including a Mailman-specific header to identify the >message's permalink (currently the SHA1 hash of the message ID >in BASE32 format, IIRC); and Kind of. Communication *into* Mailman is via LMTP, but sending a message out of mm3 into the archiver can be hidden behind the IArchiver interface. The prototype archiver, just opens a maildir and adds the new message to that. The mailarchive implementation drops the message into the outgoing queue, but with a recipient as specified in the configuration file. The mhonarc archiver calls a subprocess and pipes the message's bytes to that process's stdin. The moral is that whatever you need to do to get the message into the archiver, you can do it. You're right about the algorithm for calculating the X-Message-Id-Hash. Full details are on the wiki. The intent here is to be able to calculate the URL to the message in the archiver without actually having to communicate with the archiver. >(2) the summary, search, and retrieval UI will be a separate >application from Mailman per se, which will communicate >with Mailman via the REST API for authentication and user >configuration purposes. Yep. I've also been thinking; we have an IMessageStore interface which currently isn't much hooked up in any way. It's possible that a specific archiver could satisfy the IMessageStore API as well, and that eventually we could implement things like a mail command that, given a Message-ID (or its hash) would send you back the message from the store. >Some ideas that have *not* been elaborated as "philosophy", but >which I think are consistent with various desiderata and Barry's >general approach and specific statements are > >(3) an archiver Handler for the pipeline will be provided, which >will store posts in maildir format and do nothing else; and I've been thinking that the prototype archiver could be this thing. Well, it already essentially *is* that thing! So, it's not in a handler, but gets invoked from the ArchiveRunner. >(4) a simple default summary/search/retrieval application will >be bundled with (but have a separate development cycle from) >Mailman. +1 >Even if that's not the current philosophy <wink/> that's the >direction I would propose. > >So if you have already developed some stuff, I certainly would >love to see you put it on the table as a candidate for the >Mailman 3 default archive web UI. Definitely. My intent with mm3 is to provide an enabling platform for archiver developers. I think there's lots of opportunity for experimentation here, so let's make sure we get the API between mm3 and a generic archiver right. We'll see what pans out and then we can choose a preferred default that we can bless and bundle in some kind of sumo release. Cheers, -Barry

1 0

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Terri Oda March 28, 2012

March 28, 2012

On 03/27/2012 03:31 AM, Shayan Md wrote: > I was working on mm3. But systers' indexer/searcher was implemented for > mailman2. So it must be easy for to integrate it with mm2. Actually, the systers indexer was designed to work with mboxes (because I had a pile of data in that format that the students could use) but otherwise knows pretty much nothing about mailman 2 or 3. Other than handling mbox instead of maildir, which is only a matter of changing parsers, it shouldn't matter which it's integrated with. This was a design decision at the time, as Mailman 3 was coming but still too incomplete to test with when the code was written. > Looks like archiver for mm3 is still in development stage. As far as I > understand searcher depends on the srchiver, right? Not completely but it > somewhat depends on archiver. I am not sure if searcher can be implemented > without archiver. If possible I can implement for mm3 also. Searcher and archiver are interdependent *if* we want to share caches and data stores, which we probably do for any installation with larger archives where storing 2 copies vs 4 of each message would make a difference. Plus, many archive views may be basically searches "messages in the last month" "messages which are replies to messageid $foo" etc. Ideally, anyone working on search will interact heavily with the archiver and probably usability folk at the beginning so that you can figure out what data structures you need to store and index and what use cases you'll need to make fast. Terri

2 1

Re: [Mailman-Developers] RELEASED: GNU Mailman 3.0 beta 1 andPostorius 1.0 alpha 1
by Mark Sapiro March 28, 2012

March 28, 2012

Jeff Breidenbach wrote: >Congratulations! I was able to get Postorius running by following the >five minute quick start guide. I didn't see archiving settings in the >user interface, how do I set that up? We've finally ditched Pipermail for archiving. Archiving in MM3 will be "pluggable" and archiving projects are being worked on with more work scheduled for GSoC this summer. In the mean time, for public archives at least, you can use mail-archive.com or some other archiving service. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 1

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by David Jeske March 27, 2012

March 27, 2012

On Mar 27, 2012 5:32 AM, "Stephen J. Turnbull" <stephen(a)xemacs.org> wrote: > If the archiver/web UI is going to be distributed > *with* Mailman, Barry would probably prefer > Storm + Django because that's what > Mailman/Protorius (core and admin web UI, resp.) are > using. But I imagine that's negotiable as long as > everything is free software. I'll take a look at the Mailman2/3 code and see what the pattern looks like. Storm-ORM looks like it allows any primary key so that's good. From a quick glance it looks very similar to the Clearsilver-odb-orm in syntax and function. My primary beef with django-orm (last time I looked at it) was it requiring a unique-id primary key which is pretty poor for multi-row fetch performance design. Clearsilver templating has a slightly more aggressive enforcement of 'no code in template' than django, because it has an explicit intermediate dataset called HDF (it doesn't allow access to arbitrary python lists, structs, classes, and methods). This allows it to work from any language (not just python), and makes the binding from the ORM a bit more explicit. However, it also means it's a c-module, and I can see the advantage to keeping c-compile dependencies to a minimum. >>> AIUI, the current philosophy is that >>> >> Using SMTP for an included archiver would require >> it be a long running server instead of merely >> a handler. > > Sure, but you're already running a sufficient server, > namely your MTA. Gotcha. I thought you were saying Mailman wanted the archiver to be a long-running server accepting SMTP/LMTP on a socket. I normally manage the archive queue with the MTA, and have it launch the archive-injestor for each message.. (or for higher volume applications, long-running injester on a Maildir).. which sounds like what you're saying.. After I finish some quick cleanup and UI changes to CSLA I want to do, I'll take a look at MM2/3 and see if I have better questions once I'm informed. Thanks for the great overview.

1 0

Re: [Mailman-Developers] [Bug 937154] Re: bin/disabled.py is nonfunctional
by Barry Warsaw March 27, 2012

March 27, 2012

On Mar 26, 2012, at 08:03 PM, Andrea Crotti wrote: > Should it provide exactly the same command line interface? Not necessarily. Looking at the code now I think the long options are probably fine, but I'm not sure the short options are great (e.g. -o is usually reserved for output to a file; not relevant for this particular script, but I don't like appropriating commonly understood options if at all possible). Also --unknown won't be useful now; that was a nod to some bug in MM2.1 we never figured out. ;) If you make it a bin/mailman subcommand, you won't need to re-implement -C here. >- Does it need to be independent from the rest of the code or should > it be added to the mailman commands?? I think I would make it a bin/mailman subcommand. One of the main use cases for this script is to implement a cron job that site admins can enable to process users in various statuses. But I think a cron could invoke a subcommand just fine. Still, that use case should inform its cli. >Anything else that I should know? One thing to keep in mind is that in mm3, I'm trying to reduce the amount of logic in the actual commands. Meaning, it would be better to move as much as possible into core services, and have the command do as little as possible, over cli parsing and such.

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Stephen J. Turnbull March 27, 2012

March 27, 2012

On Tue, Mar 27, 2012 at 2:37 PM, David Jeske <davidj(a)gmail.com> wrote: > Can you share something about dependency philosophy (besides licensing) in > Mailman? Well, for the official poop you'll have to wait for Barry, but AFAICS archivers aren't restricted to Storm + RESTish (which is what Mailman itself uses) because they're separate applications. If the archiver/web UI is going to be distributed *with* Mailman, Barry would probably prefer Storm + Django because that's what Mailman/Protorius (core and admin web UI, resp.) are using. But I imagine that's negotiable as long as everything is free software. I don't think the database backend much matters, but Mailman and Django are both happy to use sqlite, I believe. > I'm not a fan of javascript client-side rendering because of the generally > poor performance, poor mobile compatibility, and lack of benefit for this > kind of application. Not my pidgin, you'll need to talk to Toshio/Barry about that. >> AIUI, the current philosophy is that >> >> (1) the communication from Mailman to the archivers >> will be via LMTP/SMTP, including a Mailman-specific >> header to identify the message's permalink (currently >> the SHA1 hash of the message ID in BASE32 >> format, IIRC); and > > Using SMTP for an included archiver would require it be a long running > server instead of merely a handler. Sure, but you're already running a sufficient server, namely your MTA. Of course that may not be efficient. If an mbox or maildir storage is sufficient, any decent MTA/MDA can do that for you. If not, I don't think that writing a separate handler is a big deal; I'm just saying that AIUI the configurable Handler provided with Mailman will certainly know how to do LMTP/ SMTP. > Are we talking about third-party-site archivers here? That is the motivation for choosing ?MTP as the default transport to archivers, yes. Steve

1 0