Mailman 3 March 2012 - Mailman-Developers

bugs on launchpad
by Andrea Crotti March 23, 2012

March 23, 2012

Hi everyone, I filed in two bugs today: https://bugs.launchpad.net/mailman/+bug/956889 https://bugs.launchpad.net/mailman/+bug/956384 because I thought we could have a discussion there about these topics. But now I was wondering if it is a good idea, is it maybe better to write on the mailing list for these kind of things?? PS. ironically today they asked me for the first time in my life to check why Mailman was not working on a Linux server. The problem was that /var was full, but well knowing more about Mailman certainly helped :D

3 3

Re: [Mailman-Developers] bugs on launchpad
by Barry Warsaw March 23, 2012

March 23, 2012

On Mar 18, 2012, at 04:36 PM, Stephen J. Turnbull wrote: >On Sat, Mar 17, 2012 at 3:05 AM, Andrea Crotti ><andrea.crotti.0(a)gmail.com> wrote: >> Hi everyone, >> I filed in two bugs today: >> https://bugs.launchpad.net/mailman/+bug/956889 >> https://bugs.launchpad.net/mailman/+bug/956384 > >These are both policy issues, I think they should be posted to the >list as well as filed on Launchpad, or instead of in the case of >continuous integration. Once it's decided what to do about them, a >specific issue should filed to track progress. > >IMO YMMV etc -- I don't pretend to channel Barry on this. But you didn't do so badly! :) -Barry

1 0

Killing off Pipermail and the effects on scrubbing in Mailman 3
by Barry Warsaw March 21, 2012

March 21, 2012

There's an aspect of the scrubber that isn't going to work in a Mailman 3 world where we have multiple, possibly external, archivers and especially where we don't have such tight integration with Pipermail (or Pipermail at all <wink>). We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them. I can think of a few ways to handle this. The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues. This makes sense because with a model of external archiving, those archivers may make different decisions about what should be removed or displayed from the original message. We can still include a little blurb saying that a part was scrubbed out, and since the messages can have the pre-calculated url to the message in one or more archivers, the user is always free to just click on the url to see the full message, displayed with whatever policy the archiver is configured with. One possibility is to save the scrubbed part inside the core and provide a url to the REST API for accessing this attachment. This can't be inserted into the scrubbed message directly though because this would be a non-public url to the resource, and it would have to be proxied by the web ui. We need better configuration for integrating the web ui with the core any way (e.g. to calculate the url to the user's options page), so this could be part of that. The interactions are trickier though because you would then have to inform the web ui that there's a new attachment it should proxy. The other, more elaborate option is to define an IScrubber interface, or alternatively a "primary" IArchiver, that the message can pass through, which would give it an opportunity to provide urls for each of the parts that will be scrubbed out. This is trickier because there can really be only one such thing defined in the system. I think it would be confusing if you received a message that had something like this: text/html part scrubbed, view it at one of the following: http://example.com/attachments/foo.html http://example.org/some/extra/path/bar.html http://another.archive.example.net/whatever/baz.html Besides, this may be nearly impossible to do without in-band communication with that external archiver, which is exactly what the RFC 5064 + message-id-hash was supposed to avoid. I think we definitely don't want to have to force such in-band communications to occur in order to scrub messages of unwanted parts. For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler! Any other suggestions? Cheers, -Barry

8 22

Re: [Mailman-Developers] Grackle archive framework
by Toshio Kuratomi March 19, 2012

March 19, 2012

On Sun, Mar 18, 2012 at 10:55:19AM +0530, Aamir Khan wrote: > On Sun, Mar 18, 2012 at 4:24 AM, Barry Warsaw <barry(a)list.org> wrote: > > > On Mar 18, 2012, at 12:23 AM, Aamir Khan wrote: > > > > >On Fri, Feb 17, 2012 at 12:55 AM, Barry Warsaw <barry(a)list.org> wrote: > > >> On IRC, we talked about a storm + Python mailbox library based backend, > > >> with a > > >> REST+JSON wsgi based application vending the data. This would allow us > > to > > >> integrate fairly easily with MM3 I think, and would possibly better > > enable > > >> some of the archiver work being done by Terri and others. > > >> > > > > > >I understand that we will store the messages in .mbox format. But I don't > > >understand why do we need to use storm for the archiving purpose. > > > > I meant to say "maildir". Please let's not use mbox format! It's way too > > easy to corrupt the file, as we did with a bug once in MM2.1, and we've > > paid > > the price ever since. > > > > I read the difference between maildir and mbox format and it clearly states > that mbox is prone to corruption while maildir is not. Also there are more > advantages using maildir in a way that there is no file locking problem. > But since we will be storing each mail in a separate file, searching > through them will not as fast enough. Using database alone also have > problems like, it will use more hard disk, more CPU cycles will be consumed. > > So, if we can store the messages in maildir format with a copy of it it > database. we can serve the searching request using database query which > will powered by full-text search engine. But then there will be problems of > synchronization between the maildir messages and messages stored in > database. What are your thoughts about it ? > > As for searching the archive, there are solutions like Elastic Search, > Solr, lucene. Can we use one of them to search directly through the maildir. > Note that a few of us have been playing with a searching-archiver. An initial prototype used notmuch. We looked into using raw xapian at pycon. And now, one of our developers (pingou on IRC) has pushed out a prototype that uses mongodb for the backend. You can take a look at our development copy here: http://mm3test.fedoraproject.org/2/list/devel@fp.o I'll be working on splitting out a tested copy from an in-development copy later today. That way we won't be creating web pages with tracebacks all the time :-) Code for this is available in the hyperkitty mongodb branch: bzr branch bzr://bzr.fedorahosted.org/bzr/hyperkitty/mongodb -Toshio

1 0

Re: [Mailman-Developers] Thoughts on processing for pre-approved messages
by Mark Sapiro March 19, 2012

March 19, 2012

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 3/18/2012 3:28 AM, Stephen J. Turnbull wrote: > > ... Having both one or more Handlers and a special seems like > overkill, especially since really one checks the header and the > other deletes, completely different functionality. Wouldn't it be > better to have a class variable Mlist.approval_headers = > ["Approve", "X-Approve"] and have > > for h in mlist.approval_headers: if msg[h] == > mlist.moderator_password: return True return False > > for the chain rule and > > for h in mlist.approval_headers: del msg[h] > > in a RemoveApprovalHeaders.py pipeline handler? If that were all that was required, that would be fine. The problem is that we allow approval via a pseudo-header as the first non-blank body line in the first text/plain part of the message, and we have to look for it there and if found, not only remove it from that part, but also from any alternative parts in which it might appear. It's the removal of this pseudo-header from text/html alternatives that is the hard part. See the comment thread at <https://bugs.launchpad.net/mailman/+bug/266220>. > ... I think that more likely the problem will be that people will > misspell the header, or use it in list that doesn't support > approval-by-header, or grab an incorrect password out of an old > message, or whatever. Incorrect password is not an issue because we remove it anyway. The other things *should* not be a problem because in theory the poster wouldn't use the header if it weren't required, and in that case, the post will be held since it isn't pre-approved. Of course in practice, held posts get approved even if they might leak a password and users do all sorts of things that don't make sense :( - -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPZiKDVVuXXpU7hpMRAnmWAKD8z234ZKDZGFkAah8jiVW4/Zmd1wCgrtF8 tzWfzmXSJ1uycAo3yeMLpQA= =08Vm -----END PGP SIGNATURE-----

2 1

Thoughts on processing for pre-approved messages
by Mark Sapiro March 18, 2012

March 18, 2012

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've had several thoughts on what is now (in MM 3) done by rules/approved.py. There is a moderator_password list attribute which can be used in a (X-)Approve(d): header or first body line pseudo-header to pre-approve a list post. I've gone around a bit on this and I've concluded this is analogous to the list poster password I implemented for 2.1. Presumably we don't want to allow this password to be used to authenticate to the web ui. We may want to allow it for authentication for certain email commands. I'm not sure about that one. My basic thought is we know who allegedly sent the post (maybe we have a few places to check - the poster might legitimately spoof the From:), so we can find a user who owns that address, see if the user has a moderator or admin role for this list, and validate the header password against that user's password. We *don't* want to do this because the user doesn't want to send her own personal password in a plain text email header. So we use this relatively easily changed and not very capable moderator_password instead. So far, so good. Now I see some issues with what rules/approved.py does. It checks for the header and validates the password. This is good. It also removes any header or body lines containing the pseudo-header from the message. Architecturally, this latter operation belongs in the pipeline, not in a chain rule. This seems to say we need both a rules module and a pipeline module (or maybe the existing pipeline/cleanse.py) to do this, and in the interest of DRY, we really need a mlist.check_approved(msg, clean=True|False) method to do the heavy lifting. Unfortunately, this adds complexity and potential for security lapses if the rule hits but the pipeline doesn't remove the authentication. Thoughts? - -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPY3sxVVuXXpU7hpMRAodWAKCCcHOzCm3n7Ik9VUVapsUAHTvONwCghcoN qbND9+Opjm2D+Lb5PTqxIpg= =4wV2 -----END PGP SIGNATURE-----

2 1

Re: [Mailman-Developers] Grackle archive framework
by Stephen J. Turnbull March 18, 2012

March 18, 2012

On Sun, Mar 18, 2012 at 2:25 PM, Aamir Khan <syst3m.w0rm(a)gmail.com> wrote: > As for searching the archive, there are solutions like Elastic Search, > Solr, lucene. Can we use one of them to search directly through the maildir. Not quickly. Many archives will have thousands of messages, some will have hundreds of thousands. There's no reasonable way to avoid indexes when you've got more than "dozens" of files. The stats alone will kill performance. If you want a full-text solution, you're going to have to use a lot of disk space (typically as much as the compressed archives themselves, and for some databases, more). Also, I don't really see why a few hours' delay for indexing recent messages would be a problem.

1 0

Re: [Mailman-Developers] Grackle archive framework
by Barry Warsaw March 18, 2012

March 18, 2012

On Mar 18, 2012, at 12:23 AM, Aamir Khan wrote: >On Fri, Feb 17, 2012 at 12:55 AM, Barry Warsaw <barry(a)list.org> wrote: >> On IRC, we talked about a storm + Python mailbox library based backend, >> with a >> REST+JSON wsgi based application vending the data. This would allow us to >> integrate fairly easily with MM3 I think, and would possibly better enable >> some of the archiver work being done by Terri and others. >> > >I understand that we will store the messages in .mbox format. But I don't >understand why do we need to use storm for the archiving purpose. I meant to say "maildir". Please let's not use mbox format! It's way too easy to corrupt the file, as we did with a bug once in MM2.1, and we've paid the price ever since. As for archiving, it isn't strictly necessary to use storm, it's just a nice lightweight ORM I happen to like. But I think it *does* make sense to back a full-fledged archiver with a database and a full-text search engine. For example, using our RFC 5064+X-Message-ID-Hash scheme, the database would handle the lookup from hash to actual message storage location. Cheers, -Barry

2 1

Re: [Mailman-Developers] Grackle archive framework
by Barry Warsaw March 17, 2012

March 17, 2012

On Feb 16, 2012, at 11:40 PM, Aamir Khan wrote: >I talked to people on #launchpad-dev, as suggested by barry to investigate >about the possibility of using grackle as new archive framework for >mailman. The project isn't functional yet. There are two parts, client and >back-end service to store the messages. Client will hopefully be completed >by end of next week. > >For backend-service implementation, i am also thinking to get involved >during its development. They are planning to have cassandra based store for >storing the messages. Any thoughts about which backend service would be >ideal to store messages for mailman installations ? On IRC, we talked about a storm + Python mailbox library based backend, with a REST+JSON wsgi based application vending the data. This would allow us to integrate fairly easily with MM3 I think, and would possibly better enable some of the archiver work being done by Terri and others. -Barry

2 1

programming languages
by Andrea Crotti March 16, 2012

March 16, 2012

I just noticed that on the launchpad page the programming languages appears: Programming Languages: Python, C but I don't see any trace of C... Cheers, Andrea

2 1