Mailman 3 March 2012 - Mailman-Developers

python 3
by Andrea Crotti 24 Mar '12

24 Mar '12

Someone told me that I should poke Barry about Python3 support, so here I am ;) It's clearly something that can be done really soon, but I think it's worth to start thinking about it. In particular I noticed that lazr.config have been untouched since 2009, which makes me think that is less likely that they will be ported to Python 3. Is it a valid assumption or there is a porting plan? Are there any other blocking factors in general from libraries or other issues??

3 2

GSoC 2012
by vikash agrawal 24 Mar '12

24 Mar '12

Hi Everyone, I am Vikash and very much interested in contributing to mailman and being a GSoC student this year. So far, I have successfully installed mailman in my system. I do have skills in Python 2.7 but as I am very new to mailman thus I am looking for something small to hack and doable in this summer. Also, the idea page doesnot mention the skills required for the project so its somewhat difficult for me to choose one. As a result I would like you to guide me over the same . I am willing to learn a lot this summer :-) Regards Vikash Agrawal -- sent via HTC Sensation

1 0

Re: [Mailman-Developers] bugs on launchpad
by C Nulk 24 Mar '12

24 Mar '12

On 3/22/2012 5:48 PM, Stephen J. Turnbull wrote: > On Fri, Mar 23, 2012 at 1:09 AM, Barry Warsaw <barry(a)list.org> wrote: > >> If we add something like $archive_url to the decoration interpolation >> dictionary, what happens when we have multiple archivers that support >> permalink() enabled? Do we chose one at random (these are unordered)? > -1 I agree so -2. I thinking is if you list administrator/owner has not defined an order, make it simple and impose a known order in that case - alphabetize/sort the order and pick the first one. > >> Do we add a configuration variable to select a "primary" archiver for this? > +1 +2 This would allow the list administrator/owner to define and pick the primary archiver. For any other archivers, people can go to the Archived-At header(s). > > If there's *no* archiver configuration variable, we don't know where to > point to anyway for Gmane-like archivers (yes, I know, catering to Gmane > will give Brad hives, but some people like Gmane). The local archive, if it > exists, may seem TOOWTDI, but it might not be the preferred one even if > it's the only one configured in Mailman (although that seems a lot less > likely than the case where somebody just grandfathers an existing Gmane > subscription as the archive URL of choice). > >> Do we allow something like $archive_url.prototype as an interpolation >> variable to select e.g. the 'prototype' archiver? > -1 > > Too complex. If people want alternative archives, they can go to > the list FAQ or the Archived-At headers. If we go this way, we'd > better have a *really* good comment and a pre-existing entry in > the FAQ to save Mark some time. :-) I agree with Stephen. Too complex and can be confusing without it being well documented with a number of examples.

1 0

Re: [Mailman-Developers] bugs on launchpad
by Stephen J. Turnbull 23 Mar '12

23 Mar '12

But how does the option get into your web UI? If you have to put it in there by editing mm_config, I don't see the gain. Remember, many archivers already have the rather convenient (to non-shell-on-Mailman-host list owners and even ordinary users) UI of simply subscribing the archiver to the list. I would say that one way to implement this would be as a role in the user database. Instead of subscribing an archiver as "member", subscribe it with the "archivist" role. Gmane would probably be willing to do this, for example, if the subscription method were as simple as subscribing an ordinary member. I suppose the same goes for mail-archive.com. On Fri, Mar 23, 2012 at 10:13 AM, andrea crotti <andrea.crotti.0(a)gmail.com> wrote: > My idea as a user would be to be able to select from my > web configuration page an option: > > send_public_url_in_mail_footers: [ > disabled, > gmane_url, > mailmanweb_url > ] > > And maybe it would be cool (if possible I have no idea) to add a mail > command that when I send > > subject: public $mailing_list $id > > I get back an email with this url. Because maybe from one mailing list > I'm normally not interested in the url and I also want it in some cases. > > Not sure I anyone else in the world would like this feature though :D > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/stephen%40xemacs.… > > Security Policy: http://wiki.list.org/x/QIA9

1 0

Re: [Mailman-Developers] bugs on launchpad
by andrea crotti 23 Mar '12

23 Mar '12

My idea as a user would be to be able to select from my web configuration page an option: send_public_url_in_mail_footers: [ disabled, gmane_url, mailmanweb_url ] And maybe it would be cool (if possible I have no idea) to add a mail command that when I send subject: public $mailing_list $id I get back an email with this url. Because maybe from one mailing list I'm normally not interested in the url and I also want it in some cases. Not sure I anyone else in the world would like this feature though :D

1 0

bugs on launchpad
by Andrea Crotti 23 Mar '12

23 Mar '12

Hi everyone, I filed in two bugs today: https://bugs.launchpad.net/mailman/+bug/956889 https://bugs.launchpad.net/mailman/+bug/956384 because I thought we could have a discussion there about these topics. But now I was wondering if it is a good idea, is it maybe better to write on the mailing list for these kind of things?? PS. ironically today they asked me for the first time in my life to check why Mailman was not working on a Linux server. The problem was that /var was full, but well knowing more about Mailman certainly helped :D

3 3

Re: [Mailman-Developers] bugs on launchpad
by Barry Warsaw 23 Mar '12

23 Mar '12

On Mar 18, 2012, at 04:36 PM, Stephen J. Turnbull wrote: >On Sat, Mar 17, 2012 at 3:05 AM, Andrea Crotti ><andrea.crotti.0(a)gmail.com> wrote: >> Hi everyone, >> I filed in two bugs today: >> https://bugs.launchpad.net/mailman/+bug/956889 >> https://bugs.launchpad.net/mailman/+bug/956384 > >These are both policy issues, I think they should be posted to the >list as well as filed on Launchpad, or instead of in the case of >continuous integration. Once it's decided what to do about them, a >specific issue should filed to track progress. > >IMO YMMV etc -- I don't pretend to channel Barry on this. But you didn't do so badly! :) -Barry

1 0

Killing off Pipermail and the effects on scrubbing in Mailman 3
by Barry Warsaw 21 Mar '12

21 Mar '12

There's an aspect of the scrubber that isn't going to work in a Mailman 3 world where we have multiple, possibly external, archivers and especially where we don't have such tight integration with Pipermail (or Pipermail at all <wink>). We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them. I can think of a few ways to handle this. The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues. This makes sense because with a model of external archiving, those archivers may make different decisions about what should be removed or displayed from the original message. We can still include a little blurb saying that a part was scrubbed out, and since the messages can have the pre-calculated url to the message in one or more archivers, the user is always free to just click on the url to see the full message, displayed with whatever policy the archiver is configured with. One possibility is to save the scrubbed part inside the core and provide a url to the REST API for accessing this attachment. This can't be inserted into the scrubbed message directly though because this would be a non-public url to the resource, and it would have to be proxied by the web ui. We need better configuration for integrating the web ui with the core any way (e.g. to calculate the url to the user's options page), so this could be part of that. The interactions are trickier though because you would then have to inform the web ui that there's a new attachment it should proxy. The other, more elaborate option is to define an IScrubber interface, or alternatively a "primary" IArchiver, that the message can pass through, which would give it an opportunity to provide urls for each of the parts that will be scrubbed out. This is trickier because there can really be only one such thing defined in the system. I think it would be confusing if you received a message that had something like this: text/html part scrubbed, view it at one of the following: http://example.com/attachments/foo.html http://example.org/some/extra/path/bar.html http://another.archive.example.net/whatever/baz.html Besides, this may be nearly impossible to do without in-band communication with that external archiver, which is exactly what the RFC 5064 + message-id-hash was supposed to avoid. I think we definitely don't want to have to force such in-band communications to occur in order to scrub messages of unwanted parts. For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler! Any other suggestions? Cheers, -Barry

8 22

Re: [Mailman-Developers] Grackle archive framework
by Toshio Kuratomi 20 Mar '12

20 Mar '12

On Sun, Mar 18, 2012 at 10:55:19AM +0530, Aamir Khan wrote: > On Sun, Mar 18, 2012 at 4:24 AM, Barry Warsaw <barry(a)list.org> wrote: > > > On Mar 18, 2012, at 12:23 AM, Aamir Khan wrote: > > > > >On Fri, Feb 17, 2012 at 12:55 AM, Barry Warsaw <barry(a)list.org> wrote: > > >> On IRC, we talked about a storm + Python mailbox library based backend, > > >> with a > > >> REST+JSON wsgi based application vending the data. This would allow us > > to > > >> integrate fairly easily with MM3 I think, and would possibly better > > enable > > >> some of the archiver work being done by Terri and others. > > >> > > > > > >I understand that we will store the messages in .mbox format. But I don't > > >understand why do we need to use storm for the archiving purpose. > > > > I meant to say "maildir". Please let's not use mbox format! It's way too > > easy to corrupt the file, as we did with a bug once in MM2.1, and we've > > paid > > the price ever since. > > > > I read the difference between maildir and mbox format and it clearly states > that mbox is prone to corruption while maildir is not. Also there are more > advantages using maildir in a way that there is no file locking problem. > But since we will be storing each mail in a separate file, searching > through them will not as fast enough. Using database alone also have > problems like, it will use more hard disk, more CPU cycles will be consumed. > > So, if we can store the messages in maildir format with a copy of it it > database. we can serve the searching request using database query which > will powered by full-text search engine. But then there will be problems of > synchronization between the maildir messages and messages stored in > database. What are your thoughts about it ? > > As for searching the archive, there are solutions like Elastic Search, > Solr, lucene. Can we use one of them to search directly through the maildir. > Note that a few of us have been playing with a searching-archiver. An initial prototype used notmuch. We looked into using raw xapian at pycon. And now, one of our developers (pingou on IRC) has pushed out a prototype that uses mongodb for the backend. You can take a look at our development copy here: http://mm3test.fedoraproject.org/2/list/devel@fp.o I'll be working on splitting out a tested copy from an in-development copy later today. That way we won't be creating web pages with tracebacks all the time :-) Code for this is available in the hyperkitty mongodb branch: bzr branch bzr://bzr.fedorahosted.org/bzr/hyperkitty/mongodb -Toshio

1 0

Re: [Mailman-Developers] Thoughts on processing for pre-approved messages
by Mark Sapiro 19 Mar '12

19 Mar '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 3/18/2012 3:28 AM, Stephen J. Turnbull wrote: > > ... Having both one or more Handlers and a special seems like > overkill, especially since really one checks the header and the > other deletes, completely different functionality. Wouldn't it be > better to have a class variable Mlist.approval_headers = > ["Approve", "X-Approve"] and have > > for h in mlist.approval_headers: if msg[h] == > mlist.moderator_password: return True return False > > for the chain rule and > > for h in mlist.approval_headers: del msg[h] > > in a RemoveApprovalHeaders.py pipeline handler? If that were all that was required, that would be fine. The problem is that we allow approval via a pseudo-header as the first non-blank body line in the first text/plain part of the message, and we have to look for it there and if found, not only remove it from that part, but also from any alternative parts in which it might appear. It's the removal of this pseudo-header from text/html alternatives that is the hard part. See the comment thread at <https://bugs.launchpad.net/mailman/+bug/266220>. > ... I think that more likely the problem will be that people will > misspell the header, or use it in list that doesn't support > approval-by-header, or grab an incorrect password out of an old > message, or whatever. Incorrect password is not an issue because we remove it anyway. The other things *should* not be a problem because in theory the poster wouldn't use the header if it weren't required, and in that case, the post will be held since it isn't pre-approved. Of course in practice, held posts get approved even if they might leak a password and users do all sorts of things that don't make sense :( - -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPZiKDVVuXXpU7hpMRAnmWAKD8z234ZKDZGFkAah8jiVW4/Zmd1wCgrtF8 tzWfzmXSJ1uycAo3yeMLpQA= =08Vm -----END PGP SIGNATURE-----

2 1

Thoughts on processing for pre-approved messages
by Mark Sapiro 18 Mar '12

18 Mar '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've had several thoughts on what is now (in MM 3) done by rules/approved.py. There is a moderator_password list attribute which can be used in a (X-)Approve(d): header or first body line pseudo-header to pre-approve a list post. I've gone around a bit on this and I've concluded this is analogous to the list poster password I implemented for 2.1. Presumably we don't want to allow this password to be used to authenticate to the web ui. We may want to allow it for authentication for certain email commands. I'm not sure about that one. My basic thought is we know who allegedly sent the post (maybe we have a few places to check - the poster might legitimately spoof the From:), so we can find a user who owns that address, see if the user has a moderator or admin role for this list, and validate the header password against that user's password. We *don't* want to do this because the user doesn't want to send her own personal password in a plain text email header. So we use this relatively easily changed and not very capable moderator_password instead. So far, so good. Now I see some issues with what rules/approved.py does. It checks for the header and validates the password. This is good. It also removes any header or body lines containing the pseudo-header from the message. Architecturally, this latter operation belongs in the pipeline, not in a chain rule. This seems to say we need both a rules module and a pipeline module (or maybe the existing pipeline/cleanse.py) to do this, and in the interest of DRY, we really need a mlist.check_approved(msg, clean=True|False) method to do the heavy lifting. Unfortunately, this adds complexity and potential for security lapses if the rule hits but the pipeline doesn't remove the authentication. Thoughts? - -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPY3sxVVuXXpU7hpMRAodWAKCCcHOzCm3n7Ik9VUVapsUAHTvONwCghcoN qbND9+Opjm2D+Lb5PTqxIpg= =4wV2 -----END PGP SIGNATURE-----

2 1

Re: [Mailman-Developers] Grackle archive framework
by Stephen J. Turnbull 18 Mar '12

18 Mar '12

On Sun, Mar 18, 2012 at 2:25 PM, Aamir Khan <syst3m.w0rm(a)gmail.com> wrote: > As for searching the archive, there are solutions like Elastic Search, > Solr, lucene. Can we use one of them to search directly through the maildir. Not quickly. Many archives will have thousands of messages, some will have hundreds of thousands. There's no reasonable way to avoid indexes when you've got more than "dozens" of files. The stats alone will kill performance. If you want a full-text solution, you're going to have to use a lot of disk space (typically as much as the compressed archives themselves, and for some databases, more). Also, I don't really see why a few hours' delay for indexing recent messages would be a problem.

1 0

Re: [Mailman-Developers] Grackle archive framework
by Barry Warsaw 18 Mar '12

18 Mar '12

On Mar 18, 2012, at 12:23 AM, Aamir Khan wrote: >On Fri, Feb 17, 2012 at 12:55 AM, Barry Warsaw <barry(a)list.org> wrote: >> On IRC, we talked about a storm + Python mailbox library based backend, >> with a >> REST+JSON wsgi based application vending the data. This would allow us to >> integrate fairly easily with MM3 I think, and would possibly better enable >> some of the archiver work being done by Terri and others. >> > >I understand that we will store the messages in .mbox format. But I don't >understand why do we need to use storm for the archiving purpose. I meant to say "maildir". Please let's not use mbox format! It's way too easy to corrupt the file, as we did with a bug once in MM2.1, and we've paid the price ever since. As for archiving, it isn't strictly necessary to use storm, it's just a nice lightweight ORM I happen to like. But I think it *does* make sense to back a full-fledged archiver with a database and a full-text search engine. For example, using our RFC 5064+X-Message-ID-Hash scheme, the database would handle the lookup from hash to actual message storage location. Cheers, -Barry

2 1

Re: [Mailman-Developers] Grackle archive framework
by Barry Warsaw 18 Mar '12

18 Mar '12

On Feb 16, 2012, at 11:40 PM, Aamir Khan wrote: >I talked to people on #launchpad-dev, as suggested by barry to investigate >about the possibility of using grackle as new archive framework for >mailman. The project isn't functional yet. There are two parts, client and >back-end service to store the messages. Client will hopefully be completed >by end of next week. > >For backend-service implementation, i am also thinking to get involved >during its development. They are planning to have cassandra based store for >storing the messages. Any thoughts about which backend service would be >ideal to store messages for mailman installations ? On IRC, we talked about a storm + Python mailbox library based backend, with a REST+JSON wsgi based application vending the data. This would allow us to integrate fairly easily with MM3 I think, and would possibly better enable some of the archiver work being done by Terri and others. -Barry

2 1

programming languages
by Andrea Crotti 17 Mar '12

17 Mar '12

I just noticed that on the launchpad page the programming languages appears: Programming Languages: Python, C but I don't see any trace of C... Cheers, Andrea

2 1

Thanks for sprinting at Pycon 2012!
by Barry Warsaw 16 Mar '12

16 Mar '12

I'll be blogging in more detail about the Pycon 2012 Mailman sprint once I've recovered from my sleep deprived jetlagged fog, but I just wanted to send a quick note of thanks to all the folks who attended: Andrea Crotti Florian Fuchs Toshio Kuratomi Daniel Mizyrycki Terri Oda Mark Sapiro Stephen Turnbull We made some *incredible* progress on the Mailman 3 core and web-ui. I saw demos of the latter that look fantastic. It was a pleasure to hang out and hack with you guys for a few days. Thanks also to Máirín Duffy who joined us on mumble to share with us 16 of her 32 very cool brainstorms on Mailman's web ui. Special thanks to Larry Hastings for helping us find some nice eats, even if I was an anti-social stick-in-the-mud after the Greek and Thai places. That's what Tamari sauce will do to you. :) Alpha releases of the web ui and a beta of the core will be coming next week once we're all recovered. Stay tuned. Cheers, -Barry P.S. I may even have a few pictures to post later too.

1 0

mailman UI
by Andrea Crotti 15 Mar '12

15 Mar '12

Look what I found reading the news, someone wants to make a social network out of Mailman ;) http://blog.linuxgrrl.com/2012/03/13/mailman-brainstorm/

1 0

courier-to-mailman.py
by Lindsay Haisley 13 Mar '12

13 Mar '12

I just got an email from a fellow in Germany wanting to know where he could find courier-to-mailman.py, a script I contributed to mailman some time ago and which is included in the distribution, but apparently not parsed for replacement tokens and installed in the contrib directory at install time. It might be a good idea to patch configure.in with the proper line to take care of this: --- configure.in.old 2012-03-12 13:41:57.512455181 -0500 +++ configure.in 2012-03-12 13:42:21.252451587 -0500 @@ -681,6 +681,7 @@ bin/rb-archfix \ contrib/check_perms_grsecurity.py \ contrib/qmail-to-mailman.py \ +contrib/courier-to-mailman.py \ contrib/rotatelogs.py \ cron/bumpdigests \ cron/checkdbs \ ... and run autoconf so that the distributed configure file reflects this change. You have the equivalent script for qmail properly generated, but not the courier script. Qmail is pretty much obsolete these days, while courier is under active development and is widely used, notably outside the US. I've been using the courier-to-mailman.py script for the better part of 10 years. -- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | --- The Roadie http://www.fmp.com |

2 1

Pycon 2012 sprint start time
by Barry Warsaw 13 Mar '12

13 Mar '12

Pycon 2012 sprinters, I will probably be in the sprint room by 9:30am tomorrow, but we won't officially start until 10am. Looking forward to seeing everyone there! -Barry

5 4

gsoc-2012
by abdul rauf 11 Mar '12

11 Mar '12

Hi, I was going through ideas page of mailman for GSoC 2012. I would like to work with mailman. If there are more ideas i would like to see it. Currently i am 3rd year of under-graduation at K.B.N.C.E Gulbarga.

1 0

fixing the bug 837700( https://bugs.launchpad.net/mailman/+bug/837700 )
by Jagannathan Tiruvallur Eachambadi 09 Mar '12

09 Mar '12

Barry, In the bug 837700, I have implemented the suggestions you have made in the comment in launchpad. I am unable to test the bug in the first place. Is there a way in which I can reproduce the bug. Since there are only three legitimate users who are are hard coded in the Rest API, how can I give them preferred addresses. Regards, Jagan

1 0

Pycon 2012 and Mailman 3 plans
by Barry Warsaw 06 Mar '12

06 Mar '12

With Pycon 2012 coming up in the next few days, and most of the core Mailman developers attending the sprints[1], we have a opportunity to reach some important milestones with Mailman 3. You are of course invited to join us. Please sign up on the project page below so we can make the appropriate plans. Don't forget to come to my talk on Saturday at 4:55pm local time[2]. We'll have two main goals for the sprint, and possibly a third: - Do a face-to-face integration of the web ui for MM3, identifying any concerns or missing functionality in the API, reviewing the ui design, fix related bugs, etc. - Solidify the core engine so that I can release beta 1. - Possibly, make progress on the next-gen archiver. Yes, you read that right. At long last, I believe the core engine is solid enough to move into beta. This means we'll freeze the database schema and use the migration mechanism for any future updates, and we'll also feature freeze so we can concentrate on bug fixing[3]. So beta 1 is planned for during or just after the sprints. Finally, I have put up a test server, which you are invited to join and use. Right now, you only have access to the email command interface, meaning you must join and leave the mailing list via email commands and confirmations. Once joined, you can of course post to the list. Note that this list is running on a virtual machine, so it, the domain, or the Mailman service can disappear at any time. Occasionally, I'll stop the system, run it in debug mode (meaning your posts will not go to the list), update the software, etc. If something looks weird, please try again, or email me, or file a bug on Launchpad[4]. Please include the original email or at least the Message-Id of the message you sent that caused the problem. I will be watching the logs when I can. Be good citizens and reasonable users. Abuse will not be tolerated and will get you banned at the MTA. - Subscribe by sending a message to playground-join(a)groon.org - Unsubscribe by sending a message to playground-leave(a)groon.org - Get help by sending a message to playground-request(a)groon.org with the word 'help' (sans quotes) in the Subject - Post to the list at playground(a)groon.org Enjoy, and I hope to see you at Pycon. Cheers, -Barry [1] https://us.pycon.org/2012/community/sprints/projects/ [2] I guess I should write it, eh? <wink> https://us.pycon.org/2012/schedule/ [3] Of course, I get to decide what's a bug and what's a missing feature :) [4] https://bugs.launchpad.net/mailman/+filebug Please tag the bug with 'mailman3'

3 2