Mailman 3 March 2012 - Mailman-Developers

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Shayan Md 29 Mar '12

29 Mar '12

On Fri, Mar 30, 2012 at 5:05 AM, Stephen J. Turnbull <stephen(a)xemacs.org>wrote: > On Fri, Mar 30, 2012 at 3:55 AM, Shayan Md <mdoshayan(a)gmail.com> wrote: > > > Assuming that we have something like this(object-ID-addressable, If I am > > not wrong, mailman3 made it possible but not yet implemented as it's part > > of archiver), is it over ambitious to plan to implement indexer/searcher > > for mailman3 and a REST API to use this searcher, extend client to use > this > > api, > > and django search form along with this client api? All this independent > of > > archiver. Because the only part common with archiver is message retrieval > > part, > > If we implement whole searcher, and rest of the client code, later when > > archiver is implemented message retrieval code can used in searcher. When > > archiver is completely mature may we can even merge them together. Is it > > possible? Or this plan has any 'non-sense' parts? > > Hi, Shayan. It's not nonsense, but (1) how do you propose to test if > you have no archives to run it on? And (2) search and retrieval may > do a *lot* of message access, for example if you want to do data > mining (see Ana from Spain's thread). In that case, you may find that > maildir imposes too many stats, which are very expensive on some > platforms (and not cheap on any that I know of) and mbox requires too > large memory. So for some purposes a summary/index/search engine may > want an optimized message store. > Isn't it the purpose of index? I mean, when we search for something we don't have stat all the messages, search the index return the best matching message IDs. If you wish to see the message then retrieve it(may be through archiver). Probably we can index messages through cron every night. > > Those may not be your purposes, of course, in which case a simple mbox > accessor and a download of a couple of mboxes from any public Mailman > list will give you test fodder. > > Testing itself is not really a matter of personal preference. > Although Mailman is not a 100% test-driven shop, Mailman 3 already has > a *lot* of tests and Barry would like to see any new modules covered, > I'm sure. Also, this area is fraught with pitfalls for the developer. > See this thread, for example: > http://thread.gmane.org/gmane.comp.python.devel/131395/focus=131406. > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > http://mail.python.org/mailman/options/mailman-developers/mdoshayan%40gmail… > > Security Policy: http://wiki.list.org/x/QIA9 >

1 0

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Barry Warsaw 29 Mar '12

29 Mar '12

On Mar 30, 2012, at 02:18 AM, Stephen J. Turnbull wrote: >> I suspect that there will be plenty of mailing lists that get fed messages >> from programs, e.g. think vcs -commit diff lists. Those programs can also >> be buggy, but again I'd prefer that Mailman not compromise on this issue >> for their sake. > >I predict you will eventually lose on this. That's not an argument >for changing, of course. :-) Possibly, but let's go down fighting. :) >> The point being that messages that flow through Mailman will have that hash >> in the message URLs in the Archived-At header and possibly in the decorated >> footers. An archiver should certainly provide an interface to look up a >> message by pure Message-ID or the hash. The hash is just a scheme to >> regularize the message id and is a tiny fraction more user-friendly >> (because of its limited alphabet and manageable, known-in-advance length). > >I would say that's actually quite significant, because the trash that >I've observed in message-ids is varied, to be polite. It will be nice >to be able to avoid URL-escaping, RFC 2047 soft line breaks, and all >the rest. Exactly! -Barry

1 0

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Barry Warsaw 29 Mar '12

29 Mar '12

(Aside: Is there some reason why you To: me and CC: the list rather than having the list address in the To: field? I ask because I'm wondering if it's a gmail thing, or something about your MUA, and because I suppress the list copy if I'm CC'd directly, I don't get a List-Post: header, so my MUA's reply-to-list won't work on your messages. I'm mostly curious because I can work around it as you see here, but just wanted to point it out in case you weren't aware of it and/or could easily fix it -- if you even agree it's broken. ;) On Mar 29, 2012, at 11:46 PM, Stephen J. Turnbull wrote: >On Thu, Mar 29, 2012 at 8:07 AM, Barry Warsaw <barry(a)list.org> wrote: > >> Mailman 3 itself requires unique Message-IDs. > >So? FWIW, I don't think I agree with that requirement (even RFC 5322 >doesn't make it a "MUST"), but I'm not going to argue with you about >Mailman 3 design, that's your pidgin. We had a fairly lengthy discussion about this a year or so ago, and I'm pretty sure we came to the consensus (even if not unanimous) that it was reasonable for Mailman to impose this restriction. It's 2012 so if your MUA/MTA can't generate unique message id's there's no reason for us to think you're not a spammer <wink>. >But there's nothing particularly Mailman-3-dependent about archiver web UIs, >though. I don't see any reason why the front end shouldn't be used on my >several gigs of personal archives going back to about 1980, eg, or as a poor >man's webmail. True, the archiver as an independent component can be more lenient. I'm just pointing out that as fed from Mailman, it will only get unique message ids. And if it doesn't, that's a bug in Mailman. >> IIRC, the Mail Archive guys found a very very low collision rate over >> millions of messages, and I think all such cases were basically spam. > >Sure, but XEmacs archives go back to at least 1994. mailarchive.com >is a more recent phenomenon. In the early days of Linux/*BSD >diffusion, there were lots of buggy MUAs/very simple MTAs out there. Sure. And I suspect that there will be plenty of mailing lists that get fed messages from programs, e.g. think vcs -commit diff lists. Those programs can also be buggy, but again I'd prefer that Mailman not compromise on this issue for their sake. >It doesn't do that for subobject content IDs, and more important, >users don't necessarily have the X-Message-ID-Hash (they may have >not-metoo set, they may have gotten the message as a direct Cc). >True, it's easy enough to compute -- if you're a Mailman 3 developer >and know it's present.<wink/> And, of course, why have a Mailman 3 >dependency that is absolutely unnecessary? Right. The point being that messages that flow through Mailman will have that hash in the message URLs in the Archived-At header and possibly in the decorated footers. An archiver should certainly provide an interface to look up a message by pure Message-ID or the hash. The hash is just a scheme to regularize the message id and is a tiny fraction more user-friendly (because of its limited alphabet and manageable, known-in-advance length). -Barry

4 3

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Stephen J. Turnbull 29 Mar '12

29 Mar '12

On Fri, Mar 30, 2012 at 3:55 AM, Shayan Md <mdoshayan(a)gmail.com> wrote: > Assuming that we have something like this(object-ID-addressable, If I am > not wrong, mailman3 made it possible but not yet implemented as it's part > of archiver), is it over ambitious to plan to implement indexer/searcher > for mailman3 and a REST API to use this searcher, extend client to use this > api, > and django search form along with this client api? All this independent of > archiver. Because the only part common with archiver is message retrieval > part, > If we implement whole searcher, and rest of the client code, later when > archiver is implemented message retrieval code can used in searcher. When > archiver is completely mature may we can even merge them together. Is it > possible? Or this plan has any 'non-sense' parts? Hi, Shayan. It's not nonsense, but (1) how do you propose to test if you have no archives to run it on? And (2) search and retrieval may do a *lot* of message access, for example if you want to do data mining (see Ana from Spain's thread). In that case, you may find that maildir imposes too many stats, which are very expensive on some platforms (and not cheap on any that I know of) and mbox requires too large memory. So for some purposes a summary/index/search engine may want an optimized message store. Those may not be your purposes, of course, in which case a simple mbox accessor and a download of a couple of mboxes from any public Mailman list will give you test fodder. Testing itself is not really a matter of personal preference. Although Mailman is not a 100% test-driven shop, Mailman 3 already has a *lot* of tests and Barry would like to see any new modules covered, I'm sure. Also, this area is fraught with pitfalls for the developer. See this thread, for example: http://thread.gmane.org/gmane.comp.python.devel/131395/focus=131406.

1 0

Re: [Mailman-Developers] mailman / archive-ui / licensing questions
by David Jeske 29 Mar '12

29 Mar '12

On Thu, Mar 29, 2012 at 9:16 AM, Stephen J. Turnbull <stephen(a)xemacs.org>wrote: > I would say you should try to retain copyright, and have the Mailman > project distribute it with the S-BSD license under the "mere > aggregation" clause of the GPL. This agrees with my view of the situation as well. Which leads to the question, is the above approach interesting/viable for Mailman-team? (assuming the code does something awesome that people want) That's not the way copyright works, though. It certainly is possible > to assert GPL (or any other) restrictions on given *copies* of > permissively-licensed code. What they mean is that it is Borg-able. You can assimilate S-BSD > code into a GPL project, and that copy is distributed under the GPL. > My response here is just academic license-logic evaluation for entertainment purposes... I know you can "claim" to assert GPL restrictions on copies of S-BSD code. I just don't understand how it could ever be enforced. Any changes made post "assimilation" would be in a state of confusion WRT GPL-restrictability. Even FSF-copyright assignment for those changes isn't sufficient to prove that the initial developer never gave a less-restricted version of his changes out directly to the S-BSD pre-assimilation project under their license. You're back to requiring afadavits from all authors stating they never did so, or perhaps challenging projects to produce checkin-logs to show who saw the change first. ...the thought experiment is merely academic though. I don't want my code in the hands of folks that can even "claim" to restrict it with the GPL.. which means FSF-assignment is not something I'm willing to do for my stuff.

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by Toshio Kuratomi 29 Mar '12

29 Mar '12

On Mon, Mar 26, 2012 at 03:20:05PM -0700, David Jeske wrote: > I'm writing to find out the state of and philosophy surrounding pipermail > in mailman, to see if there is a productive way to provide some > code/development-time to that part of mailman. > > I know there are several decent third-party archivers out there, but many > of the mailing list archives I browse regularly are using pipermail because > it's the mailman default.... one less thing to install, administer, and > upgrade. Unfortunately, pipermail doesn't do a good job formatting > messages for html.. (messages with no line-breaks are the most annoying > problem I regularly run into) > > I've written code for this a number of times (eGroups, Yahoo Groups, Google > Groups). I also released an open-source python/clearsilver/sqlite based > archiver with redundant text-eliding, a few different thread views, and > search... ( http://www.clearsilver.net/archive/ ) which is hardly used > both because I don't try to popularize it, and because many sites just > leave the default (pipermail). > > If there is some code (and time) I can contribute to mailman/pipermail, I'd > like to do so. I'm writing this message to "take a temperature" and find > out what, if any, contributions would be appreciated the mailman > development team. I could imagine answers like: > > a) pipermail is fine... if you want to fix a bug or two submit a patch, but > we don't want to improve it > b) we're ditching pipermail entirely... in the future sites will have to > choose an install an external archiver > c) we'd love pipermail to be improved... but we still want it to be simple, > static-html, and dependency free > d) we'd love a dynamic-ui replacement for pipermail... as long as it uses > the same cgi/templating model as mailman ui > > Thoughts? Is there any help I can offer up here? > We'd love to have work done on the archier! I know that we're ditching pipermail entirely and that archivers are becoming separate from the core mailman. What I don't know is whether mailman3 will eventually have a standard archiver which lives outside of the core mailman but is recommended for installation along with it. At PyCon a few weeks ago, I demoed hyperkitty which showed some of the things that a next generation archiver could do. hyperkitty is continuing to be developed. As I was talking about hyperkitty we touched briefly on what I think is one of the central conundrums about having only unofficial third party archivers: how to have a consistent programatic interface available over http. Grackle is another archiver for mailman that doesn't have the UI bells and whistles of hyperkitty but it does make an effort to expose a REST UI to the world. I think that's a beautiful thing. But I don't like that a site that wanted both would need to run two archivers that were saving mail into two sets of storage. I think there's several ways we could go about this. * We could create a standard REST API that archivers were free but encouraged to implement. * We could expand the python API that archivers needed to expose and then implement the REST API inside of mailman Core (or a re-envisioned, lightweight Grackle). * We could promote a standard archiver much as we're going to promote posterius as the standard admin front-end and that archiver would provide a standard REST API that others could then emulate. hyperkitty: Project page: https://fedorahosted.org/hyperkitty/ Code: http://bzr.fedorahosted.org/bzr/hyperkitty/ Demo: http://mm3test.fedoraproject.org/2 grackle: https://launchpad.net/grackle (One thing I notice about grackle now is that it's AGPL... that's going to be unpleasant for some sites to run. Perhaps we can change that or get some changes added to the AGPL.) -Toshio

5 4

Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
by Shayan Md 29 Mar '12

29 Mar '12

On Wed, Mar 28, 2012 at 6:59 AM, Stephen J. Turnbull <stephen(a)xemacs.org>wrote: > On Wed, Mar 28, 2012 at 4:21 AM, Terri Oda <terri(a)zone12.com> wrote: > > >> Looks like archiver for mm3 is still in development stage. As far as I > >> understand searcher depends on the srchiver, right? Not completely but > it > >> somewhat depends on archiver. I am not sure if searcher can be > implemented > >> without archiver. If possible I can implement for mm3 also. > > > > Searcher and archiver are interdependent *if* we want to share caches and > > data stores, which we probably do for any installation with larger > archives > > where storing 2 copies vs 4 of each message would make a difference. > Plus, > > many archive views may be basically searches "messages in the last month" > > "messages which are replies to messageid $foo" etc. > > Actually, as far as I can see, the summary/search/index/retrieval > functions depend only on the API for the message store. If you > want, you can split this into the database layer and a presentation > layer, of course. However, the database layer is surely going to > have its own schema optimized for the kinds of retrieval its > designer considers important. If the designer emphasizes > threads, however, she is *not* going to try to store messages in > thread order or anything like that. Rather, any reasonable store > will be message-ID-addressable. > > The only tricky issue is that we *do* have to worry about > message-ID collisions of truly different messages and about > messages without message IDs, especially for converted > historical archives. So the API needs to be able to deal > with these issues, probably by returning a set or sequence > of messages. > > Oh, and we probably ought to have a more general notion > of retrievable "object" rather than just messages, as some > archive/retrieval backends may store some types of MIME > part separately. Hopefully these would be presented to > us as MIME parts with external bodies and content IDs. > > I would guess she'll probably store messages in > YY-MM/MSGID, or as git does in "unpacked" > XX/YYYYYYYY... format, where XX are the first two digits > of the hash ID, and YY... are the remaining ones). But it > could easily be backed by an IMAP store or something > more specialized; we don't really care as long as it's > object-ID-addressable. > Assuming that we have something like this(object-ID-addressable, If I am not wrong, mailman3 made it possible but not yet implemented as it's part of archiver), is it over ambitious to plan to implement indexer/searcher for mailman3 and a REST API to use this searcher, extend client to use this api, and django search form along with this client api? All this independent of archiver. Because the only part common with archiver is message retrieval part, If we implement whole searcher, and rest of the client code, later when archiver is implemented message retrieval code can used in searcher. When archiver is completely mature may we can even merge them together. Is it possible? Or this plan has any 'non-sense' parts? > And that's all we want to say about the archiver and the > associated message-retrieval logic, I think. (In fact, it occurs to > me that maybe we should say "RFC 3501" and be done with > it. I don't mean that we necessarily implement IMAP protocol > per se, but some subset of its functionality probably is what we > need from an archiver.) > > Then the schema-specific stuff will use hash IDs to represent > message objects in a portable but schema-specific way. As > it's schema-specific, I don't really see how data structures > can be shared by different searchers. > > So I would say not to worry about the archiver side at all. If > large installations want to implement specialized message- > retrieval, bully for them. But we can go with simple backends, > maildir, mbox, and maybe IMAP, I think. > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > http://mail.python.org/mailman/options/mailman-developers/mdoshayan%40gmail… > > Security Policy: http://wiki.list.org/x/QIA9 >

1 0

Re: [Mailman-Developers] RELEASED: GNU Mailman 3.0 beta 1 and Postorius 1.0 alpha 1
by Terri Oda 29 Mar '12

29 Mar '12

On 03/29/2012 02:51 AM, Odhiambo Washington wrote: > Okay. I have to figure out why the consensus is that I am running django > 1.4 while I actually installed 1.3.1: Just for the record, the reason we suspect Django 1.4 conflicts is that the link you posted with the errors <http://196.200.26.114/~wash/mailman-error.txt> was tossing a lot of errors that came from a Django-1.4 package. For example: File "/usr/local/lib/python2.7/site-packages/Django-1.4-py2.7.egg/django/core/handlers/base.py", line 179, in get_response response = self.handle_uncaught_exception(request, resolver, sys.exc_info()) That rather implies that it was using some pieces that were not 1.3.1, which could account for the errors. Terri

1 0

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
by David Champion 29 Mar '12

29 Mar '12

* On 29 Mar 2012, Stephen J. Turnbull wrote: > On Thu, Mar 29, 2012 at 4:04 PM, David Jeske <davidj(a)gmail.com> wrote: > >> - Merging of forums, archives, newsgroups, and IMAP. > > > > You like to bite off the big ones eh? NNTP, then IMAP. > > > > Does anyone even use fat-mail-clients anymore? > > Yes, and yes. But I don't think anybody has really thought carefully > about these desiderata yet, they're just frequently suggested on > mailman-users and sometimes mailman-developers. +1 For what it's worth: I bridged MM archives to IMAP once using UW imapd and some scripts for proxying Mailman membership data into something that imapd could use for access control. When I presented it to this list it was generally identified as something nobody (w|sh)ould ever want. I don't care particularly about the details of what happened in 2002 or whenever it was, but I conclude that the much bigger problem than building this ship is choosing a destination. It needs to start with clear goals, because the wrong implementation is easy. -- David Champion • dgc(a)uchicago.edu • IT Services • University of Chicago

1 0

Re: [Mailman-Developers] RELEASED: GNU Mailman 3.0 beta 1 and Postorius 1.0 alpha 1
by Florian Fuchs 29 Mar '12

29 Mar '12

Hi Odhiambo, Am 29.03.12 18:12, schrieb Odhiambo Washington: > My issue here is that I am not running Django 1.4 - unless I am missing > something. I am not sure where it is coming from. > I am on FreeBSD and I installed django-1.3.1 from the ports. A check on my > installed apps shows just that. There is a port for py-django-devel but I > did not install that at all: > > See this > > [root@jaribu] /usr/ports/www/py-django-devel# make fetch > > ===> py27-django-devel-17269,1 conflicts with installed package(s): > py27-django-1.3.1 > > They install files into the same place. > You may want to stop build with Ctrl + C. > ===> License check disabled, port has not defined LICENSE > => Django-r17269.tar.xz doesn't seem to exist in > /usr/ports/distfiles/python. > => Attempting to fetch > http://people.cs.nctu.edu.tw/~lwhsu/ports/distfiles/Django-r17269.tar.xz > Django-r17269.tar.xz 85% of 4135 kB 135 kBps > 00m04s > > That shows I have django-1.3.1 installed. > > I went as far as this: > `mv /usr/local/lib/python2.7/site-packages/django > /usr/local/lib/python2.7/site-packages/_django-1.4` > Then reinstalled django-1.3.1, but I am pretty sure that the above action > wasn't necessary. Generally, *if* you have multiple versions installed and want to know which version is actually imported, cd to postorius/dev_setup, open up a Python shell and try the following: >>> import django >>> django.__file__ # prints the package path >>> django.VERSION # prints the django version > > So I wiped put everything - python, django, mailman, mailman.client, > py-sqlite3 - and reinstalled. Mailman gave me enough grief > with bin/test and I had to wipe and reinstall several times. Perhaps it > wasn't getting all the files during bin/build. > > Now things look promising, but I still get errors about some templates I > need to create - 500.html, 400.html.... > > > Traceback (most recent call last): > > File "/usr/local/lib/python2.7/site-packages/django/core/servers/basehttp.py", > line 283, in run > self.result = application(self.environ, self.start_response) > > File "/usr/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", > line 272, in __call__ > response = self.get_response(request) > > File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", > line 169, in get_response > response = self.handle_uncaught_exception(request, resolver, sys.exc_info()) > > File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", > line 218, in handle_uncaught_exception > return callback(request, **param_dict) > > File "/usr/local/lib/python2.7/site-packages/django/utils/decorators.py", > line 93, in _wrapped_view > response = view_func(request, *args, **kwargs) > > File "/usr/local/lib/python2.7/site-packages/django/views/defaults.py", > line 30, in server_error > t = loader.get_template(template_name) # You need to create a > 500.html template. > > File "/usr/local/lib/python2.7/site-packages/django/template/loader.py", > line 157, in get_template > template, origin = find_template(template_name) > > File "/usr/local/lib/python2.7/site-packages/django/template/loader.py", > line 138, in find_template > raise TemplateDoesNotExist(name) > > TemplateDoesNotExist: 500.html Django tries to find a 500.html file to display a standard error page. It cannot find one and raises TemplateDoesNotExist. But the more important question is: What's the actual cause for the 500 internal server error? IIRC Django only looks for a 500.html in non-DEBUG mode, so my guess is that your DEBUG setting in dev_setup/settings.py is set to False. If that is the case a good next step would be to set it to True and have a look at the traceback again - it should be much more detailed than the one you're seeing right now and (hopefully) give us a clue what's actullay going wrong. If you like paste the traceback to some of the various pasters out there (like paste.ubuntu.com) and post the link to it here - it's much easier to read then. Florian > > > > How do I create these and where do I place them? Where is the guide?? > > > > > > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind… > > Security Policy: http://wiki.list.org/x/QIA9

1 0