Mailman 3 March 2015 - Mailman-Developers

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by Aurelien Bompard 25 Mar '15

25 Mar '15

Hey David, here are my thoughs on the challenges: > 1) Determine which messages to include in the mbox. > An entire list archive is clearly one choice, but is there also > interest in generating mbox files for specific threads, list archieves > between specific dates, etc.? Hmm, depending on the architecture we choose, we may not have a lot of options. I'd like to see at least "whole-list" and "last 30 days" archives though, this last one being useful to those who want to use their mail client and "seed" it with the latest discussion to reply in-thread. > 2) For each message, append plaintext to mbox file. > Is this the part where we risk "blocking the UI"? Certainly for > hundreds of thousands of messages, this will be a computationally intensive > step, so will this have to be run in a separate thread? Yeah, with a lot of messages, and with possible attachments, we may be creating hundred of megabytes or maybe gigabytes of data. This has to be done outside of the webserver process, so we'll need something like a task queue and a daemon process or a cron job. Or we could be building and appending to the mbox files when new messages arrive, which would take up more disk space but would be more fluid from a UI point of view. It would also probably be much more resource-intensive than a cron job, because the mbox files will be large and should be gzipped, so it would be better to append a batch of emails than opening and closing on each incoming email. I'm leaning towards pre-rendering the mbox files in a regular cron job and warning the user in the UI that the archive contains all email up to the last hour, for example. We can't use the prototype archiver because we need to filter the messages content and escape email adresses to protect from spam harvesters, like MM2.1 currently does. > 3) Present mbox file to user for download. > I'm hoping this is a trivial step, but I'm not sure about some of the > specifics. For example, is Hyperkitty only able to run on apache, or is the > choice of web server entirely up to the web admin? How we ultimately serve > the file will depend on these details. HyperKitty runs on Django, which can be served by whichever WSGI-compliant server the admin chooses (Apache's mod_wsgi, uWSGI, gunicorn, etc.). If we choose to pre-build the mbox files, we can't simply have them served through the webserver, because some lists are private (only available to subscribers). I hope that clearifies a bit. Aurélien

1 0

Javascript Client for Mailman
by Ana Badescu 25 Mar '15

25 Mar '15

Hello, While writing my proposal I came across 2 important issues related to the Javascript Client for the Mailman project that have yet to be raised: 1. What I initially had in mind, was to build a Mailman client in Javascript that provides the same API and functionality that the current one in Python does. The Python client API can be fully mimicked using a number of design patterns even though Javascript doesn't offer properties capability for example. However, node.js modules tend to take advantage of Javascript's asynchronous nature and have adopted sort of a standard practice to achieve it: it's called error-first callbacks, and most if not all APIs enforce it. You can read about it here http://thenodeway.io/posts/understanding-error-first-callbacks/ and here https://docs.nodejitsu.com/articles/errors/what-are-the-error-conventions This boils down if we want to provide a client API that adopts the node.js conventions and works the same way most modules do or do we want to offer something almost similar to the current Python client API. There's a lot more to be said on this matter (for eg: callbacks being the reason why "callback-hell" exists) but I'd like to get your input first. 2. I'd also like to make part of the project, a node.js application that uses the Mailman Javascript client and offers all the functionality Postorious does. Is that a good idea? This doesn't make the scope of the project too large or unattainable. Thank you, Ana

1 0

Re: [Mailman-Developers] Common use case archiving via configuration
by Andrew Stuart 25 Mar '15

25 Mar '15

Thanks Aurélien Would you mind if I used the Hyperkitty POST code as the basis for an example POST archiver for Mailman? i.e. this: https://github.com/hyperkitty/mailman-hyperkitty/blob/master/mailman_hyperk… I’m happy to rewrite from scratch but it seems to make sense to use what you have. as On 24 Mar 2015, at 12:45 pm, Aurelien Bompard <aurelien(a)bompard.org> wrote: > Aurelien I’d be interested to hear your thoughts on this topic. Well, the implementation of the archiver for HyperKitty does POST to archive emails : https://github.com/hyperkitty/mailman-hyperkitty/blob/master/mailman_hyperk… However, there are a few things specific to HyperKitty: - authentication is done via a shared API key, it was easier than HTTP basic auth because those tokens are not passed to the WSGI application by default (at least not in Apache's mod_wsgi): https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIPassAuth… - when requesting a message URL, the server only replies with the "local" part, the server name is taken from the configuration. The reason is that the client usually talks over localhost, and we want the front-facing server name. Doing something more generic is possible but would be a bit harder, and the current archiving plugin is still rather short. I'm not sure we can design an HTTP API for every need at the moment. Aurélien

4 7

Getting "approval" to go ahead
by Andrew Stuart 25 Mar '15

25 Mar '15

I haven’t really worked on an open source project before. It wouldn’t make sense to come up with an idea, write some code, submit it and have it rejected because it’s not OK with the project owners and doesn’t fit with project goals. So at what point do I know that if I write some code that the concept is solid and will be integrated into the project assuming the quality is there? Do I need explicit approval from someone like Barry or Stephen? Also, is there a development workflow checklist or something I can use to ensure that I am meeting all requirements for submitted code? thanks as

5 7

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by David Udelson 25 Mar '15

25 Mar '15

I have submitted a proposal on Google Melange. Any feedback I could get about my project constraints would be great! Thanks, David On Tue, Mar 24, 2015 at 8:23 AM, Aurelien Bompard <aurelien(a)bompard.org> wrote: > >> I'm interested in contributing to the Hyperkitty archiver. Specifically, > >> it looks like some requested features for Hyperkitty include rss > syndication > >> for entire mailing lists/specific users/specific threads, and the > ability > >> to view entire threads as plaintext and download that plaintext. > > We don't have either feature yet, but I think RSS syndication would be > much too short for a GSOC project. > There's always been demand for a way to download a list archive as an > mbox file, and it's reasonably complicated if you want to avoid > blocking the UI, so this may be a good project similar to the > plaintext threads feature you mention. > > Aurélien > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/dru5%40cornell.e… > > Security Policy: http://wiki.list.org/x/QIA9 >

1 0

Fwd: Re: GSoC 15 - Interested in contributing to Postorius
by y1dothie＠gmail.com 25 Mar '15

25 Mar '15

(Sorry Stephen) ---------- Forwarded message ---------- From: "Akshay Shah" <y1dothie(a)gmail.com> Date: Wed, Mar 25, 2015 at 12:21 AM Subject: Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Postorius To: "Stephen J. Turnbull" <stephen(a)xemacs.org> > I wanted to work on "Feature request: Editable fields in User profile > <https://bugs.launchpad.net/postorius/+bug/1432383>" in the postorius web > ui. Would this be an easy enough issue to pursue? Where do I ask if I have > any questions about it? I have already commented on the page but not reply > yet. > On Tue, Mar 24, 2015 at 8:41 AM, Stephen J. Turnbull <stephen(a)xemacs.org> > wrote: >> Akshay Shah writes: >> >> > My name is Akshay Shah. >> >> >> Pleased to meet you, Akshay. Welcome to Mailman! >> >> > 1. Can the Dashboard and Subscriber profile page be done together >> > within the GSoC timeline? >> >> I think this would not be useful as the users' tasks are quite >> different. It's better to pick one and do a super job on it than to >> do merely satisfactory work on both. >> >> > 2. I have fairly good amount of experience with NodeJs, will my >> > resume be a good enough factor for you to decide or would you >> > rather have me do a pre-project (I would love to do a >> > pre-project)? >> >> Rather than a pre-project we prefer you to do a bugfix. Any bug, easy >> is OK. Feature implementation is also OK. The point is that you >> become familiar with our workflow as much as it is for us to see your >> work output. >> >> https://bugs.launchpad.net/mailman >> >> Use the "advanced search" link with tag = easy (the tag entry field is >> way down at the bottom). Note that most of the "easy" issues seem to >> be "taken" in that somebody's already working on them, check the >> comments to find out who. Most likely most of the issue have *not* >> been checked for "easy" so don't let lack of an "easy" tag stop you >> from working on an interesting issue. If it isn't clear how to >> proceed after a few minutes, you might want to ask here if the issue >> is maybe "too hard" for a GSoC application before continuing (and >> maybe try a different issue in the meantime). >> >> > 3. [NEW IDEA] Introducing search feature in the Web UI. Search >> > lists, members, settings, and/or domains. Would introduction of >> > Elastic Search with NodeJs be a feasible option for the GSoC >> > Timeline? How detailed should I get about introduction of this >> > topic? >> >> I don't see why this feature needs to be so complex. There are a few >> sites like universities and project "forges" that deal with thousands >> of lists and tens of thousands of users, but "thousands" is still easy >> and efficient enough to do with a simple database lookup (and regexp >> matching on the results if desired) through the Django ORM (HyperKitty >> and Postorius) or SQLAlchemy and the REST interface (Mailman core), >> and it will be consistent with the code in the rest of the >> applications. >> >> If you have use cases where a more complex search would be helpful, >> please explain it. >> >> Regards >> >>

1 0

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by David Udelson 24 Mar '15

24 Mar '15

Sent this email to Aurélien's personal email by mistake the first time (sorry Aurélien!). I'll get used to the mailing list evenually.... > There's always been demand for a way to download a list archive as an mbox file This is a feature I am interested in pursuing. From a very (very) high level, the process of creating an mbox file out a list archive should entail: 1) Determine which messages to include in the mbox. An entire list archive is clearly one choice, but is there also interest in generating mbox files for specific threads, list archieves between specific dates, etc.? 2) For each message, append plaintext to mbox file. Is this the part where we risk "blocking the UI"? Certainly for hundreds of thousands of messages, this will be a computationally intensive step, so will this have to be run in a separate thread? 3) Present mbox file to user for download. I'm hoping this is a trivial step, but I'm not sure about some of the specifics. For example, is Hyperkitty only able to run on apache, or is the choice of web server entirely up to the web admin? How we ultimately serve the file will depend on these details. Aurélien, just because I'm new to the codebase, what are the challenges you see in implementing this feature? Whatever details you can give me will help me set my milestones. Thanks, David On Tue, Mar 24, 2015 at 8:23 AM, Aurelien Bompard <aurelien(a)bompard.org> wrote: > >> I'm interested in contributing to the Hyperkitty archiver. Specifically, > >> it looks like some requested features for Hyperkitty include rss > syndication > >> for entire mailing lists/specific users/specific threads, and the > ability > >> to view entire threads as plaintext and download that plaintext. > > We don't have either feature yet, but I think RSS syndication would be > much too short for a GSOC project. > There's always been demand for a way to download a list archive as an > mbox file, and it's reasonably complicated if you want to avoid > blocking the UI, so this may be a good project similar to the > plaintext threads feature you mention. > > Aurélien > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/dru5%40cornell.e… > > Security Policy: http://wiki.list.org/x/QIA9 >

1 0

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by Terri Oda 24 Mar '15

24 Mar '15

On 2015-03-22 10:23 PM, David Udelson wrote: > I'm interested in contributing to the Hyperkitty archiver. Specifically, it > looks like some requested features for Hyperkitty include rss syndication > for entire mailing lists/specific users/specific threads, and the ability > to view entire threads as plaintext and download that plaintext. I have the > following questions regarding these features: I talked with David on IRC, but in case anyone else is wondering: RSS feeds was a project a few years ago: http://www.google-melange.com/gsoc/project/details/google/gsoc2013/joanna/5… I don't know offhand if the code was merged into upstream, but my preference is as always to make use of code we have if we can, so RSS feeds are maybe not going to make for a good hyperkitty project this year. Terri

2 1

Common use case archiving via configuration
by Andrew Stuart 24 Mar '15

24 Mar '15

As I understand it, currently there are a few default archivers including prototype, mhonarc and mailarchive. New archivers can be plugged in by implementing an archiver which complies to the archiver interface. Not hard but still requires coding skills. Specifically if would be good if a Mailman admin could define in the Mailman config file a set of destination email addresses and POST urls that archive messages are sent to. This would meet many common archiving needs via configuration instead of code. It’s a little odd that Mailman basically does this at the moment but only for mail-archive.com I’m proposing this be a bit more generalised and also that a generalised POST archiver be included. I’d also include a filesystem/zip archiver which writes each message to a zipped file on the disk. I’d be happy to implement assuming anyone else sees value in it, ideally with some architectural input from core developers. Thoughts anyone? as

3 5

GSoC 15 - Interested in contributing to Postorius
by Stephen J. Turnbull 24 Mar '15

24 Mar '15

Akshay Shah writes: > My name is Akshay Shah. Pleased to meet you, Akshay. Welcome to Mailman! > 1. Can the Dashboard and Subscriber profile page be done together > within the GSoC timeline? I think this would not be useful as the users' tasks are quite different. It's better to pick one and do a super job on it than to do merely satisfactory work on both. > 2. I have fairly good amount of experience with NodeJs, will my > resume be a good enough factor for you to decide or would you > rather have me do a pre-project (I would love to do a > pre-project)? Rather than a pre-project we prefer you to do a bugfix. Any bug, easy is OK. Feature implementation is also OK. The point is that you become familiar with our workflow as much as it is for us to see your work output. https://bugs.launchpad.net/mailman Use the "advanced search" link with tag = easy (the tag entry field is way down at the bottom). Note that most of the "easy" issues seem to be "taken" in that somebody's already working on them, check the comments to find out who. Most likely most of the issue have *not* been checked for "easy" so don't let lack of an "easy" tag stop you from working on an interesting issue. If it isn't clear how to proceed after a few minutes, you might want to ask here if the issue is maybe "too hard" for a GSoC application before continuing (and maybe try a different issue in the meantime). > 3. [NEW IDEA] Introducing search feature in the Web UI. Search > lists, members, settings, and/or domains. Would introduction of > Elastic Search with NodeJs be a feasible option for the GSoC > Timeline? How detailed should I get about introduction of this > topic? I don't see why this feature needs to be so complex. There are a few sites like universities and project "forges" that deal with thousands of lists and tens of thousands of users, but "thousands" is still easy and efficient enough to do with a simple database lookup (and regexp matching on the results if desired) through the Django ORM (HyperKitty and Postorius) or SQLAlchemy and the REST interface (Mailman core), and it will be consistent with the code in the rest of the applications. If you have use cases where a more complex search would be helpful, please explain it. Regards

1 0