Mailman 3 March 2015 - Mailman-Developers

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by Aurelien Bompard March 26, 2015

March 26, 2015

> In my proposal I suggested using any of several asynchronous job queue > libraries, such as Celery or Huey. These all use redis as a back-end. > Because I have no experience with asynchronous job queues, I'm not sure if > this is too much baggage for our purposes. Maybe we just don't want the > extra dependencies. Yeah, we don't want to add another database or an AMQP server just for that. We must keep it simple for admins to deploy. > Regarding cron jobs, there's also django-background-task which is a simple > django addon that might do what we need. Again, if we don't want/need the > extra dependency, rolling our own cron job should be fairly > straight-forward. I'm already using the "jobs" infrastructure provided by the django-extensions package: http://django-extensions.readthedocs.org/en/latest/jobs_scheduling.html I did consider django-background-task but django-extensions seemed like a better fit, because django-background-task seems written for delayed tasks, not periodic tasks (well, a task could call itself again when done, but it seems like a hack). I'm not opposed to switching to django-background-task if we use the "delayed job" feature or if we need the extra flexibility of choosing exactly how many seconds apart we want our tasks to run. >> If we choose to pre-build the mbox files, we can't simply have them > served through the webserver, because some lists are private > > Then there is also an authentication step? Yeah, we must use HyperKitty's authentication and check if the user is allowed to see the archive. So the files can't be served by the webserver like static files. > I noticed on the test server > that I can't actually look at any of the mailing lists because they're all > private. If you're looking at lists.stg.fedoraproject.org, it's currently very outdated (still running the Python2-compatible branch of Mailman 3). I have another test server with more current info if you want, but I break it regularly. It's lists-dev.cloud.fedoraproject.org > When we create the mbox file, do we simply note that an attachment existed > (e.g. "Attachment: myattachment.txt") or do we actually put the attachment > in the mbox? AFAIK mbox is a plaintext format, so if the latter is the case > then I'm not exactly sure how this would work... We do put the attachment in the mbox, as a MIME component like in every email. If you choose "view source" when looking at an email with attachments, you'll see how it's done. > Are there going to be any issues handling unicode foreign characters or > with file locks? Right now it looks like we should only have one process > handling the mbox, but is it possible that more than one could be spawned > somehow? No, mbox files are not designed for concurrent writes, so it's better to have a single process write to them. > Another possible "nice-to-have" feature I thought of yesterday is a > download link that scripts can use to get archives (e.g. > "/download?year=x&month=y"). On the other hand, maybe this is just a > security risk that has no actual use case, but I'd still like to have a > second opinion on this. Well, there still is the authentication issue. Aurélien

1 0

Regarding GSoC 2015
by Aneesh Anil March 26, 2015

March 26, 2015

Hello Sir, My name is Aneesh Anil, and I'm very much interested in contributing to Mailman 3 project. But because I don't have any prior experience in any real world projects I am confused as to what to select. But since I have knowledge of javascript/html with python I think I would do better if I started working on : 1) A Dashboard for Admins/Owners/Moderators ,or 2) Subscriber profile page (Also taking in the fact that these are beginner friendly) I also am working on increasing my knowledge of Django. If you would be kind enough to suggest a project which would be better for me to understand and actually grasp the real world coding concepts It will be much appreciated. Thanking You , Aneesh Anil

1 0

Re: [Mailman-Developers] Javascript Client for Mailman
by Stephen J. Turnbull March 26, 2015

March 26, 2015

Ankush Sharma writes: > I cannot comment on this as I am a student just like you. First, thank you very much for commenting in the first place! Students helping each other is one of the most exciting aspects of GSoC. But I think you misunderstand "student" in the context of GSoC. It's an accounting term. "Students" get paid, mentors get tired, er, "T-shirts", that's it! Mentors get T-shirts! Other than that, we're all Mailman developers. You can and should comment on anything where you have an opinion. Some developers have more experience and skills, some have a natural talent for programming or design or building UIs and others don't have any of the above but need a feature or bugfix, and nobody else seems to want to do it. The good reasons for not giving your opinion are (a) you don't have one (a common reason for those with less experience in the project), (b) somebody's already expressed pretty much the same opinion (in cases where values differ, you might even add a "+1" here), and (c) you're waiting for somebody else to comment first for some reason. That's a little bit oversimplified, but it's important. I don't know if it's made explicit in the GSoC handbooks, but if you get a chance to talk to Carol or Cat you'll hear that one of the things they enjoy most is hearing about students who become mentors. That's a core value in GSoC: the kind of professional growth that turns students into teachers. And it happens in just one or two years for many students. New members normally don't get that experience in a company (unless you're running it :-). They also hear stories about how that assumption of responsibility carries over to the new mentor's school and employment opportunities.

2 1

GSoC 15-Proposal for "A Dashboard for Admins/Owners/Moderators"
by Shreyas Lakhe March 26, 2015

March 26, 2015

Hi guys, I have updated my application on melange for "A Dashboard for Admins/Owners/Moderators" project. Reviews on it are welcome. Regards, Shreyas Lakhe

1 0

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by Stephen J. Turnbull March 26, 2015

March 26, 2015

David Udelson writes: > My proposal has been updated. I apologize if I breached mailing-list > ediquette, I'll get the hang of this eventually :) No etiquette problem. Those are my personal views about effective use of the mailing list and Melange, respectively. I suspect from past experience my views reflect those of other mentors and mailman developers pretty well, but if they don't, I'm sure they'll speak up. :-) There's no doubt about updating Melange from your point of view: we mentors are human, we get a ton of mail (Melange pings us on every change to any proposal in our org, and several of us are org admins increasing the mail signficiantly), and mail is easy to drop on the floor unless you deal with it completely immediately. Unfortunately, we do that. OTOH, in theory you proposal document on Melange can always reflect your best efforts in one place, and we won't miss it. ;-)

1 0

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by Stephen J. Turnbull March 26, 2015

March 26, 2015

David Udelson writes: > Thanks for your feedback Aurelien. > > > we'll need something like a task queue and a daemon process or a cron job > > In my proposal I suggested using any of several asynchronous job queue > libraries, such as Celery or Huey. [...] So far, this is good discussion for the list. IMO, you want to link to the archives for this thread in your proposal on Melange. > Additionally, here are some tentative weekly goals I have for the > project. Feedback on the order/plausibility of these would be > awesome! IMHO, this part should just have been directly edited into the proposal on Melange with a remark here that you did it. Most of the people on this list are either uninterested or not competent to comment on the list of goals. Alternatively, do both but it should definitely be in your proposal already.

2 1

Re: [Mailman-Developers] Javascript Client for Mailman
by Stephen J. Turnbull March 26, 2015

March 26, 2015

Ana Badescu writes: > The Javascript client is actually a node.js module, and users will be able > to use it as part of a node.js application which just as Postorious, is > server-side. Oops, that's right, the REST API normally isn't exposed except to localhost. My bad. > However, we can set up proper test cases for each API, in > Javascript as well. There is a large number of testing strategy > modules that offer everything to set up a testing environment, > stubs, mocks. My point is that no, you can't. If you don't have real use cases, you will write tests that don't matter, and you will fail to cover things that some real use case depends on. Often enough, those things can be really basic, and embarrass both your programmer's pride and your schedule. Both strategies, a priori (here's my API definition, make sure it works) and use-case-based (here's my use case, make sure the API can implement it) are necessary.

1 0

Re: [Mailman-Developers] Javascript Client for Mailman
by Florian Fuchs March 26, 2015

March 26, 2015

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 25.03.2015 um 07:51 schrieb Ana Badescu: > Hello, > > While writing my proposal I came across 2 important issues related > to the Javascript Client for the Mailman project that have yet to > be raised: > > 1. What I initially had in mind, was to build a Mailman client in > Javascript that provides the same API and functionality that the > current one in Python does. The Python client API can be fully > mimicked using a number of design patterns even though Javascript > doesn't offer properties capability for example. However, node.js > modules tend to take advantage of Javascript's asynchronous nature > and have adopted sort of a standard practice to achieve it: it's > called error-first callbacks, and most if not all APIs enforce it. > You can read about it here > http://thenodeway.io/posts/understanding-error-first-callbacks/ and > here > https://docs.nodejitsu.com/articles/errors/what-are-the-error-conventi ons Another > *very* nice way to handle asynchronous behaviour in JS (and one the avoids callback hell) is the Promises paradigm. Promises are a native ES6 feature per-spec, but not all platforms support it. Node doesn't, but there's a drop-in polyfill[1] and a number of other libraries you could use. > This boils down if we want to provide a client API that adopts the > node.js conventions and works the same way most modules do or do we > want to offer something almost similar to the current Python client > API. There's a lot more to be said on this matter (for eg: > callbacks being the reason why "callback-hell" exists) but I'd like > to get your input first. The JS client should be able to do the things mailman.client can do, but that doesn't mean its internals need to be anything like mailman.client's. *Some* similarity to the Python version would be nice, but not to the extend that we do something in a way you normaly wouldn't in JS. The intended audience are folks who use JS, so I'd rather not have a client that runs on node but looks like Python. :-) > 2. I'd also like to make part of the project, a node.js application > that uses the Mailman Javascript client and offers all the > functionality Postorious does. Is that a good idea? This doesn't > make the scope of the project too large or unattainable. "All the functionality that Postorious does" sounds like your summer is *a lot* longer than mine is. :-) I definitely don't see that. What you could do though is add an *optional* task to your timeline, building a *very small* and stripped down web ui proof of concept, with only a small handful of features that you can use to demonstrate how to integrate the client. But that shouldn't be worked on before anything else has been finished. Cheers, Florian [1] https://github.com/jakearchibald/es6-promise -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVEsTAAAoJEEceGbPdavl7Vi8H/0+oj9soM2oRDHBYpnPZo4OG NSwobau3vKrmja/OU0lrXOwlYxfYo+WeQnmf17dlvJOk9zYqPCtSLT5emcV3gQjv vWgzBFwG2sXY0MM/rvfSS+CUIzpNPI+kdhOXriQxhkylBu4aDl8XCiII1xmry7u6 5aFzMygFuV6ixyZRUwriB+qbqvxYdQ+eFkW+LyBvk4y0fkjTEK0/9fXnghkc8fuq taYAbCnhpKZtO6hAsIgQoPPtuGDEqPq5VjjIU89JGeaRjxZ/LX/SOHh2thpgXhre JD+3W5FdziCp3qFijeQCVuK/fErEF1nevC5/wTeeW9fY7/ZOT5GhYCDs4i6EjkM= =3uJ7 -----END PGP SIGNATURE-----

3 3

Re: [Mailman-Developers] GSoC 15 - Interested in contributing to Hyperkitty
by David Udelson March 25, 2015

March 25, 2015

Thanks for your feedback Aurelien. > we'll need something like a task queue and a daemon process or a cron job In my proposal I suggested using any of several asynchronous job queue libraries, such as Celery or Huey. These all use redis as a back-end. Because I have no experience with asynchronous job queues, I'm not sure if this is too much baggage for our purposes. Maybe we just don't want the extra dependencies. Regarding cron jobs, there's also django-background-task <https://github.com/lilspikey/django-background-task>, which is a simple django addon that might do what we need. Again, if we don't want/need the extra dependency, rolling our own cron job should be fairly straight-forward. > If we choose to pre-build the mbox files, we can't simply have them served through the webserver, because some lists are private Then there is also an authentication step? I noticed on the test server that I can't actually look at any of the mailing lists because they're all private. So we should be able to use pre-existing code for this step? > with possible attachments, we may be creating hundred of megabytes or maybe gigabytes of data When we create the mbox file, do we simply note that an attachment existed (e.g. "Attachment: myattachment.txt") or do we actually put the attachment in the mbox? AFAIK mbox is a plaintext format, so if the latter is the case then I'm not exactly sure how this would work... Are there going to be any issues handling unicode foreign characters or with file locks? Right now it looks like we should only have one process handling the mbox, but is it possible that more than one could be spawned somehow? Another possible "nice-to-have" feature I thought of yesterday is a download link that scripts can use to get archives (e.g. "/download?year=x&month=y"). On the other hand, maybe this is just a security risk that has no actual use case, but I'd still like to have a second opinion on this. Additionally, here are some tentative weekly goals I have for the project. Feedback on the order/plausibility of these would be awesome! Week 1) Given an email message, the message headers and body are extracted and stored in a local file in mbox format. All unit tests passing. Week 2) Attachments are represented in the mbox file as well. Email addresses are escaped. There are no encoding errors (no boxes or ?s). All unit tests passing. Week 3) Explore options for possible asynchronous queue managers. Weeks 4-5) When a mailing archive is created, a background process (implemented using chosen backend) is attached to it for managing its mbox files. Existing processes are started when the server starts, and the server can efficiently manage all of these (possibly tens/hundreds?) of tasks. All unit tests passing. Week 6) Clean code and tests before midterm review. All unit tests passing. Week 7-8) Each background process unzips two mbox files, one for the entire list and one for the past month, adds any messages that have come in in the past hour (in mbox format) and rezips the archive. All unit tests passing. Week 9-10) Mbox archives are served by hyperkitty upon request. Hyperkitty does not at this point authenticate users. All unit tests passing. Week 11) Hyperkitty authenticates the user before serving the mbox request. If the request is denied, the user is notified via the UI. All unit tests passing. Week 12) Code review and cleaning, final check on unit tests (they should all be passing). Thanks, David On Wed, Mar 25, 2015 at 4:18 AM, Aurelien Bompard <aurelien(a)bompard.org> wrote: > Hey David, here are my thoughs on the challenges: > > > 1) Determine which messages to include in the mbox. > > An entire list archive is clearly one choice, but is there also > > interest in generating mbox files for specific threads, list archieves > > between specific dates, etc.? > > Hmm, depending on the architecture we choose, we may not have a lot of > options. I'd like to see at least "whole-list" and "last 30 days" > archives though, this last one being useful to those who want to use > their mail client and "seed" it with the latest discussion to reply > in-thread. > > > 2) For each message, append plaintext to mbox file. > > Is this the part where we risk "blocking the UI"? Certainly for > > hundreds of thousands of messages, this will be a computationally > intensive > > step, so will this have to be run in a separate thread? > > Yeah, with a lot of messages, and with possible attachments, we may be > creating hundred of megabytes or maybe gigabytes of data. This has to > be done outside of the webserver process, so we'll need something like > a task queue and a daemon process or a cron job. Or we could be > building and appending to the mbox files when new messages arrive, > which would take up more disk space but would be more fluid from a UI > point of view. It would also probably be much more resource-intensive > than a cron job, because the mbox files will be large and should be > gzipped, so it would be better to append a batch of emails than > opening and closing on each incoming email. > I'm leaning towards pre-rendering the mbox files in a regular cron job > and warning the user in the UI that the archive contains all email up > to the last hour, for example. > We can't use the prototype archiver because we need to filter the > messages content and escape email adresses to protect from spam > harvesters, like MM2.1 currently does. > > > 3) Present mbox file to user for download. > > I'm hoping this is a trivial step, but I'm not sure about some of the > > specifics. For example, is Hyperkitty only able to run on apache, or is > the > > choice of web server entirely up to the web admin? How we ultimately > serve > > the file will depend on these details. > > HyperKitty runs on Django, which can be served by whichever > WSGI-compliant server the admin chooses (Apache's mod_wsgi, uWSGI, > gunicorn, etc.). If we choose to pre-build the mbox files, we can't > simply have them served through the webserver, because some lists are > private (only available to subscribers). > > I hope that clearifies a bit. > > Aurélien > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/dru5%40cornell.e… > > Security Policy: http://wiki.list.org/x/QIA9 >

1 0

Re: [Mailman-Developers] Javascript Client for Mailman
by Ankush Sharma March 25, 2015

March 25, 2015

Hey, On Wed, Mar 25, 2015 at 12:21 PM, Ana Badescu <anabee.emacs(a)gmail.com> wrote: > Hello, > > While writing my proposal I came across 2 important issues related to the > Javascript Client for the Mailman project that have yet to be raised: > > 1. What I initially had in mind, was to build a Mailman client in > Javascript that provides the same API and functionality that the current > one in Python does. The Python client API can be fully mimicked using a > number of design patterns even though Javascript doesn't offer properties > capability for example. However, node.js modules tend to take advantage of > Javascript's asynchronous nature and have adopted sort of a standard > practice to achieve it: it's called error-first callbacks, and most if not > all APIs enforce it. You can read about it here > http://thenodeway.io/posts/understanding-error-first-callbacks/ > and here > https://docs.nodejitsu.com/articles/errors/what-are-the-error-conventions > > This boils down if we want to provide a client API that adopts the node.js > conventions and works the same way most modules do or do we want to offer > something almost similar to the current Python client API. There's a lot > more to be said on this matter (for eg: callbacks being the reason why > "callback-hell" exists) but I'd like to get your input first. > Since, you will be working with Node,js ; you cannot get away from callbacks. This JavaScript port is not a simple port :) 2. I'd also like to make part of the project, a node.js application that > uses the Mailman Javascript client and offers all the functionality > Postorious does. Is that a good idea? This doesn't make the scope of the > project too large or unattainable. I cannot comment on this as I am a student just like you. > > > Thank you, > Ana > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/ankush.sharma.ec… > > Security Policy: http://wiki.list.org/x/QIA9 > Thanks, Ankush Sharma

1 0