Mailman-Developers November 2017

mailman-developers@python.org

11 participants
10 discussions

3.1.x REST API enhancements around preferred email address

by Scott Koranda

Hello, I am interested in contributing enhancements around preferred address to the REST API for version 3.1.x. To prove out my use case(s) and familiarize myself with the code I have made changes already. Here are some example curl command line invocations and output to illustrate: 1) Create an address for an existing user and mark that address as the user's preferred address: curl -u restadmin:restpass \ -X POST \ -d email=skoranda(a)example.nil \ -d preferred=1 \ http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8/addres… This returns a '201 Created'. 2) Return the preferred address, if set, for a user: curl -u restadmin:restpass \ -X GET \ http://mystestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8 | python -m json.tool { "created_on": "2017-11-11T22:20:45.950949", "display_name": "Scott Koranda", "http_etag": "\"26bc0d9f21eb145248884aedae4c1ffd00b608d2\"", "is_server_owner": false, "password": "$6$rounds=656000$gWGfwpajfZ7rVn5O$Sl559B2TgtpJWXA2i67G5ukjzkV6iTp4NgP.6FJpMFMUTDdDXULAmwdN8YW92w87EdctgqFqAUkUqS6.EOTCz/", "preferred_address": "skoranda(a)example.nil", "self_link": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8", "user_id": "e57ecfd0c0c74a319ac958b18882a6d8" } 3) Update/patch a user to set a preferred address: curl -u restadmin:restpass \ -X PATCH \ -d preferred_address=skoranda(a)example.nil \ http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8 This returns a '204 No Content'. 4) Get all addresses for a user with the preferred address (if set) having the attribute 'preferred' set to 'true': curl -u restadmin:restpass \ -X GET \ http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8/addres… | python -m json.tool { "entries": [ { "email": "skoranda(a)example.nil", "http_etag": "\"492824d150f9f11e32d530d3cad0f76422bddb48\"", "original_email": "skoranda01(a)example.nil", "preferred": true, "registered_on": "2017-11-13T22:35:17.232180", "self_link": "http://mytestbed.com:8001/3.1/addresses/skoranda@example.nil", "user": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8", "verified_on": "2017-11-20T16:48:51.738145" }, { "email": "skoranda01(a)example.nil", "http_etag": "\"f0f28c1c6a925a595269d3fe21ef7fc1515d7670\"", "original_email": "skoranda01(a)example.nil", "registered_on": "2017-11-20T16:46:20.941262", "self_link": "http://mytestbed.com:8001/3.1/addresses/skoranda01@example.nil", "user": "http://mytestbed.com:8001/3.1/users/e57ecfd0c0c74a319ac958b18882a6d8", "verified_on": "2017-11-20T16:46:20.941595" }, ], "http_etag": "\"89c46bd312c6f1a344e6e6180aed6f11b62e6048\"", "start": 0, "total_size": 2 } Before I submit a pull request I would be grateful for any comments or feedback on the functionality as illustrated above, or any other comments or feedback. I will also begin the "copyright assignment" paperwork as explained at https://wiki.list.org/DEV/Home Thank you for your consideration. Sincerely, Scott Koranda

3 years, 3 months

ARC PR for Mailman 3

by Gene Shuman

Hello all, Over the past few months I've developed an patch for Mailman 3, <https://gitlab.com/mailman/mailman/merge_requests/252> giving it ARC <https://tools.ietf.org/html/draft-ietf-dmarc-arc-protocol-09> email authentication capabilities. The work is (99%)completed, and it just needs somebody to do a final review of the ARC functionality itself so I can implement any requested changes. I had been working with Barry Warsaw, who had done a preliminary stylistic review, and subsequently, I've eliminated all raised issues. The branch is rebased as of last week, and passes all of qa, except one instance, which is something I'd like to discuss with somebody.(It fails a style test for line lengths in a test file which contains a bunch of cryptographic hashes) . Aside from that all qa pipelines pass. Also, there is a decent amount of included documentation, but it could still use a bit more work. I'm planing on doing right before the branch is merged, and maybe with some additional guidance of the community. So, in communicating with Barry, it seems he doesn't have the time to continue working on this with me. So, I'm reaching out to see if there is anybody with the interest and availability to review the PR. It's not particularly complex, and just introduces 2 new handlers into the pipeline. Thanks & Regards, =Gene Shuman PS. The PR uses a library dkimpy with code that I wrote which passes the ARC reference test suite(which I also wrote) 100%. So all of the cryptographic signing & validating in the PR validates against the current rfc of the protocol. So the crypto can be largely ignored. However you can easily test this functionality by sending ARC signed messages to say gmail, which does ARC validation, & looking at the headers.

3 years, 6 months

Time Stand Still

by Barry Warsaw

Somewhen in the dark recesses of intarweb history, I found myself as the project leader for both Jython (née JPython) and GNU Mailman. I'd been involved with Jython since it was invented by Jim Hugunin around the time he came to work with us at Pythonlabs. I'd been contributing to Mailman since we inherited John Viega's Python-based Dave Matthews Band list server, and put it to use replacing python.org's Majordomo installation. I'd enjoyed both projects, but knew I could not lead both, so I had to make a choice. I chose to turn over Jython to a team that's done a much better job over the years than I ever could have. Something about email, and especially the communication and collaboration patterns that it facilitates, really fascinated me. I know, I know, but we all have our lapses of sanity. Mine has lasted almost 20 years, a bit more than "momentary" perhaps. I've rarely gotten paid to work on Mailman, but it did provide me some great opportunities. Most notably it led to my 10 year stint at Canonical. I was originally hired on there to integrate mailing lists with Launchpad, and Mailman was the obvious choice. I learned a ton doing that project, and working within the constraints of integrating the two Python-based systems, especially since Launchpad was originally not free software and Mailman was GPL'd. Later, the Zope-based Launchpad source code was released under the AGPL, making much of the monkeypatching unnecessary, but by then the system was solid and reliable, and you don't fix what's not broken. Except, I guess I did. I took a lot of the lessons from that work, along with a good hard look at all the problems with Mailman 2, and began to break another cardinal rule of software development: second system syndrome. The result is Mailman 3. It took forever, and we're still not at complete feature parity with Mailman 2, but at least it's Real Enough to be used at many Real Sites, including python.org and lists.fedoraprojects.org. It would be ridiculous for me to take significant credit for this. I have to acknowledge the amazing user community -- you! -- for all the support, patches, suggestions, feedback, patience, criticism, donations, and contributions that you've given to the project, and to me personally over the years. And my deepest gratitude goes to all the core developers that have stayed or come and gone, but most especially the current Cabal: Abhilash Raj, Aurelien Bompard, Florian Fuchs, Mark Sapiro, Stephen J. Turnbull, Terri Oda. You should know that each and every one of them is truly awesome, both in what they contribute technically, and in their amazing friendships. Mailman is infinitely better because of their involvement, and I've loved spending time with them over the years at the Pycon sprints, making releases and sharing teas and meals. My blog is called We Fear Change, and that's humorously taken from a 90's bit in Mike Myer's excellent Wayne's World movie (a phrase actually uttered by the brilliant Dana Carvey as Garth). The irony of course is that while we all may fear change, it's the one constant thing we can count on. And in fact, we *require* change to thrive, because if you aren't changing, you aren't alive. Time, and being engaged with life's vagaries, means there's no alternative to change; it must be embraced. And so, with a vague reference to the many (good!) changes in my personal and professional life, I'm announcing that I'm stepping down from the project leadership role of GNU Mailman, effective... nowish! And it's with unanimous agreement among the GNU Mailman Steering Committee (a.k.a. the Mailman Cabal), that we are announcing Abhilash Raj as the new project leader. If you don't recognize Abhilash's name, you probably aren't paying attention, at least to Mailman 3. Abhilash came to us in 2013 as a Google Summer of Code student, and he's become one of the project's most valuable contributors. His list of accomplishments is long, and it includes everything from redesigning the website, to integrating CI with our GitLab build system, porting our code to the SQLAlchemy ORM, adding MySQL support, revving up adoption through his Docker images, along with his great coding work on Core, Postorius, HyperKitty, and mailmanclient. This transition is good for the project too. Email, its defining protocols and standards, and its role in our daily lives, has changed profoundly since the early days of Mailman. A fresh perspective and enthusiasm will help keep Mailman relevant to the changing ways we -- especially the FLOSS and tech communities -- communicate. Please join me in supporting Abhilash in every way possible as he takes over in this new role as project leader. I'll be here when and if needed, even as I create space in my "spare" time for... Something Else. I look forward to the vision that Abhilash will bring to the project, and I know that he will do a great job. To me, Mailman has always been about collaboration, and the best way for it to succeed is for you to continue to contribute your insights, experiences, opinions, and skills with positive intention. -Barry https://www.wefearchange.org/

3 years, 6 months

Mailman 2.1.25 released

by Mark Sapiro

I am pleased to announce the release of Mailman 2.1.25. Python 2.4 is the minimum supported, but Python 2.7 is strongly recommended. This is a routine bug fix release with a minor new feature and some accessibility improvements for screen readers. See the attached README.txt for details. Mailman is free software for managing email mailing lists and e-newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites. For more information, please see our web site at one of: http://www.list.org https://www.gnu.org/software/mailman http://mailman.sourceforge.net/ https://mirror.list.org/ Mailman 2.1.25 can be downloaded from https://launchpad.net/mailman/2.1/ https://ftp.gnu.org/gnu/mailman/ https://sourceforge.net/projects/mailman/ -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

3 years, 6 months

Mailman Core 3.1.1 and Postorius 1.1.1 released

by Abhilash Raj

Hello Everyone, I am pleased to announce the release of Mailman Core 3.1.1 and Postorius 1.1.1. Python 2.7 is the only supported version for Postorius 1.1.1 and Python 3.5-3.6 are supported for Mailman Core 3.1.1. Both of these are bugfix releases and include some improvements. For a full list of changes, please see [1] for Core and [2] for Postorius. [1]: http://mailman.rtfd.io/en/release-3.1/src/mailman/docs/NEWS.html [2]: http://docs.mailman3.org/projects/postorius/en/latest/news.html For more information about GNU Mailman, please see our wesbite at one of the following: http://www.list.org https://www.gnu.org/software/mailman http://mailman.sourceforge.net/ https://mirror.list.org/ Documentation for Mailman 3 is available at: http://docs.mailman3.org/en/latest/ Both of these can be downloaded from PyPI. https://pypi.org/project/mailman/ https://pypi.org/project/postorius/ thanks, Abhilash Raj

3 years, 7 months

Re: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3

by Abhilash Raj

Update on this (sorry for the top post), So we are going to move forward with Python 3 only and will be dropping Python 2 support in next major release for each components. I am (slowly) working through the test suite to improve the coverage so that nothing breaks when we move to Python 3. There are already some pending merge requests from Simon (thank you!) on both Django-Mailman 3 and Postorius. Once I am done with tests, I will dig in to review them both and co-ordinate with Simon to land them. thanks, Abhilash On Sun, 2017-10-01 at 14:12 -0700, Abhilash Raj wrote: > Hi All, > > Mailman uses Django web framework for the web based frontends, > Postorius > - The Official UI, and Hyperkitty - The official Archiver. They are > both > Django "apps" which means that they can be plugged into any other > existing Django "project" (aka Django "installation") to work > alongside > other apps that people might be running. > > Currently, both the Django apps we have are Python 2 only, we have > talked about moving to Python 3 but we decided we want it to be > bilingual (support both 2 & 3). The reason we decided that was > because > if people would want to embed Postorius & Hyperkitty in their > installations, they need to be able to run it under whatever python > versions they are using. > > I want to revisit this assumption for being bilingual. Currently, > there > is no supported version of Django which doesn't support Python 3. > Starting from v2.0, set to release in December 2017, Django is going > to > drop Python2 support. Now, that doesn't mean no one can run Django > under > Python2, 1.11 (LTS version) supports Python2 and will be supported > probably till Python 2 is supported (April 2020 according to [1]). > > I believe that our (limited) development efforts would be best > utilized > if we just drop the support for Python 2 in Postorius & Hyperkitty > instead of trying to be bilingual. Any day one of our dependencies > may > decide to do the same, and we would have to then use Python 3 anyway. > Also, dropping Python 2 support doesn't seem like a lot of pain for > anyone, you just need another instance of Django running, which is > not > *that* hard using uwsgi (in Emperor mode). I believe most of our > dependencies should support Python 3, or should have a good enough > replacement if it doesn't. > > > Thoughts? > > [1]: https://www.djangoproject.com/download/ >

3 years, 7 months

Rolling releases for Container Images and Funding Campaign for Mailman

by Simon Hanna

>>> These images are built using git-heads *only* if they are passing our >>> test suite and are re-generated weekly. You should be aware that while >>> all these components are tested with their individual test suites, their >>> combination might sometimes not be stable. This will get you >>> updates/bug-fixes much faster :) >> This contradicts what you said in my merge request for Postorius >> https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756 >> >> You either use released versions, or make sure that the master branches >> are always compatible. You can't have both with the current development >> model. > Not exactly, I still believe all our git-heads should be compatible with > released (i.e. stable) versions of dependencies, regardless of the fact > that we maintain those dependencies. > > Also, there is at least one test per-project which tests with git-heads > of dependencies we maintain, so that we know if we make any incompatible > changes. > > I am not sure what in the current development model prevents us > to do both? > > The story for container images is different than testing with git-heads. > We need better integration testing for the entire suite to be stable in > conjunction, rather than independently. In fact, containers might be the > perfect way to actually do the integration testing. Scenario: Core changes something about rest that is backwards incompatible. The change is commited to master. Since your containers use the master branches, all future builds will fail until mailmanclient and postorius/hyperkitty. If you want to always have Postorius (and others) compatible with the latest release of master, development for Postorius will be a Pain, because you can't use the master branch of mailman for testing and quick changes. Also your containers and all integration tests that can be done with them will always fail. Until you fix Postorius and the other clients... So why not just have the master branches always compatible and make life easier for developers. I don't see a good reason why you want to do it the other way round... Any bug in core that affects Postorius will also just remain there until you release a new version of core. If you really really want to have it your way, there have to be more frequent releases. >>> - Improve the container images to work with new micro-services >>> architecture, >>> to achieve scaling and redundancy in services. >> The current bottleneck is the rest api from core. IMO bulk actions, >> filtering and sorting should be added there before trying to have >> multiple postorius/hyperkitty instances serving the same core... > I didn't say they will be served from the same core ;-) > > Yes, the Core's API can be further improved to be more efficient. But > that > doesn't prevent us from scaling. I understand there are issues around > multiple > instances of containers, but I have ideas to get around most of them. I just don't want that people that have issues with big installations get the answer: We know of that issue, please deploy Mailman on X containers/machines as a workaround

3 years, 7 months

Re: [Mailman-Developers] Rolling releases for Container Images and Funding Campaign for Mailman

by Simon Hanna

On 11/08/2017 12:03 AM, Abhilash Raj wrote: > New images are available on quay.io and, moving forward, the rest of > the image builds will also be moved to Quay[4][5]. Since docker deployment is currently the recommended install, with manual installs using the master branches for "experts" I wonder why you are using Quay. It looks like it's a non-free system. Why use Quay but refuse to use a non-free translation system like transifex? > These images are built using git-heads *only* if they are passing our > test suite and are re-generated weekly. You should be aware that while > all these components are tested with their individual test suites, their > combination might sometimes not be stable. This will get you > updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756 You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model. > If this campaign succeeds, here is a road map of what I intend to get > done: Sounds great, what is the limit where the campaign succeeds? What will happen to the funds if it doesn't? > - Add Admin Dashboard project from GSoC 2014 (maybe?) Didn't find anything using google. What is that project about? > - Add better testing of container images and provide deployment > instructions for Kubernetes & Docker Swarm To be honest, I don't think we should all in on containers. Yes they are nice, but I guess most people would prefer distro packages. Even if people want containers, I for instance would prefer plain systemd-nspawn > - Improve the container images to work with new micro-services > architecture, > to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core...

3 years, 7 months

Rolling releases for Container Images and Funding Campaign for Mailman

by Abhilash Raj

Hi Everyone, As you all know I have been working on container images for Mailman 3. We now have a new "rolling" tag for both mailman-core and mailman-web images. These images have latest source installed for every Mailman component. You can find more information about them on the website [3]. New images are available on quay.io and, moving forward, the rest of the image builds will also be moved to Quay[4][5]. These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) As most of you already know, Mailman 3 is the new and improved version with extra features, better security and much better architecture. We released Mailman Suite 3.0 in April 2015 and have come a long ways since then. Mailman Suite 3.1, release May 2016, was aimed to provide feature-parity with Mailman 2.x series and we think we _almost_ hit that goal. Apart from no monthly password reminders, Mailman 3 has a much better Administrator/User interface, REST API for scripting, a really awesome archiver, support for multiple domains, support for external plugins, support for SSO/social login and so much more! I love working on Mailman and would enjoy being able to do so full time for next 6-8 weeks. Mailman 3 is not very far from becoming the default version everyone would use, but it still needs some work to get there. I need help from you, the users of Mailman, to get us there. If you or your organization would like to move to (or, already moved to) Mailman 3, I urge you to donate[1] to us. There are options to donate using Credit Card, Paypal, Bitcoin, Wire Transfer (of any currency), Check and money order. If this campaign succeeds, here is a road map of what I intend to get done: - Move Django apps(UI/Archiver) to Python 3 (or bilingual) - Fork `mailman import` command to provide an upgrade path to Mailman 3.x from Mailman 2.x - Fix MySQL compatibility in Core - Changes in Postorius: - Add support for missing options that are already exposed in Core’s API - e.g. Support for setting templates - Find the commonly used options that are not exposed in Core, add them to Core's API and add to Postorius - Add Admin Dashboard project from GSoC 2014 (maybe?) - Add better testing of container images and provide deployment instructions for Kubernetes & Docker Swarm - Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. - Administrator/User documentation for Postorius & Mailman - (optional) Fork [mmcli](https://github.com/rajeevs1992/mailmancli) project from Rajeev, fix if there is anything missing and add it as an additional command line tool to work with Mailman Core. Maybe pull it under Mailman umbrella. Except for these, if there is something more important that is preventing the adoption of Mailman 3 from your end, we can discuss them. I'd like to mention that I have been working on Mailman 3 for quite some time now and I intend to implement every single item on the list. You donations would help it get done much sooner, hopefully in time for 3.2 release schedule (at PyCon US 2018). You can follow the progress of this campaign here[2]. [1]: https://my.fsf.org/civicrm/contribute/transact?reset=1&id=22 [2]: https://wiki.list.org/x/17892025 [3]: https://asynchronous.in/docker-mailman/#rolling-releases [4]: https://quay.io/repository/maxking/mailman-web [5]: https://quay.io/repository/maxking/mailman-core -- Abhilash Raj maxking(a)asynchronous.in

3 years, 7 months

Use of the public suffix list

by Alessandro Vesely

Hi all, I noticed (from a DMARC mitigation utility that Lindsay extracted) that Mailman features its own approach to using the PSL. Of course, development must go on, and sometimes it is a waste of time trying to make a super-duper scaffolding for a job that can be carried out complying to the KISS principle. At any rate, what is the future of DMARC lookups in Mailman? * The specs say that "DMARC should be amended to use [a method better than PSL] as soon as it is generally available" [1]. I believe that sentence refers to RDAP, which was released more or less at the same time (March 2015) [2]. [1] https://tools.ietf.org/html/rfc7489#appendix-A.6 [2] https://datatracker.ietf.org/wg/weirds/documents/ * There are various Python packages for domain name splitting. They obviously use the PSL, but supposedly would transparently switch to a better method in case. If Mailman used one such package, a practical advantage for users would be to update the PSL in only one place, if they happened to use the same dependency. I found six packages. tldextract [3] is the only one of them which caches a JSON object rather than the original textual representation of the list. It uses a frozenset. tld [4] and publicsuffixlist [5] also build a set. publicsuffix[6] and publicsuffix2 [7] build lists of nested dictionaries from all the labels. dnspy [8] builds a dictionary of FQDNs, somewhat like Mailman. How does the time to build the structure compare with the time taken by DSN queries? [3] https://pypi.python.org/pypi/tldextract [4] https://pypi.python.org/pypi/tld [5] https://pypi.python.org/pypi/publicsuffixlist [6] https://pypi.python.org/pypi/publicsuffix [7] https://pypi.python.org/pypi/publicsuffix2 [8] https://pypi.python.org/pypi/dnspy * Debian distributes a publicsuffix package which brings a textual version of the list. Since stretch, it also brings a "dafsa" version. Nowadays, most C implementations (Firefox, Chromium) use dafsa. They build the structure using offsets rather than pointers, so that the blob can be defined in a source file as a literal static array of chars, in order to minimize loading time. That strategy works well as long as the relevant package is upgraded more frequently than the PSL. Otherwise, as for libpsl, one ends up using obsolete data. Surprisingly, the publisuffix package itself is not upgraded as frequently as the PSL. This bug [9] is what prompted me to write this message. I guess you, as Mailman developers, have pondered this subject and I'd be interested to know what you think. [9] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879008 TIA for any reply Ale --

3 years, 7 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

Mailman-Developers November 2017