Over the past few months I've developed an patch for Mailman 3,
<https://gitlab.com/mailman/mailman/merge_requests/252> giving it ARC
authentication capabilities. The work is (99%)completed, and it just needs
somebody to do a final review of the ARC functionality itself so I can
implement any requested changes.
I had been working with Barry Warsaw, who had done a preliminary stylistic
review, and subsequently, I've eliminated all raised issues. The branch is
rebased as of last week, and passes all of qa, except one instance, which
is something I'd like to discuss with somebody.(It fails a style test for
line lengths in a test file which contains a bunch of cryptographic hashes)
. Aside from that all qa pipelines pass.
Also, there is a decent amount of included documentation, but it could
still use a bit more work. I'm planing on doing right before the branch is
merged, and maybe with some additional guidance of the community.
So, in communicating with Barry, it seems he doesn't have the time to
continue working on this with me. So, I'm reaching out to see if there is
anybody with the interest and availability to review the PR. It's not
particularly complex, and just introduces 2 new handlers into the
Thanks & Regards,
PS. The PR uses a library dkimpy with code that I wrote which passes the
ARC reference test suite(which I also wrote) 100%. So all of the
cryptographic signing & validating in the PR validates against the current
rfc of the protocol. So the crypto can be largely ignored. However you
can easily test this functionality by sending ARC signed messages to say
gmail, which does ARC validation, & looking at the headers.
Somewhen in the dark recesses of intarweb history, I found myself as the project leader for both Jython (née JPython) and GNU Mailman. I'd been involved with Jython since it was invented by Jim Hugunin around the time he came to work with us at Pythonlabs. I'd been contributing to Mailman since we inherited John Viega's Python-based Dave Matthews Band list server, and put it to use replacing python.org's Majordomo installation.
I'd enjoyed both projects, but knew I could not lead both, so I had to make a choice. I chose to turn over Jython to a team that's done a much better job over the years than I ever could have. Something about email, and especially the communication and collaboration patterns that it facilitates, really fascinated me. I know, I know, but we all have our lapses of sanity. Mine has lasted almost 20 years, a bit more than "momentary" perhaps.
I've rarely gotten paid to work on Mailman, but it did provide me some great opportunities. Most notably it led to my 10 year stint at Canonical. I was originally hired on there to integrate mailing lists with Launchpad, and Mailman was the obvious choice. I learned a ton doing that project, and working within the constraints of integrating the two Python-based systems, especially since Launchpad was originally not free software and Mailman was GPL'd. Later, the Zope-based Launchpad source code was released under the AGPL, making much of the monkeypatching unnecessary, but by then the system was solid and reliable, and you don't fix what's not broken.
Except, I guess I did. I took a lot of the lessons from that work, along with a good hard look at all the problems with Mailman 2, and began to break another cardinal rule of software development: second system syndrome. The result is Mailman 3. It took forever, and we're still not at complete feature parity with Mailman 2, but at least it's Real Enough to be used at many Real Sites, including python.org and lists.fedoraprojects.org.
It would be ridiculous for me to take significant credit for this. I have to acknowledge the amazing user community -- you! -- for all the support, patches, suggestions, feedback, patience, criticism, donations, and contributions that you've given to the project, and to me personally over the years. And my deepest gratitude goes to all the core developers that have stayed or come and gone, but most especially the current Cabal: Abhilash Raj, Aurelien Bompard, Florian Fuchs, Mark Sapiro, Stephen J. Turnbull, Terri Oda. You should know that each and every one of them is truly awesome, both in what they contribute technically, and in their amazing friendships. Mailman is infinitely better because of their involvement, and I've loved spending time with them over the years at the Pycon sprints, making releases and sharing teas and meals.
My blog is called We Fear Change, and that's humorously taken from a 90's bit in Mike Myer's excellent Wayne's World movie (a phrase actually uttered by the brilliant Dana Carvey as Garth). The irony of course is that while we all may fear change, it's the one constant thing we can count on. And in fact, we *require* change to thrive, because if you aren't changing, you aren't alive. Time, and being engaged with life's vagaries, means there's no alternative to change; it must be embraced.
And so, with a vague reference to the many (good!) changes in my personal and professional life, I'm announcing that I'm stepping down from the project leadership role of GNU Mailman, effective... nowish! And it's with unanimous agreement among the GNU Mailman Steering Committee (a.k.a. the Mailman Cabal), that we are announcing Abhilash Raj as the new project leader.
If you don't recognize Abhilash's name, you probably aren't paying attention, at least to Mailman 3. Abhilash came to us in 2013 as a Google Summer of Code student, and he's become one of the project's most valuable contributors. His list of accomplishments is long, and it includes everything from redesigning the website, to integrating CI with our GitLab build system, porting our code to the SQLAlchemy ORM, adding MySQL support, revving up adoption through his Docker images, along with his great coding work on Core, Postorius, HyperKitty, and mailmanclient.
This transition is good for the project too. Email, its defining protocols and standards, and its role in our daily lives, has changed profoundly since the early days of Mailman. A fresh perspective and enthusiasm will help keep Mailman relevant to the changing ways we -- especially the FLOSS and tech communities -- communicate.
Please join me in supporting Abhilash in every way possible as he takes over in this new role as project leader. I'll be here when and if needed, even as I create space in my "spare" time for... Something Else. I look forward to the vision that Abhilash will bring to the project, and I know that he will do a great job. To me, Mailman has always been about collaboration, and the best
way for it to succeed is for you to continue to contribute your insights, experiences, opinions, and skills with positive intention.
Update on this (sorry for the top post),
So we are going to move forward with Python 3 only and will be dropping
Python 2 support in next major release for each components.
I am (slowly) working through the test suite to improve the coverage so
that nothing breaks when we move to Python 3. There are already some
pending merge requests from Simon (thank you!) on both Django-Mailman 3
and Postorius. Once I am done with tests, I will dig in to review them
both and co-ordinate with Simon to land them.
On Sun, 2017-10-01 at 14:12 -0700, Abhilash Raj wrote:
> Hi All,
> Mailman uses Django web framework for the web based frontends,
> - The Official UI, and Hyperkitty - The official Archiver. They are
> Django "apps" which means that they can be plugged into any other
> existing Django "project" (aka Django "installation") to work
> other apps that people might be running.
> Currently, both the Django apps we have are Python 2 only, we have
> talked about moving to Python 3 but we decided we want it to be
> bilingual (support both 2 & 3). The reason we decided that was
> if people would want to embed Postorius & Hyperkitty in their
> installations, they need to be able to run it under whatever python
> versions they are using.
> I want to revisit this assumption for being bilingual. Currently,
> is no supported version of Django which doesn't support Python 3.
> Starting from v2.0, set to release in December 2017, Django is going
> drop Python2 support. Now, that doesn't mean no one can run Django
> Python2, 1.11 (LTS version) supports Python2 and will be supported
> probably till Python 2 is supported (April 2020 according to ).
> I believe that our (limited) development efforts would be best
> if we just drop the support for Python 2 in Postorius & Hyperkitty
> instead of trying to be bilingual. Any day one of our dependencies
> decide to do the same, and we would have to then use Python 3 anyway.
> Also, dropping Python 2 support doesn't seem like a lot of pain for
> anyone, you just need another instance of Django running, which is
> *that* hard using uwsgi (in Emperor mode). I believe most of our
> dependencies should support Python 3, or should have a good enough
> replacement if it doesn't.
> : https://www.djangoproject.com/download/
>>> These images are built using git-heads *only* if they are passing our
>>> test suite and are re-generated weekly. You should be aware that while
>>> all these components are tested with their individual test suites, their
>>> combination might sometimes not be stable. This will get you
>>> updates/bug-fixes much faster :)
>> This contradicts what you said in my merge request for Postorius
>> You either use released versions, or make sure that the master branches
>> are always compatible. You can't have both with the current development
> Not exactly, I still believe all our git-heads should be compatible with
> released (i.e. stable) versions of dependencies, regardless of the fact
> that we maintain those dependencies.
> Also, there is at least one test per-project which tests with git-heads
> of dependencies we maintain, so that we know if we make any incompatible
> I am not sure what in the current development model prevents us
> to do both?
> The story for container images is different than testing with git-heads.
> We need better integration testing for the entire suite to be stable in
> conjunction, rather than independently. In fact, containers might be the
> perfect way to actually do the integration testing.
Scenario: Core changes something about rest that is backwards
incompatible. The change is commited to master. Since your containers
use the master branches, all future builds will fail until mailmanclient
If you want to always have Postorius (and others) compatible with the
latest release of master, development for Postorius will be a Pain,
because you can't use the master branch of mailman for testing and quick
changes. Also your containers and all integration tests that can be done
with them will always fail.
Until you fix Postorius and the other clients... So why not just have
the master branches always compatible and make life easier for developers.
I don't see a good reason why you want to do it the other way round...
Any bug in core that affects Postorius will also just remain there until
you release a new version of core.
If you really really want to have it your way, there have to be more
>>> - Improve the container images to work with new micro-services
>>> to achieve scaling and redundancy in services.
>> The current bottleneck is the rest api from core. IMO bulk actions,
>> filtering and sorting should be added there before trying to have
>> multiple postorius/hyperkitty instances serving the same core...
> I didn't say they will be served from the same core ;-)
> Yes, the Core's API can be further improved to be more efficient. But
> doesn't prevent us from scaling. I understand there are issues around
> instances of containers, but I have ideas to get around most of them.
I just don't want that people that have issues with big installations
get the answer:
We know of that issue, please deploy Mailman on X containers/machines as
On 11/08/2017 12:03 AM, Abhilash Raj wrote:
> New images are available on quay.io and, moving forward, the rest of
> the image builds will also be moved to Quay.
Since docker deployment is currently the recommended install, with
manual installs using the master branches for "experts" I wonder why you
are using Quay.
It looks like it's a non-free system. Why use Quay but refuse to use a
non-free translation system like transifex?
> These images are built using git-heads *only* if they are passing our
> test suite and are re-generated weekly. You should be aware that while
> all these components are tested with their individual test suites, their
> combination might sometimes not be stable. This will get you
> updates/bug-fixes much faster :)
This contradicts what you said in my merge request for Postorius
You either use released versions, or make sure that the master branches
are always compatible. You can't have both with the current development
> If this campaign succeeds, here is a road map of what I intend to get
Sounds great, what is the limit where the campaign succeeds? What will
happen to the funds if it doesn't?
> - Add Admin Dashboard project from GSoC 2014 (maybe?)
Didn't find anything using google. What is that project about?
> - Add better testing of container images and provide deployment
> instructions for Kubernetes & Docker Swarm
To be honest, I don't think we should all in on containers.
Yes they are nice, but I guess most people would prefer distro packages.
Even if people want containers, I for instance would prefer plain
> - Improve the container images to work with new micro-services
> to achieve scaling and redundancy in services.
The current bottleneck is the rest api from core. IMO bulk actions,
filtering and sorting should be added there before trying to have
multiple postorius/hyperkitty instances serving the same core...
As you all know I have been working on container images for Mailman 3.
We now have a new "rolling" tag for both mailman-core and
mailman-web images. These images have latest source installed
for every Mailman component. You can find more information about them
on the website .
New images are available on quay.io and, moving forward, the rest of
the image builds will also be moved to Quay.
These images are built using git-heads *only* if they are passing our
test suite and are re-generated weekly. You should be aware that while
all these components are tested with their individual test suites, their
combination might sometimes not be stable. This will get you
updates/bug-fixes much faster :)
As most of you already know, Mailman 3 is the new and improved version
with extra features, better security and much better architecture. We
released Mailman Suite 3.0 in April 2015 and have come a long ways since
then. Mailman Suite 3.1, release May 2016, was aimed to provide
feature-parity with Mailman 2.x series and we think we _almost_ hit that
Apart from no monthly password reminders, Mailman 3 has a much better
Administrator/User interface, REST API for scripting, a really awesome
archiver, support for multiple domains, support for external plugins,
support for SSO/social login and so much more!
I love working on Mailman and would enjoy being able to do so full time
for next 6-8 weeks. Mailman 3 is not very far from becoming the default
version everyone would use, but it still needs some work to get there. I
need help from you, the users of Mailman, to get us there. If you or
your organization would like to move to (or, already moved to) Mailman
3, I urge you to donate to us.
There are options to donate using Credit Card, Paypal, Bitcoin, Wire
(of any currency), Check and money order.
If this campaign succeeds, here is a road map of what I intend to get
- Move Django apps(UI/Archiver) to Python 3 (or bilingual)
- Fork `mailman import` command to provide an upgrade path to Mailman
3.x from Mailman 2.x
- Fix MySQL compatibility in Core
- Changes in Postorius:
- Add support for missing options that are already exposed in Core’s
- e.g. Support for setting templates
- Find the commonly used options that are not exposed in Core, add
them to Core's API and add to Postorius
- Add Admin Dashboard project from GSoC 2014 (maybe?)
- Add better testing of container images and provide deployment
instructions for Kubernetes & Docker Swarm
- Improve the container images to work with new micro-services
to achieve scaling and redundancy in services.
- Administrator/User documentation for Postorius & Mailman
- (optional) Fork [mmcli](https://github.com/rajeevs1992/mailmancli)
project from Rajeev, fix if there is anything missing and add it as
additional command line tool to work with Mailman Core. Maybe pull it
under Mailman umbrella.
Except for these, if there is something more important that is
preventing the adoption of Mailman 3 from your end, we can discuss them.
I'd like to mention that I have been working on Mailman 3 for quite some
time now and I intend to implement every single item on the list. You
donations would help it get done much sooner, hopefully in time for 3.2
release schedule (at PyCon US 2018).
You can follow the progress of this campaign here.
I noticed (from a DMARC mitigation utility that Lindsay extracted) that Mailman
features its own approach to using the PSL. Of course, development must go on,
and sometimes it is a waste of time trying to make a super-duper scaffolding
for a job that can be carried out complying to the KISS principle. At any
rate, what is the future of DMARC lookups in Mailman?
* The specs say that "DMARC should be amended to use [a method better than PSL]
as soon as it is generally available" . I believe that sentence refers to
RDAP, which was released more or less at the same time (March 2015) .
* There are various Python packages for domain name splitting. They obviously
use the PSL, but supposedly would transparently switch to a better method in
case. If Mailman used one such package, a practical advantage for users would
be to update the PSL in only one place, if they happened to use the same
dependency. I found six packages.
tldextract  is the only one of them which caches a JSON object rather than
the original textual representation of the list. It uses a frozenset. tld 
and publicsuffixlist  also build a set. publicsuffix and publicsuffix2
 build lists of nested dictionaries from all the labels. dnspy  builds a
dictionary of FQDNs, somewhat like Mailman.
How does the time to build the structure compare with the time taken by DSN
* Debian distributes a publicsuffix package which brings a textual version of
the list. Since stretch, it also brings a "dafsa" version. Nowadays, most C
implementations (Firefox, Chromium) use dafsa. They build the structure using
offsets rather than pointers, so that the blob can be defined in a source file
as a literal static array of chars, in order to minimize loading time. That
strategy works well as long as the relevant package is upgraded more frequently
than the PSL. Otherwise, as for libpsl, one ends up using obsolete data.
Surprisingly, the publisuffix package itself is not upgraded as frequently as
the PSL. This bug  is what prompted me to write this message. I guess you,
as Mailman developers, have pondered this subject and I'd be interested to know
what you think.
TIA for any reply