Mailman 3 December 2014 - Mailman-Developers

before next release: disable backscatter in default installation
by Jo Rhett Nov. 7, 2022

Nov. 7, 2022

Hi. There's a fairly simple problem here that needs to be addressed. And it's mostly a documentation/install problem. I'm hoping we can get this resolved before the next release. PROBLEM: Mailman comes out of the box ready to backscatter spam people. Yes, it's easy enough to fix. But because it comes stock this way, and is documented to install this way, most people install it to do this. Those of us who work in abuse departments are tired of hearing "well that's how Mailman works". We also object to having to teach people how to fix their mailman installations because it's not documented in the current manual. This is *exactly* like Sendmail 14 years ago. We didn't accept it then, and Sendmail fixed the problem. RESOLUTION: Mailman default installation should not backscatter in a default configuration. 1. Don't create backscatter aliases for subscribe/unsubscribe/etc by default. Nearly everyone uses web based signup. 2. Discard or hold messages from non-subscribers by default. I would think that it would be perfectly reasonable to have documentation on how to enable the 1980s-style -request / -subscribe etc aliases. However this documentation should have a note that this is against the AUP of nearly every network provider, and enabling it will likely cause them to get listed in various blacklists as a backscatter source. FYI: I know that this goes against the instincts of many old-time mailing list advocates here. But after dealing with a 10k/hour backscatter DoS my tolerance for this problem is understandably limited. Yes, it was a sweet day back in the 1980s. I was running a mailing list server and several UUCP gateways at the time, so I remember them well. But those days are past, and we need to deal with the reality of today. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness

21 168

Adding CSS to mailman archives
by Andrew Daviel Feb. 12, 2015

Feb. 12, 2015

A little while ago (November 2012 probably) I changed a mailing list from Hypermail to Mailman and wanted to keep some of the look-and-feel of the archives, e.g. background colour and fonts. So I patched mailman 2.1.9 in CentOS 5 to include a CSS stylesheet link in each generated HTML page, replacing e.g. <BODY BGCOLOR="#ffffff"> with just <body> and adding e.g. body {background-color: #ffffff ; color: black; } in /pipermail/mailman.css, so that HTML style such as FONT and SIZE is removed from the Python code. (I only have one list on that server and didn't implement per-list style) I also changed a few things e.g. content-type to get zero warnings and errors from the W3C online validator. Now if I want to get DMARC functionality I'll have to jump past the RedHat/CentOS release and build 2.1.18, and re-patch for CSS. I was wondering if there was any interest in incorporating CSS into the official releases. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time)

7 17

How can I run the REST tests against live servers?
by Andrew Stuart Jan. 24, 2015

Jan. 24, 2015

I want to run the REST tests against my live Mailman server with its Postfix mail server and Postgres database, via: nose2 -v -P rest After many (many) hours of digging deep and figuring out how it works, I suspect at this stage that maybe the tests set up their own mock servers? If so, I’m guessing these mock servers are getting in the way of using the live servers and thus preventing the tests from running? Assuming it is correct that the tests are setting up their own mock http smtp ltmp responders, is there a way to turn this behaviour off so the tests will run against my live REST/Postgres/Postfix? thanks Andrew

3 12

RELEASED: GNU Mailman (core) 3.0b5
by Barry Warsaw Jan. 7, 2015

Jan. 7, 2015

Hello friends of Mailman, and Happy New Year! You can roll that stone To the top of the hill Drag your ball and chain behind you Once again, it's time for the traditional "avoid the copyright year bump" release. I'm happy to announce the fifth beta release of Mailman 3.0 core, code named "Carve Away The Stone". We're really quite close now, but this release is a little different. Those of you who follow the proceedings on the mailman-developers mailing list will note that I have ported the core engine to Python 3.4. While it is my intent to release Mailman 3.0 core[*] final as a Python 3 application, some of the details are still be hashed out on the mailing list, and in the code base. For this reason, I am releasing two "flavors" of 3.0b5: * "A" release, which remains on Python 2.7 * "B" release, which is only compatible with Python 3.4 The A and B releases are functionally equivalent. There may be different bugs in them, but they both implement the exact same feature set, with the only difference being the version of Python they are compatible with. Some time early in 2015, I will be merging the Python 3 branch back into trunk, and dropping Python 2 support. I would encourage you to download the "B" release and try it out. If you do try both, please submit bugs for any functional differences you encounter. You can download GNU Mailman 3.0b5 core (both "A" and "B" releases) from the Python Cheeseshop: https://pypi.python.org/pypi/mailman/3.0.0b5 The documentation is available online at: http://pythonhosted.org/mailman Mailman 3.0 is released under the terms of the GNU General Public License version 3 or later. Detailed changes in 3.0b5 are available here: http://tinyurl.com/p9fpgrh Bugs can be reported here: https://bugs.launchpad.net/mailman Special thanks go to Abhilash Raj and Aurélien Bompard, who implemented the conversion of the ORM layer from Storm to SQLAlchemy and Alembic. Thanks also to Kurt Griffiths for his excellent Falcon Framework package, which now replaces restish as our WSGI-compatible REST layer. Both of these changes were critical in allowing me to port the core to Python 3. Enjoy, and Happy New Year. -Barry [*] Postorious and HyperKitty will port to Python 3 on their own time schedule, and may remain Python 2 applications for the final release of the GNU Mailman Suite. mailman.client, the standalone REST client will be a bilingual library, supporting both Python 2 and Python 3.

6 10

Some raw thinking on user level authorization and authentication
by Andrew Stuart Jan. 6, 2015

Jan. 6, 2015

This is as much gathering my thoughts on research. Anyone with questions or comments or ideas welcome. Just raw ideas at this stage. The goal is to build user level REST authentication for Mailman on top of the existing REST API. Mailman user REST authentication/authorization. Use Falcon/Talons wherever appropriate for consistency with Mailman. Authentication - establish the identity of the user Authorization - allow or deny access to specific Mailman resources based on the user identity ################Authentication & authorization process overview: step 1: authentication —- HTTP client provides username and password —- username and password sent to Mailman REST server —- if Mailman REST server authenticates username and password: —- authentication token returned to client step 2: authorization -- examines inbound requests —- if request contains valid authentication information, continue —- examine requested Mailman resource -— send REST request to Mailman REST server to find out if this user is allowed to do this resource request —- if user is allowed access to resource, goto step 3 step 3: action At this stage I have three candidates for what happens next: action 1: —- treat the inbound request as an external proxy authentication subrequest action 2: —- directly proxy the request action 3: -- wsgi drop-through - just let the request pass through to the next level in the wsgi stack ################Possible actions action 1: —- treat the inbound request as an external proxy authentication subrequest In which the inbound HTTP request is first handled by an external proxy server (for example Nginx), which first sends a “subrequest” to our authentication/authorization server. If our authentication/authorization server approves the request, it responses 200 OK to the external proxy server which then directly sends the request to the Mailman REST API ** downside of subrequest based proxying —- depends on external proxy (i.e. Nginx) that suports subrequests based authentication http://nginx.org/en/docs/http/ngx_http_auth_request_module.html —- it would be possible to write the pure Python proxy to user this mechanism, making an external subrequesting authenticating proxy an option rather than requirement ** upside of subrequest based proxying: — once the request is authenticated/authorized, it makes alot of sense for a specialised, high performance proxy to handle the proxying directly with the back end. All proxying edges cases and gotchas are left to the specialised proxy (i.e. Nginx) to handle, and proxy performance is as fast as the external proxy server and the Mailman REST API can talk to each other. action 2: —- directly proxy the request In which the request is then sent on to the Mailman REST API and the response returned to the HTTP client **downside of proxying: —- proxying is a minefield of gotchas and edge cases —- places the authenticating proxy in the path of all requests, certainly slower than raw access to Mailman REST API **upside of proxying: —- can be done as pure Python, no reliance on the behaviour of external proxy servers —- the are probably fewer gotchas when dealing with one know API (i.e. Mailman REST API) versus trying to implement a generalised proxy server action 3: -- wsgi drop-through - just let the request pass through to the next level in the wsgi stack In which the inbound HTTP request comes first through the authorization/authentication wsgi layers then directly to the existing Mailman API via wsgi ** downside of wsgi drop through: —- requires execution of the Mailman REST API as a wsgi application, which is potentially complex and more tightly coupled than desirable and would require implementation into the Mailman code base —- put another way, the auth functions will operate entirely via queries to the Mailman REST API, and such decoupling of auth functions makes for a simpler system

2 1

Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
by Florian Fuchs Jan. 6, 2015

Jan. 6, 2015

Am 26.12.2014 um 16:38 schrieb Aurelien Bompard: > And mailman.client must be ported too, since it does import stuff from > mailman for testing purposes. I think Postorious is safe, but that must be > verified (I believe it imports the testing framework from mailman.client > and thus must be ported too). I think, this is worth another thread. So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. The first solution for that was to start the core (out of process) with a temporary configuration and on a different port before running the tests and tearing it down afterwards. That way mailman.client/Postorius would be tested with a real MM core underneath it. Compatibility with the API was guaranteed, but it was painfully slow and also a little unstable. It's not fun when every typo is punished with a 10 or 20s penalty! That's why I was extremely happy when Aurélien came up with a much better idea a while ago. His solution made use of the webtest package, which made it possible to test the MM3 API wsgi app without the HTTP overhead or the need to start/stop the main MM process. Much faster, more stable, a real delight compared to the situation we had before. Unfortunately, this now means direct "from mailman" imports which will break with incompatible Python versions. Too bad, we didn't think about this (I, for one, was aware of the fact that Barry plans to do a monolingual port at some point in the future. So I could have thought about that earlier. :-/ meh...). Of course, in order to make our tests work with a py3 MM core, we could go back to the slow and no-fun way of testing that we had before. But maybe there's better way... Any ideas? Cheers Flo

4 11

Re: [Mailman-Developers] Python 3
by Barry Warsaw Jan. 6, 2015

Jan. 6, 2015

On Dec 28, 2014, at 01:51 PM, Aurelien Bompard wrote: >> As I mentioned, I think LMTP *could* work, but REST (inside HK) could >> work too. Aurelien, what do you think? > >I'd go with REST, it seems more flexible and we already have nice libraries >for it. REST will be very nice, and I think with something like Falcon, pretty easy to implement. You probably don't need the object-based (i.e. restish-like) traversal stuff, but if so, I'd be happy to split that out into a separately library that HK could use (and keep it bilingual). Keep it simple, but think through the resource tree and leave open some opportunities for expanding the API later, both for use by MM3 and perhaps other scripting applications. Cheers, -Barry

3 2

Re: [Mailman-Developers] Python 3
by Stephen J. Turnbull Jan. 3, 2015

Jan. 3, 2015

Aurelien Bompard writes: > Barry writes: > > One of the advantages of accessing the core through the REST API > > is that it doesn't matter what clients like HyperKitty and > > Postorious are written in. > I'm not exclusively using the REST API though. I'm importing a > couple interfaces, mostly the archiver interface. This was a design mistake, I think. An IArchiver really needs to be in core for two reasons: first, it needs to generate a permalink and attach it to the message as distributed. Second, it needs to associate that permalink with the message so the "real" archiver process will do set that link correctly. For a production-quality archiver, that's *all* it should do. The archiver itself should be a separate process, receiving the message and permalink data by IPC. @Barry: maybe rename IArchiver to IPermalinker? ;-) Remember, Mailman core is going to be distributing Internet mail. Except in the case where 100.00% of the users are on one host, that's going to be the bottleneck on message processing. Archiving simply does not need to be fast. The archiver can implement LMTP even, although that would be overkill if we didn't already have an LMTP server. The simplest approach would be to simply put the Archive-Permalink header in the message and stream it to the archiver which would parse it out. > I'm also using the custom Message class a lot in the tests. Can't avoid that with Python 2, I guess, but using Message will be *so* much less painful with Python 3. > But I think the main problem is the import of mailman's config > object in the class that implements the IArchiver interface. I > don't believe there's another way to get the configuration. If you need that configuration (which, come to think of it, you probably do, at least parts of it), you could have a private protocol for communicating it as metadata (a message header or as metadata in the stream). > And now that I think of it, the archiver interface will be imported > by Mailman core, and will thus run with a Python3 interpreter. As a > result, all of KittyStore must at least be Python3 compatible. In the long run (ie, when nobody who's anybody uses Python 2 at all) I think everybody would be happier if you refactor to keep KittyStore at arm's length from Mailman core.

6 11

Python 3
by Barry Warsaw Dec. 30, 2014

Dec. 30, 2014

One of my top priorities has been to port Mailman 3 (core) to Python 3. This work is now complete, and ready to be merged into trunk. No doubt bugs still lurk, but at least the entire test suite is passing. My intent is to merge this to trunk and do another beta release before the end of the year. If you want to take a look at the branch, try this: $ bzr branch lp:~barry/mailman/py3 The diff is available online via the merge proposal: https://code.launchpad.net/~barry/mailman/py3/+merge/245313 It's a big diff. $ bzr diff . --old ../3.0 | diffstat -s 166 files changed, 1881 insertions(+), 1538 deletions(-) Python 3.4 will be the minimum required version. (It make work with earlier Python 3's but I will not officially support anything earlier.) There are a few things to keep an eye out for, especially if you are currently experimenting with Mailman 3 in production: * A few database columns have changed from LargeBinary to Unicode. I have not yet figured out how to add the Alembic schema migrations to handle this, but I'll do so before the release. * The internal format of the requests table has changed. While the column type is a string, the internal format was a sort of home baked pickle-like format. This turned out to be incompatible with Python 3, due to some assumptions made about the format. I've changed this to using a JSON representation, which is safe and platform independent. This means that any held requests (subscriptions, unsubscriptions, held posts) must be cleared before switching to the Python 3 code. I don't think it's worth trying to migrate this internal format. * The new email API in Python 3 is really wonderful, but I am mostly not yet taking advantage of it. The one area that required a significant rewrite is in the Subject prefix munging code. Some Python 2 behavior is different in Python 3 and the old algorithms had to be rewritten. I've added a lot of tests, but there are probably bugs in the new code, especially when various mixed charsets appear in the Subject header. Please do test, review, grouse, compliment, curse, rejoice, and enjoy. Over the holidays I'll spend some time cleaning a few things up (especially the documentation), then I'll merge this to trunk and do a 3.0b5 release as a Python 3-only application. Mailman 3 will not be bilingual so Python 2 support will be dropped. Cheers, -Barry

4 4

Re: [Mailman-Developers] Python 3
by Pierre-Yves Chibon Dec. 28, 2014

Dec. 28, 2014

On Fri, Dec 26, 2014 at 04:38:08PM +0100, Aurelien Bompard wrote: > [...] then I'll merge this to trunk and do a 3.0b5 release as a > > Python 3-only application. Mailman 3 will not be bilingual so Python 2 > > support will be dropped. > > > > Wow. I am very surprised. > So we went from "Python3 compatibility is not on the blocker list for > Mailman 3.0" to "Mailman 3.0 will only work on Python 3". > That's quite a change. > This means I must now port HyperKitty and Kittystore to Python3, check that > we don't use Py2-only libraries, etc. > And mailman.client must be ported too, since it does import stuff from > mailman for testing purposes. I think Postorious is safe, but that must be > verified (I believe it imports the testing framework from mailman.client > and thus must be ported too). This also means that mailman 3 will pretty much only not work on server-oriented distro (except probably Debian that likely ships a python 3 version, although not being involved in Debian, I do not know for sure). For the record, neither RHEL7 nor SL7 nor CentOS7 currently have a python3 stack... > And I thought the 3.0 release was almost there. I must say this is rather > discouraging. Same for me. Pierre

5 6