On Thu, 2020-02-27 at 09:58 -0800, Mark Sapiro wrote:

On 2/27/20 7:22 AM, Dennis Putnam wrote:

...

I think this may have been addressed but I can't find it. Now that I am munging the from address to mitigate DMARC, recipients can no longer tell who the message is from. What are other folks doing to handle that? Other than having list members add their own signature? Thanks.

The From: header in the munged message contains the sender's display name as in

From: Jane Doe via ListName <listname@example.com>

I've been wondering if we should change that to something like this:

From: Jane Doe (jane.doe@domain.tld) via Listname <listname@example.com>

This would be better for mobile email clients which can't/don't easily display reply-to or other headers.

-Jim P.

On 2/27/2020 1:23 PM, Mark Sapiro wrote:

On 2/27/20 10:05 AM, Jim Popovitch via Mailman-Users wrote:

...

I've been wondering if we should change that to something like this:

From: Jane Doe (jane.doe@domain.tld) via Listname <listname@example.com>

We specifically do not do that because it is said that multiple email addresses in From: trigger spam filters.

From: Jane Doe (jane.doe at domain.tld) via Listname <listname@example.com>

On Thu, 2020-02-27 at 10:56 -0800, Mark Sapiro wrote:

for Mailman 3, but that seems unduly kludgy. There won't be any change in Mailman 2.1 which is only waiting for i18n updates for the final 2.1.30 release which will be the last release from the GNU Mailman project.

Who decides that there will be no more releases of MM2 from the GNU Mailman project?

I've got to be honest, Mailman 3 still looks unstable to me. I get that it's working on python.org where there are people working on it day after day, but surely you realize there are a ton of Mailman2 sites that don't have the time to develop and maintain their own install day after day. Look at the MM3 list, there are people who do nothing but offer full time Mailman hosting and they have problem after problem. And then there's the whole "I don't need a CMS for a MLM" argument. I personally believe there's a lot more life left in MM2 than a few people want to admit.

OK, there's the Python2 EOL issue, but python2 isn't disappearing overnight, certainly not this month or next (as you say the case should be with MM2).

I guess I'm just still a bit shocked to see you rush to abandon something so popular and established.

Personally, I'd like to see the GNU Mailman project have a formal Mailman 2.3 release that supports Python3, I feel that there would be a lot of support for that.

-Jim P.

On 2/27/20 2:44 PM, Mark Sapiro wrote:

If you want to port Mailman 2 to Python 3, you are welcome to do it. I have said before that a much better use of time and resources would be the implementation of a light weight, non-Django web UI for Mailman 3, but I don't see anyone raising a hand to do either.

I am doing that. I have hired a programmer and work beings for a new Mailman 3 UI next week.

-- Please let me know if you need further assistance.

Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com

-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase

EMWD's Community Forums http://discourse.emwd.com/

On 2/27/20 2:44 PM, Mark Sapiro wrote:

I have said before that a much better use of time and resources would be the implementation of a light weight, non-Django web UI for Mailman 3, but I don't see anyone raising a hand to do either.

Let me be more clearer on this.

I have hired a professional PHP developer to begin work on a new admin/forum interface for Mailman 3. The work begins next week. We are only focusing on the admin (Postorius) interface initially. I also am in the process of hiring a front end developer for the new interfaces. I believe in the Mailman project, and as someone who has benefited from offering Mailman hosting services for years, I don't want to see it go away. The problems however are this:

1. Mailman 2 is great. The interface for it is very outdated. This turns people off. UI design has come a long way and people are use to using modern UIs.

2. Mailman 2 does need to be ported to python 3 eventually but Mailman 3 is already there so why spend extra time and resources on doing that? That is a good question and the answer may be to look at the way Mailman 3 development is currently being handled. Also Mark Sapiro clearly wants us to put our support behind Mailman 3 which is good and he has my support with that.

3. Mailman 2 doesn't integrate well with other applications due to no REST api which I think is what modern users want these days. Mailman 3 has a REST api which is great but again I am having issues with the way Mailman 3 is being developed.

4. So lovers/users of Mailman are stuck between a rock and hard place: Mailman 2 or Mailman 3? Which way to go?

For me, Mailman 3 is the way to go but I can no longer wait on the two interfaces of Mailman 3 to be brought to modern standards. Postorius/Hyperkitty came out flawed right from the beginning:

1. Outdated U.I.

2. All of Mailman Core features/functions not being revealed in Postorius. This is something I intend to fix quickly with the new U.I. as soon as I figure out what those features/functions are. Anyone want to provide a list of that to me off-list so I can pass it on to my programmer?

3. The decision to use Django. Maybe great for Python users but not for me and perhaps for others. This is also brings additional confusion. Mailman 3 has THREE interfaces: Postorius, Hyperkitty, and the Django admin interface.

4. Very poor documentation for Mailman 3 and way too many methods of installing it which means all kinds of versions of Mailman 3 are in production today because Mailman versions are dependent upon what method of installation a person chooses: Distro Package, Docker, Source, others?

5. MM3 DMARC handling seems to have improved from reviews I have seen but NO BOUNCE PROCESSING. My goodness. How do list managers keep their mailing lists clean? I know how much a hit to an IP address reputation can be done when a server is sending messages out to invalid email addresses. However I don't think I can fix that right? It is a Mailman core issue? So let's say it does get added to MM Core. How long will it show up in Postorius? Especially since not every feature in MM Core is revealed in Postorius already.

6. Social Media integration via Django is awful.

7. Hyperkitty just does not cut it in appearance and usability when it comes to a modern list forum. I am simply unable to compete with the growing number of applications that are being offered that has a better browser UI for communicating with list members.

I think the highest priority is to get Mailman 3 core up to speed in offering everything that Mailman 2 offers such as bounce processing. Then perhaps a whole new approach to interfacing with Mailman 3 core is in order. That part I have decided to work on because no matter how great MM3 core is, if the interfaces are poor then modern users will move on to something else.

I hope I did not offend anyone here or show disrespect to the hours and hours that have been spent on the Mailman 3 project. That was never my intention and I love Mailman and its community of users.

I would be interested to know if the developers of Mailman 3 are interested in the initiative I have taken to develop new interfaces for Mailman 3 that are more modernized and user-friendly.

-- Please let me know if you need further assistance.

Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com

-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase

EMWD's Community Forums http://discourse.emwd.com/

On 2/27/20 5:30 PM, Mark Sapiro wrote:

I don't know why Mailman 3's DMARC mitigation is considered improved over Mailman 2.1. It's the same. The Settings and Postorius UI for them are more logical than MM 2.1, but they ultimately boil down to the same things.

The latest Mailman core (not yet released but available at <https://gitlab.com/mailman/mailman/-/tree/master> fully implements bounce processing. Prior to this, bounce events were stored in the database but not processed. Now they are. As I said, it's in the latest version of core. The list specific settings: 'bounce_info_stale_after', 'bounce_notify_owner_on_disable', 'bounce_notify_owner_on_removal', 'bounce_score_threshold', 'bounce_you_are_disabled_warnings', 'bounce_you_are_disabled_warnings_interval' and 'process_bounces' are not currently exposed in Postorius, but if Mailman 2.1 lists are imported with import21, they will be set appropriately and they will be in Postorius eventually.

I don't speak from experience in regards to my comment made on DMARC mitigation. It was based on observing comments that have been made.

Bounce processing will still not be available for new users of Mailman which is my big concern. I assume new lists will have to have those settings adjusted via the Mailman shell?

-- Please let me know if you need further assistance.

Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com

-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase

EMWD's Community Forums http://discourse.emwd.com/

On 2/27/20 5:01 PM, Mark Sapiro wrote:

True, until they are exposed in some list admin UI they will need to be set via mailman shell or the REST API, ...

That is IF they need to be changed from the default.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/27/20 5:44 PM, Dave McGuire wrote:

Not to hijack, but is it possible to set the maximum message size by the mailman shell? I've a problem with that in one of my MM3 lists, I really need to set that, but the web interface does not allow me to set it. I get the dreaded "Unknown attribute: max_message_size" error. Could you loan me a clue?

Not only does this not really belong in this thread, it doesn't even belong on this list. <https://lists.mailman3.org/mailman3/lists/mailman-users@mailman3.org/> would be a better place, but since you asked...

Setting Maximum message size works in current Postorius. What version do you have?

Yes you can set it in mailman shell

mailman shell -l list@example.com

The variable 'm' is the list@example.com mailing list

...

...

m.max_message_size = 100 (or whatever you want to set it to) commit()

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/27/20 6:21 PM, Dave McGuire wrote:

...

Setting Maximum message size works in current Postorius. What version do you have?

I'm running 1.1.2; it's from the Ubuntu repo.

Actually, the issue is Mailman core, not Postorius. max_message_size was not exposed in REST before version 3.2.0.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Brian Carpenter writes:

I have hired a professional PHP developer to begin work on a new admin/forum interface for Mailman 3.

Too bad. I really sympathize with your goals but am unlikely to be able to contribute directly to implementation (assuming an eventual open-sourcing). Never learned PHP, not going to do it anytime soon. That's OK, the point of REST is so *you* *can* do that. I can only speak for myself, but we can help to some extent on the Python side of the REST interface.

A word of advice: we, too, talked about "modern forum software and interfaces that users expect", but implementing them well is a lot harder than we expected. I'm not saying it's too hard for your developer, but stay on top of that project! Mail is hard to combine with forums, and it's easy to stray into the weeds.

2. Mailman 2 does need to be ported to python 3 eventually

Sure, but that's still quite a ways away. The main issues I can see would be related to TLS, where old versions of the protocol seem to be deprecated more and more rapidly, but it's probably easier to patch Python 2 for that than to port Mailman 2 to Python 3. Sure, there may be non-TLS CVEs against Python 2, but I really can't see them being as serious as the kinds of issues that Mailman 2 itself, not to mention any site with mail service, would have.

3. The decision to use Django. Maybe great for Python users

This makes no sense to me. If your problem with Django is that it's written in Python, you've got the REST interface. That's as far as our responsibility goes. See "REST is the approach" below.

4. [W]ay too many methods of installing it which means all kinds of versions of Mailman 3 are in production today because Mailman versions are dependent upon what method of installation a person chooses: Distro Package, Docker, Source, others?

That's not a problem we created. Our recommendation is, as it always has been, build from source. And the problem is not going to go away. Users want turnkey packages such as containers, distros can't be stopped from building distro packages. If you open source your interfaces, you'll run into the same issues.

I think the highest priority is to get Mailman 3 core up to speed in offering everything that Mailman 2 offers such as bounce processing.

Agreed. I didn't even know bounce processing wasn't working until this summer (my production lists are all in-house to personal acquaintances to same-university addresses -- if mail isn't flowing to somebody, it's not going to anybody, even Mailman!)

Then perhaps a whole new approach to interfacing with Mailman 3 core is in order.

REST *is* the "Mailman 3 approach" to interfacing. Historically, at the time Mailman 3 core got whipped into shape to start beta testing, we went with Django for Postorius, because it was the "hot" framework of the day, and that's what the developers who volunteered wanted to work with. Of course it had to be a Python framework since we'd be maintaining it. HyperKitty was a little bit different: the Fedora (or Red Hat?) people wanted Mailman 3 for internal reasons, and they contributed a pile of labor, and (AFAIK) independently chose Django and developed the UI based on it (which is why we have two separate Django configs).

*However*, the original idea was that *we* didn't know much about UI development, especially the peripheral features of archiving such as search and access control, and we wanted to encourage third parties to develop their own, or to integrate Mailman lists into their larger platforms which already provided user interfaces.

I hope I did not offend anyone here

The main Postorius devs aren't hanging out here, and we get only a little contact in the summer with the HyperKitty devs since the Fedora support got cut three or four years ago. ;-) If I know Mark he started a little miffed but calmed down quickly since diverse UIs have always been part of the vision.

I would be interested to know if the developers of Mailman 3 are interested in the initiative I have taken to develop new interfaces for Mailman 3

As far as I know this is precisely what we wanted to happen in the first place. We knew that we would have to have a bundlable user/admin interface and an archiver with a web interface, but the original intent was not that they dominate Mailman installations. We should have known better, given the huge popularity of Mailman 2 with Pipermail (oh, Lordy) as the archiver, but hope springs eternal ....

I think further discussion should move to mailman-developers, though, and please introduce your UI developer(s) to us on mailman-developers soonish. Nobody has huge amounts of time to put into work on Mailman right now AFAIK, but I expect we will be willing to cooperate on any Mailman features or fixes your developer needs in the REST interface "in good time".

I'm going to be very busy until March 11, but after that I'll have some time. Ginning up a list of REST endpoints is the kind of thing I'm good at, so maybe I personally could start there.

Steve

On 2/28/20 6:17 AM, Jim Popovitch via Mailman-Users wrote:

On Fri, 2020-02-28 at 19:52 +0900, Stephen J. Turnbull wrote:

...

Brian Carpenter writes:

...

I have hired a professional PHP developer to begin work on a new admin/forum interface for Mailman 3.

Too bad. I really sympathize with your goals but am unlikely to be able to contribute directly to implementation (assuming an eventual open-sourcing). Never learned PHP, not going to do it anytime soon.

Stephen, It's difficult for me to parse your thought process on this. Why do you say "Too bad" about someone wanting to improve something that you admit you want no part of?

Well, Steve channeled me earlier, so I'll return the favor. I think Steve is saying "Too bad" he is only talking about the choice of PHP as a platform rather than Python. We absolutely encourage people to develop alternatives to Postorius and HyperKitty for archiving and web management of Mailman. I think all Steve is saying is he could be more helpful with Python as opposed to PHP.

See this paragraph:

...

That's OK, the point of REST is so *you* *can* do that. I can only speak for myself, but we can help to some extent on the Python side of the REST interface.

Who is this "we", you referenced them a few times in this email.

See the initial paragraph in the Acknowledgements section at <https://www.list.org/>.

I'm fairly confident in saying that Mark has said (repeatedly now) that there will never ever ever ever ever be another Mailman2 release beyond v2.1.30. Stephen, why do you say there will be? Do you have the project authority to make that statement? Who do I beleive?

Actually, I have said there will not be another release from the GNU Mailman project. That does not preclude anyone from forking that project from Launchpad and doing whatever with it.

I personally am not interested in porting Mailman 2 to Python 3. That's already been done. The result, with a real backend database and some extensions such as the concept of "user" that doesn't exist in MM 2, is Mailman 3 core.

What I'm reading between the lines is that nothing but Django was considered for MM3 (by "we") and everything else is inferior and not worth the time. I'd love to be wrong on that.

The web based archiving and list management we distribute are based on Django because that's how the people who developed those things wanted to do it.

The whole point of Mailman 3 is all that stuff is separate from the core engine and communicates with core via a REST API, so there can be lots of different web management UIs. We knew if we released Mailman 3 core only without a web UI for list management and archive access, no one would use it, so we needed those things and the people who were willing to implement them built what we have.

We certainly hoped that there would be people like Brian implementing alternatives and we're glad to see it.

...

Agreed. I didn't even know bounce processing wasn't working until this summer (my production lists are all in-house to personal acquaintances to same-university addresses -- if mail isn't flowing to somebody, it's not going to anybody, even Mailman!)

MM3 has been on python.org for a while, was it not noticed there either?

Of course. We began discussing this almost 3 years ago <https://gitlab.com/mailman/mailman/issues/343#note_31870366>. The implementation was mostly done last year by a GSOC student.

We are a small project. We have very few people working on Mailman on a regular basis, and everyone is a volunteer, no one is paid. If you want things to happen faster, please contribute.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/28/20 10:24 AM, Jim Popovitch via Mailman-Users wrote:

I think that is Brian's and a lot of other people's concern. 3 years to implement something into MM3 that was a core feature in MM2. I realize this next question is going to sound bombastic, I assure you it's not meant that way: What other things are missing or not available presently in MM3 that are taken for granite in MM2?

I think almost nothing is missing from Mailman core. We don't have 'sibling lists' or 'topics', but other than that, I think it's all there.

Quite a few core settings are not exposed in Postorius but we're chipping away at that.

...

We are a small project. We have very few people working on Mailman on a regular basis, and everyone is a volunteer, no one is paid. If you want things to happen faster, please contribute.

~$ grep "Jim Popovitch" ~/devel/Mailman/NEWS | wc -l 10

I don't think that I've been sitting on the fence, in fact I think I've been contributing a lot if you include not just source contributions but also the PPAs. I wouldn't say that I'm a principal developer, but I'm not off in some remote corner unfamiliar with the product and project.

I know that you (Jim P) have contributed a lot to MM 2 and we all and I in particular appreciate that, but I was thinking specifically of MM 3.

Note also that I still spend a lot of time helping people with MM 2 questions. This is time that could otherwise be spent on MM 3. This is a big part of why I've said 2.1.30 will be the last MM 2 release.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Brian Carpenter writes:

On 2/28/20 1:55 PM, Mark Sapiro wrote:

...

Quite a few core settings are not exposed in Postorius but we're chipping away at that.

Is there a list somewhere to see what core settings have not been exposed in Postorius yet?

Implicit in the Postorius UI and the list of REST API endpoints. ;-) Making that explicit is part of the task I proposed for myself.

I don't think that there are a lot.

Steve

On Sat, 2020-02-29 at 16:28 +0900, Stephen J. Turnbull wrote:

Mark Sapiro writes:

...

Well, Steve channeled me earlier, so I'll return the favor.

And did it with extreme precision and accuracy. Sorry if I created any misunderstandings.

The only thing I have to add is that mailman-users@python.org is not going away. Furthermore, I expect that Mark and I, at least, will be here for the foreseeable future. That's because not only are existing Mailman 2 installations not going away any time soon, there's every reason to believe that people will continue installing Mailman 2 for quite some time (even if it might be more convenient for *us* if everybody would switch to Mailman 3 ;-).

Steve

Steve, given your comments above, please expand upon this scenario: If a CSF/CSS is identified in Mailman v2.1.30 in May-2020, what will be done to address it?

-Jim P.

On Sat, 2020-02-29 at 10:46 -0800, Mark Sapiro wrote:

On 2/29/20 7:02 AM, Jim Popovitch via Mailman-Users wrote:

...

If a CSF/CSS is identified in Mailman v2.1.30 in May-2020, what will be done to address it?

I'd say it depend on the details of how serious the vulnerability is, how easy it is to exploit and how hard it is to fix. I am not opposed to Mailman 2.1.30-x security fix releases.

Thank you, it is reassuring to hear you say that.

-Jim P.

On 2/28/20 11:28 PM, Stephen J. Turnbull wrote:

Mark Sapiro writes:

...

Well, Steve channeled me earlier, so I'll return the favor.

And did it with extreme precision and accuracy. Sorry if I created any misunderstandings.

None whatsoever, at least not from me ;)

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

@mailman-users I'm going to get into a lot of design here, so I'm moving the thread to mailman-developers@python.org. Reply-To set; please respect. Brian kept in Reply-To as a courtesy, don't know if he's subscribed over there.

@mailman-developers Brian is planning to develop an alternative web UI for admin and archives. I don't see this as a disincentive to Postorius or HyperKitty development (proposed implementation language is PHP, so we'll still need something we can support that we can bundle), and alternatives have always been part of the vision. I for one plan to cooperate, and hope others feel the same. Thread starts here on mailman-users:

https://www.mail-archive.com/mailman-users@python.org/msg72530.html

@Brian If you aren't subscribed to mailman-developers, and don't want to subscribe,, I'll try to keep you in the loop.

Brian Carpenter writes:

I agree and understand. The forum side is not being considered at this time until we get the admin interface nailed down. Right now I am looking at Discourse as a motivation for developing the forum side. I also think for mailing lists to survive in the future, integrating both approaches to communications is what modern users are looking for.

The Python developers looked at Discourse, I think there's actually a fair amount of activity. "I think" because I don't participate in their forum server, not even entirely sure it's Discourse (checked, it is). I don't miss it at all. I don't think the Discourse users miss mailing lists at all. There seems to be near zero crosstalk, even less crosstalk between Discourse and the lists than there is between the lists and the issue tracker.

I don't know what would happen with better integration. Discourse integration of email, uh, "is poor" IMHO, which in my opinion is an indication that integration is somewhere between very hard and impossible -- the original author of Discourse seems to be a brilliant designer and programmer, with plenty of sympathy for user needs. If he didn't do it well, it's surely not at all easy. A lot of people feel as you do that "both" is the right answer, and there certainly was a lot of demand for "both" in Python when Discourse was set up there.

I just think the current interfaces and the decision to go with Django has hurt Mailman 3 rather than help it.

That's assuming that the likely alternative was a non-Django framework rather than "ssh to the host and fire up python mailman.client". It was "ssh to the host and fire up python mailman.client", though. ;-)

I also mirror Jim's question of who is the "we" in this conversation.

In practice, it was an "I": Barry Warsaw started rewriting the core around a decade ago. Then when a beta-ready version became imminent a few years later, a couple more "I"s, IIRC Terri Oda and Florian Fuchs wrote much of Postorius (which Barry named), and the Fedora folks did HyperKitty because they wanted a forum-like archiver it for the Fedora lists. For the last few years, both Postorius and HyperKitty have been maintained and developed by the "Mailman project team", but those are the folks primarily responsible for the original design decisions AIUI.

That's how this works. People see a need, they start hacking on it without fanfare because they're not committed to it, once they have an idea of how much work a commitment involves and they're OK with that they commit and announce, which often has a chilling effect on independent alternatives, and tends to cut out users who don't know they need pay attention if they want input to something that will be available years later (if at all). I don't know what to do about the users; Barry did talk about Mailman 3 on-list occasionally, mostly in response to issues raised about Mailman 2.

Why wasn't I invited? :-D

As always in open source, everybody in general is invited, (almost) nobody gets a personal invitation.[1] It's unfortunate that the way things work folks like you and Jim don't find it so easy to pop over to mailman-developers to find out about these things in advance, but we also don't want to burden mailman-users with nitty-gritty details that that may never be relevant to them.

What prevents Mailman 3 from replacing Mailman 2 completely?

Mailman 2 ain't broke, so I don't advise people who are happy with their installations to try to fix it. Not even by installing Mailman 3. ;-)

Is it because of the interfaces for Mailman 3 totally left Mailman 2 behind or was it the decision to use Django?

Mailman 3 cannot be a drop-in replacement for Mailman 2 because by design Mailman 3 core has no comprehensive administrative or user access, except via shell access to the Mailman server. Otherwise, the only user access is subscribe/unsubscribe by mail, and I don't think there's any administrative access by mail (maybe moderation can be enabled? but it would be disabled by default because mail is tres insecure by default).

Cannot Mailman 3 be used as a standalone 'traditional' mailing list without the need for something like Hyperkitty?

You really need Postorius for the administrator, and it's extremely desirable for the users. HyperKitty is trivial to avoid: just don't have archives. Almost as trivial is to use mail-archive.com or similar.

Can Pipermail be modified to work with Mailman 3 as perhaps a stopgap for Mailman 2 users to feel more comfortable with migrating to Mailman 3?

Yes, but why? I don't know any users who prefer Pipermail to HyperKitty. It's the site admins who (occasionally) end up tearing their hair out over configuration.

Maybe you mean Mailman 2's native CGIs for admin and user configuration vs. Postorius rather than the Pipermail archiver?

I host hundreds of Mailman 2 lists and I cannot get my clients interested in Mailman 3. It doesn't have all of the features that Mailman 2 has when it comes to list settings, at least not visible

This is true. I think the main interest from users and list admins is that for some lists HyperKitty is an acceptable alternative for people who strongly prefer forum-in-a-browser (not denying your point about its UI, just saying that on some lists it's good enough). It's not prominent to most users, but having a single user that can have multiple addresses and multiple subscriptions is an advantage (this is supported in core, not just in Postorius). And for site admins, having proper support for virtual domains is big (though you may not see it that way if you're happy enough with cPanel's workaround).

and Hyperkitty is just not impressive to look at when it comes to providing a community feel.

Can you be more specific? User feedback here has mostly been that HyperKitty is more Bright! Shiny! and has better search than Pipermail (which has none, natively), and I have no intuition for what you mean by "community feel". Again, perhaps you mean "Postorius" rather than "HyperKitty" for some of this?

BTW, any confusion between the two is natural, IMO. Most sites (and I believe the default) are configured to assume that visiting the archives is frequent, subscription and user management rare. Therefore the default for the "lists" subdomain is HyperKitty, with some other URL to access Postorius. And of course the user thinks of the whole thing just as "Mailman". :-( It's not a seamless UX, but there's usually no explicit divide, either.

I want to research to see if it is possible to provide a browser base interface to convert/import a Mailman 2 list into a Mailman 3 list without the need of using a command line. Again I am just focusing on the list (admin) settings to be imported at this point not archives into a forum setting.

This can happen -- there's a pretty good script for this already -- but I think at present about 5% of lists require some admin intervention besides running the script. I'm not sure what it will take to get enough of that last 5%. Mark and Abhilash would have a better handle on this, I haven't been keeping up with Mailman 3 user support much lately.

...

This makes no sense to me. If your problem with Django is that it's written in Python, you've got the REST interface. That's as far as our responsibility goes. See "REST is the approach" below.

I assume Django was primarily chosen was because it was written in Python.

Of course, that was a requirement. *We* have to maintain the bundlable web interfaces. There were no external volunteers to write an administrative interface. The Fedora people who wrote HyperKitty were ultimately funded by Red Hat, a company which has been all in on Python for more than two decades. I don't think they were *required* to use Python, but I'm not at all surprised they had a strong bias. Also, the mailman.client Python bindings for the REST API were already available, and provide that API to Python applications. No such bindings were available for other languages or platforms. (I think there are now node.js bindings for the REST API, but that was a couple years ago and I don't know if they are maintained or even used.)

There were alternative frameworks; I'm pretty sure Flask (very minimal) and Pyramid (much more modular than Django), as well as the venerable Zope of course, were available and all meet the "Python" desideratum. The fact is that at that time Django was considered Python's rival to running on Rails. It was the current hotness (AFAIK, it still is in the Python world), and it had an ORM and plugins for all the stuff we thought we would need for a robust administrative application (such as social auth).

My main problem with Django is you have to handle a 3rd interface with Mailman 3. So we have three: Postorius, Hyperkitty, and Django.

I don't understand the Django part. After installation I've never touched django-admin, I've never used the Django web administration interface, and I recall only a few cases where I needed to mess with Django's config files at installation.

In other words, [I'm aiming at] One Admin Interface To Rule Them All.

If it's not in Python, we won't be able to support its internals, and that probably means we won't be able to bundle it. So it won't be the OAITRTA. It might be the VBAITRTA, but people will only get it in a hosted environment such as EMWD or a distro uber-package that defaults to it.

BTW, even if you eliminate Django, you may still have the problem of administration for a database backend, depending on how barebones the VBITRTA is. Mailman core has no provision for adding additional profile information to its database, while both Postorius and HyperKitty do keep such information in separate tables (of course in most cases the same RDBMS backend is used for all three). The archiver will almost certainly want to keep additional information (thread and author indicies, maybe a keyword and/or full-text search index) in a backend, although I guess you could stick with Pipermail or a similar architecture and use static index files for thread, author, and date views.

I am still, btw, saying that about Django because I am having a very difficult time wrapping my head around its logic.

Nothing wrong with that! Django is quite opinionated, as well as being remarkably powerful. But if we'd been a Ruby shop, we would have written it in Rails, I'm pretty sure, and you'd need to deal with that. That's how these things work.

I just seems to me there is a lot of confusion with the use of Mailman 3.

There's a lot of confusion with Mailman 2, as well, as a review of the last year of malman-users would show. I guess you just don't notice because you're a hosting service with a long history of Mailman 2 use. It's email, OF COURSE there's a lot of confusion. :-(

The problem is the discrepancies between the versions. Once bounce processing shows up in Postorius that discrepancy will spread even further.

Sure. Despite our complaints about lack of resources, there is a lot of progress being made. There's also a ton of change in the email and web app world in the last five years: DMARC, ARC, social auth, 2-factor auth, SSL and TLS protocol version deprecations. We have to keep up with some of them, and we need to prioritize.

The confusing state of installation documentation is also a serious problem. I ended up writing my own that I can get consistent results from with installing Mailman 3 on new servers.

Yeah, we tried participating in Google Season of Docs. It produced some new docs I'll be integrating soon, but it wasn't what I'd hoped. Docs that anticipate user problems are just really hard. Much easier to improve them in retrospect. (I'll buy you a drink if out of the first ten new Mailman 3 site admins you hand your docs to, fewer than five have problems installing Mailman 3 using them. I'm sure they're very good docs, but I doubt that they handle more than one MTA, for example, or more than one httpd.)

So one of my goals in coming up with a new approach is to make the full setup of a Mailman 3 server far easier to do AND document.

An excellent goal, and I'm confident you'll succeed. It may take longer than you expect, that's all.

That is fine but what I did not see is the use of U.I. designers.

There was one who was starting work on HyperKitty (she said, on the Fedora lists many moons back). Don't know what happened to that. Probably $DAYJOB. She's very good and in high demand.

So where was this encouragement [to provide alternative interfaces]?

The REST interface itself.

It's not like we had money to put behind third party projects, and we already had Postorius, and an outside project working on HyperKitty. That's the business model behind volunteer open source. People come together to work on projects for their own reasons. They do what they feel like, and eventually they leave. They're strong on producing code. They even produce code to automate management of the project, they even produce code to automatically document the project, but they're normally not so strong on the actual management or documentation. ;-)

To be honest, I can't say that for-profit dev organizations are all that much better in my experience (although my experience is skewed to very long-lived open source projects, so they've been improving their workflows and their docs, and cultivating "organizational culture", for a long time).

What larger platform did you have in mind to integrate Mailman lists into?

Anything that needed to distribute mail in a user-configurable manner. But *we* weren't going to do the integration.

I am still surprised to see no one has come up with a new interface for Mailman 3.

As I wrote earlier, we were and are amazed at the prevalence of sites that just stand up Mailman 2 and Pipermail with no search, especially if they have a lot of private lists inaccessible to Google and friends. So nobody has a strong incentive to develop new interfaces as long as we provide something that mostly works. Until somebody, such as you, does.

Mark is awesome and I have a great working relationship with him via these lists throughout the years. I believe I will have that with him in the future as well.

Awesome indeed (stop blushing, Mark!), and that's why *we* look forward to him spending more time on Mailman 3. :-)

Steve

Footnotes: [1] I did, once, so I can't say *no*body.

On 2020-02-27 14:51, Bill Cole wrote:

On 27 Feb 2020, at 14:24, Jim Popovitch via Mailman-Users wrote:

...

Personally, I'd like to see the GNU Mailman project have a formal Mailman 2.3 release that supports Python3, I feel that there would be a lot of support for that.

I'm sure there would be widespread applause and congratulations if such a thing were actually released. That sort of "support" is unhelpful towards actually making such a release.

The needed support is the actual skilled effort of writing the required Python3 code. I don't have the time to hunt down the specific statements, but I have vague recollections that both Barry and Mark have said repeatedly that doing so would be substantially more effort than they are willing to put into anything built on the MM2 architecture.

Rewriting without breaking is hard.

There is a Python framework called Twisted. It has a lot of useful features. Also a lot of vices, but a lot of useful features. As best I can determine, the task of updating it to be Python 3 compatible has now been under way for ten years (with most of that time, only one person working on it).

What has this yielded?

"Most of the most commonly used parts" of Twisted are now Python 3 compatible.

-- Phil Stracchino Babylon Communications phils@caerllewys.net phil@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958

Phil Stracchino writes:

Rewriting without breaking is hard.

True.

There is a Python framework called Twisted.

Not an example appropriate to Mailman, though. The Twisted people were doing amazing things with str, to which Unicode was irrelevant, because their job was to shovel bytes from here to there correctly but as fast as possible. (Mercurial has a similar story, and a similar visceral hatred for Python 3.)

Mailman has the opposite problem. We *wish* str was Unicode from the get-go, but it wasn't, and Mailman 2 is rife with potential encoded/ decoded confusion because of the nature of email and the dual usage of str in Python 1, and the history of Mailman as an MLM for an American rock band (who needs no steekin' accents, we just hammer and bend the strings!) There are two decades of hacks and patches in Mailman 2 to catch the squirmers that somehow manage to be str where unicode is needed or vice versa, and every single one of those would need to be reverse engineered in a Python 3 environment. Not a job I would want to do: like Barry, I would rewrite from scratch (and probably redesign as well). But every part converted would be a joy to work with in the future.

As best I can determine, the task of updating it to be Python 3 compatible has now been under way for ten years (with most of that time, only one person working on it).

But that's because Python 3 deliberately encouraged decoding streams of bytes, by making it hard to process bytes the same way as str in Python 2. It wouldn't have been hard to make the bytes type identical to str except for the internal representation, so that programs like Twisted and Mercurial would just need to be converted so that *everything* was bytes except for the error messages. But that was deliberately avoided: a lot of (Python 2) str methods were not inherited by bytes. (In fact, some were re-added later, but too late to make the bit-shovelers happy.) So the Twisted people hated Python 3 with a passion.

I'm not surprised that only one person would work on the port, I'm surprised they found even one!

Steve

Phil Stracchino writes:

On 2020-02-28 05:44, Stephen J. Turnbull wrote:

...

str in Python 1, and the history of Mailman as an MLM for an American rock band (who needs no steekin' accents, we just hammer and bend the strings!)

This is clearly a story I didn't know. :) And now I'm curious...

John Viega was a friend of somebody in the Dave Matthews Band, maybe Matthews himself. In the mid-90s, they needed a mailing list to tell people where they were playing, John didn't like any of the MLMs available, so he wrote Mailman. For more info, you'd have to chase down John or Barry Warsaw, I think. Maybe Mark knows more.

Barry wrote a chapter on Mailman in "The Architecture of Open Source Applications", there is some historical stuff in there. And https://www.google.com/search?client=firefox-b-d&q=dave+matthews+band+GNU+Mailman brings up a bunch of relevant-looking links.

Thanks for asking, I may have to follow some of those myself!

Steve

P.S. It is a great story, and a great advert for free/open source software!

Jim Popovitch via Mailman-Users writes:

Interestingly enough, here's a roadmap on exactly how to do it: :)

Jim, you're not helping. Until there are "I'll do it" hands up, no port to Python 3 that is faithful to current Mailman 2 is viable. Pushing it just serves to annoy those who are currently doing work for Mailman that they care more about.

By contrast, your question about security fixes was an entirely appropriate clarification, and #ThankYouForPersisting on that subthread.

Steve

On 3/2/20 8:56 AM, Jim Popovitch via Mailman-Users wrote:

Barry's roadmap for Python2 -> Python3 seems to counter the narrative that MM2 is ill- advised to be ported to Python3 (btw, that was posted in Jan of this year).

The question is what do people want when they say they want Mailman 2 ported to Python 3.

If it means, porting to Python 3 and fixing a few things on the way such as adding a real backend database, a concept of "user" and a REST API, it's at least partially done. It's Mailman 3 core.

If it means cloning the MM 2.1 web UI and pipermail archiver, that is almost certainly not worth the effort.

A compromise is exactly what Brian proposes. Mailman 3 with a new web UI, light weight, not based on Django and easy to install. Mailman 3 was explicitly designed to be separate from a web management UI and Archiver and to allow different implementations of those to integrate easily with core.

Postorius and HyperKitty are part of Mailman 3 because we needed something and that is what people were willing to commit to do. We always hoped there would be alternatives, and it seems that now Brian is working on one. There's room for more.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 3/2/20 4:55 PM, Jim Popovitch via Mailman-Users wrote:

While I applaud Brian's efforts, I'm not convinced that I would run PHP on a public facing portal, even in 2020. But that's just me, others may feel differently.

And so it begins.

-- Please let me know if you need further assistance.

Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com

-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase

EMWD's Community Forums http://discourse.emwd.com/

On Mon, 2020-03-02 at 17:18 -0800, Mark Sapiro wrote:

On 3/2/20 1:55 PM, Jim Popovitch via Mailman-Users wrote:

...

There are plenty of people who are still happy with pipermail and some of the other search options (Google, htdig, etc) What benefit does a REST api provide to church groups, and tech lists like nanog or mailop?

It provides a stable, documented management interface so people can create their own web UIs to control Mailman 3 in whatever way they want. Granted your end user's aren't going to do this, but the people who want it can, and more easily than by porting Mailman 2.1 to Python 3.

Can you share with me (us) the number and size, along with the industry or operations arena, of those people who are creating their own web UI.

I honestly don't believe that there is that much interest for that outside of a handful of entities (Brian, CPanel, Canonical, and LinkedIn?). I feel like if the interest was greater, we'd see more evidence of that in the Gitlab issue tracker and or on the MM3 lists.
Convince me that I'm wrong.

-Jim P.

On Sat, 29 Feb 2020 10:53:19 -0500 Jim Popovitch via Mailman-Users <mailman-users@python.org> wrote:

but I have vague recollections that both Barry and Mark have

...

said repeatedly that doing so would be substantially anything built on the MM2 architecture.

assuming that's so I think the "anything built on the MM2 architecture" is perhaps misconceived. I don't need to be told that MM2 is awkward to set up and run but millions of people get and send mail that way every day and it mostly "just works." This very large body of users it what matters most to actually getting work done, not the developers' wishes and preferences - "more effort than they are willing to put into...". I think increasingly as time goes by that the new New Coke analogy is a good fit.

D

On 2/27/2020 12:58 PM, Mark Sapiro wrote:

On 2/27/20 7:22 AM, Dennis Putnam wrote:

...

I think this may have been addressed but I can't find it. Now that I am munging the from address to mitigate DMARC, recipients can no longer tell who the message is from. What are other folks doing to handle that? Other than having list members add their own signature? Thanks.

The From: header in the munged message contains the sender's display name as in

From: Jane Doe via ListName <listname@example.com>

and depending on list settings the original From: is in either Reply-To: or Cc:.

If this is not sufficient, perhaps the recipients can use smarter email clients ;)

Also, if you are Munging the From: on all messages via the from_is_list setting, it is better to use dmarc_moderation_action for this so only those From: headers that need it are munged. The only reason to use from_is_list is if those users whose domains publish DMARC reject or quarantine policy feel they are singled out and treated as second class users.

Hi Mark,

Thanks for the reply. I am not seeing that. The From: looks like this:

From: Rushtalk Discussion List via Rushtalk <rushtalk@csdco.com>

In "General Options" for that list I set the item "Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies." with "Munge From." Did I set the wrong thing?

On 2/27/2020 1:38 PM, Mark Sapiro wrote:

On 2/27/20 10:17 AM, Dennis Putnam wrote:

...

Thanks for the reply. I am not seeing that. The From: looks like this:

From: Rushtalk Discussion List via Rushtalk <rushtalk@csdco.com>

That must be a RedHat thing having to do with their backport of DMARC mitigations. If you don't like it, install from source.

...

In "General Options" for that list I set the item "Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies." with "Munge From." Did I set the wrong thing?

That will apply From: munging to all posts. I have no idea what the RedHat package does, but if in Privacy options... -> Sender filters you have dmarc_moderation_action and dmarc_quarantine_moderation_action set General Options -> from_is_list to No and set dmarc_moderation_action to Munge From and dmarc_quarantine_moderation_action to Yes and if you have it, dmarc_none_moderation_action to No.

This will apply From: munging only to those messages From: a domain that publishes DMARC policy = reject or quarantine.

Hi Mark,

I didn't realize that there were OS dependencies in the DMARC mitigation. I thought it was all within the mailman code.

In any case I'll look through those options and see what they do in RHEL. Thanks.