Hi all
I'm upgrading a mailman2 installation to mailman3. And I using a docker implementation, created by me and not Maxking's implementation.
Everything is installed and is served correctly. I have created a postfix container that relays emails to a smtp server, and I can send emails by telnet from it. When I create a new thread in Hyperkitty, I receive an error from postfix container that the user (mailing list name) is not found.
And I believe that is obvious because the aliases db are not accessible by postfix.
So, my questions are:
Is there a way to share aliases db between containers(different hosts) of mailman and postfix? I thought about sharing volumes but I'm not quite sure if it's the best solution
I've seen: virtual_alias_maps = mysql:/etc/postfix/virtual/existing_aliases.cf, var/lib/mailman/data/virtual-mailman to add in postfix main.cf. Is it possible to do the same thing to aliases and transport_maps? And if so, how to set it in mailman configuration, because this is for mailman
I've seen this thread: https://mail.python.org/archives/list/mailman-developers@python.org/thread/O... And it seams kind or less the same situation.
Does anyone have an idea or can give some pointers in how to accomplish this?
Kind regards, Emanuel
On 12/18/20 9:49 AM, costavitorino@gmail.com wrote:
Hi all
I'm upgrading a mailman2 installation to mailman3. And I using a docker implementation, created by me and not Maxking's implementation.
Everything is installed and is served correctly. I have created a postfix container that relays emails to a smtp server, and I can send emails by telnet from it. When I create a new thread in Hyperkitty, I receive an error from postfix container that the user (mailing list name) is not found.
And I believe that is obvious because the aliases db are not accessible by postfix.
In Mailman 3, Postfix uses transport_maps to deliver to Mailman via LMTP. It doesn't use aliases.
So, my questions are:
- Is there a way to share aliases db between containers(different hosts) of mailman and postfix? I thought about sharing volumes but I'm not quite sure if it's the best solution
I don't know Docker and can't answer this.
- I've seen: virtual_alias_maps = mysql:/etc/postfix/virtual/existing_aliases.cf, var/lib/mailman/data/virtual-mailman to add in postfix main.cf.
That looks very strasnge. I don't think Postfix would like a relative path, but ignoring that, it certainly wants a table type.
Is it possible to do the same thing to aliases and transport_maps? And if so, how to set it in mailman configuration, because this is for mailman
I don't know what you mean by "the same thing". See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.html#postfix for configuring Postfix for Mailman 3.
Mailman can generate either hash tables or regexp tables. The above doc has details.
Am 18.12.20 um 18:49 schrieb costavitorino@gmail.com:
So, my questions are:
- Is there a way to share aliases db between containers(different hosts) of mailman and postfix? I thought about sharing volumes but I'm not quite sure if it's the best solution
It is possible to share volumes between containers (or preferably use a host directory that's mounted into both containers). You need to be careful with userid issues, though, if you're running the processes within the containers as non-root users.
- I've seen: virtual_alias_maps = mysql:/etc/postfix/virtual/existing_aliases.cf, var/lib/mailman/data/virtual-mailman to add in postfix main.cf. Is it possible to do the same thing to aliases and transport_maps? And if so, how to set it in mailman configuration, because this is for mailman
As Mark already wrote, Mailman3 uses LMTP, but you still need to share the transport map created by Mailman3 with postfix. With 2 containers having access to the same shared filesystem (either running on the same docker host, or using a network filesystem when you've got a cluster) it is pretty straightforward.
In a configuration where Mailman3 and Postfix are running on separate hosts, I have used the Mailman3 "postmap-command" option in var/etc/postfix-mailman.cfg to call a shell script which first runs postmap and then copies the map to the postfix host. That works well enough, but it took some time experimenting with several options. You need to set the lmtp_host and lmtp_port options in mailman.cfg to point to the container running Mailman3. If you expose the container LMTP port to the docker host, it might be easier, but it's likely safer to keep the inter-container communication really internal.
We're currently working on moving our mail system over to a "mailu" dockerized implementation, so I will gather some experience in connecting Mailman3 with a dockerized postfix.
Cheers, Hans-Martin
Hi Hans-Martin,
On Sat, Dec 19, 2020 at 1:05 PM Hans-Martin Mosner hmm@heeg.de wrote:
We're currently working on moving our mail system over to a "mailu" dockerized implementation, so I will gather some experience in connecting Mailman3 with a dockerized postfix.
Yeah, using dockerised postfix is straight-forward in the same docker-compose stack - one just need to mount var/data folder of Mailman core into the Postfix container, set transport_maps, local_recipient_maps and relay_domains parameters to correct values and set Postfix service name into SMTP_HOST environment variable of Core and Web services.
(I'm using my simple Postfix Docker image for this purpose - works pretty well: https://hub.docker.com/r/danilsmirnov/postfix )
But how about that idea to offload Postfix-Mailman communication from a filesystem to a network using Postfix socketmap_tables feature which I like a lot as it allows me to get rid of using any shared volumes in the stack completely - it's quite convenient in multi-host setups like Docker Swarm or Kubernetes...
Best regards, Danil
Danil Smirnov writes:
But how about that idea to offload Postfix-Mailman communication from a filesystem to a network using Postfix socketmap_tables feature which I like a lot as it allows me to get rid of using any shared volumes in the stack completely - it's quite convenient in multi-host setups like Docker Swarm or Kubernetes...
Would you be willing to write up a description of your configuration that we can add to the wiki and/or documentation? Communication in multihost setups, especially those intended to isolate hosts exposed to the internet from the intranet, is a sore point for us since Mailman 3 is really designed as a single-host, three application system.
Steve
Hey Steve,
вс, 20 дек. 2020 г., 6:03 Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp>:
Danil Smirnov writes:
But how about that idea to offload Postfix-Mailman communication from a filesystem to a network using Postfix socketmap_tables feature which I
like
a lot as it allows me to get rid of using any shared volumes in the
stack
completely - it's quite convenient in multi-host setups like Docker
Swarm
or Kubernetes...
Would you be willing to write up a description of your configuration that we can add to the wiki and/or documentation? Communication in multihost setups, especially those intended to isolate hosts exposed to the internet from the intranet, is a sore point for us since Mailman 3 is really designed as a single-host, three application system.
Sorry for the confusion - I should write "would allow" instead of "allows" as using socketmap_table would be a great feature but it obviously requires a new endpoint in Mailman core which does not exist yet.
I've experimented a lot with Mailman 3 in Kubernetes and I've faced only two disappointing issues: 1) HK whitelisting mechanism https://gitlab.com/mailman/hyperkitty/-/issues/18 which makes no sense in a dynamic Kubernetes environment and 2) a shared volume, which is needed for Core and Postfix to communicate.
Otherwise everything looks fine and with those two issues resolved/mitigated there is no problem to deploy Mailman 3 in Kubernetes, I can prepare an example Helm chart for it.
Danil
Danil Smirnov writes:
Sorry for the confusion - I should write "would allow" instead of "allows" as using socketmap_table would be a great feature but it obviously requires a new endpoint in Mailman core which does not exist yet.
Ah, OK, so what exists now is the Postfix docker (which you provided a reference).
Otherwise everything looks fine and with those two issues resolved/mitigated there is no problem to deploy Mailman 3 in Kubernetes, I can prepare an example Helm chart for it.
That would be great!
Steve
Hey,
As promised a while ago, I've prepared Kubernetes PoC for Mailman 3 (Helm chart) which is currently deployed into GCP (GKE Autopilot) and exposed at http://test.mailman4.com/postorius/lists/
The setup is based on Abhilash's container images https://github.com/maxking/docker-mailman (up to version 0.3.11 - see below why). Outbound delivery is relayed via Mailgun (due to the outbound port 25 is blocked in GCP), while income emails received by the local Postfix container.
When tried previously,
...I've faced only two disappointing issues: 1) HK whitelisting mechanism
which makes no sense in a dynamic Kubernetes environment and 2) a shared volume, which is needed for Core and Postfix to communicate.
The first issue has been resolved by allowing Django's any host notation https://gitlab.com/mailman/hyperkitty/-/merge_requests/297 (thank you folks!), and the second one I've resolved by myself with help of a sidecar container built from the following code: https://github.com/danil-smirnov/mailman-postfix-extender
This solution moves Postfix configuration from file-based to network-based without touching a line of Mailman code. (Of course, it's PoC still and would look much better if it's part of the Mailman 3 codebase, covered by tests, etc.)
There are a couple of other issues that I've worked around but have to address to move to a production-grade solution:
- Hardcoded hostnames issue (
https://github.com/maxking/docker-mailman/issues/449) Worked around by adding some redundant code https://github.com/danil-smirnov/mailman-helm-chart/blob/main/templates/mailman-web/deployment.yaml#L108-L113 . 2. lmtp_host parameter issue ( https://gitlab.com/mailman/mailman/-/issues/852) Worked around by using IP addresses in postfix_lmtp file (possible up to 0.3.11 version of docker images only - due to recent change https://github.com/maxking/docker-mailman/commit/c10aa6fce479a78a6c51ac41f0f1f0963bd20d0b#diff-27e621da191f9972846a94828429f1f91d0cdde5cb7ff3ddddc5fe8db38797f2R33 ).
The code could be found in this repository: https://github.com/danil-smirnov/mailman-helm-chart
I'm glad I've finally shared this, I hope it'll help to improve Mailman 3 compatibility with the modern orchestration tools.
With my best regards, Danil Smirnov Mailman3.com
On Sun, Dec 20, 2020 at 1:51 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Danil Smirnov writes:
Sorry for the confusion - I should write "would allow" instead of
"allows"
as using socketmap_table would be a great feature but it obviously
requires
a new endpoint in Mailman core which does not exist yet.
Ah, OK, so what exists now is the Postfix docker (which you provided a reference).
Otherwise everything looks fine and with those two issues resolved/mitigated there is no problem to deploy Mailman 3 in
Kubernetes, I
can prepare an example Helm chart for it.
That would be great!
Steve
participants (5)
-
costavitorino＠gmail.com -
Danil Smirnov -
Hans-Martin Mosner -
Mark Sapiro -
Stephen J. Turnbull