Hi everybody,
I'm getting very near ready to release another alpha of Mailman 3, and on the prompting of a private message from Robert Niederreiter I took some time to fire up a VM and actually try Postfix+LMTP delivery in an experimental production system. I'd like to get some feedback from the Postfix experts in the crowd so that we can update the wiki page here:
http://wiki.list.org/display/DEV/LMTP+processFTR: the VM is running Ubuntu Karmic Koala 9.10 server with Postfix 2.6.5 and Mailman 3.0 from the lp:mailman bzr trunk.
To start with, I mostly followed the instructions in the wiki. I tried using the standard lmtp transport in master.cf, creating /etc/postfix/mailman_lists as described in the wiki (not the dedicated list server example, but the earlier one on the page). I actually used:
# Key # Value test@xxx.example.com lmtp:xxx.example.com
where 'xxx.example.com' is replaced by my VM's host name.
The first gotcha is that unless you start Mailman as root, you can't bind its LMTP server to port 24, which is Postfix's default. By default Mailman's LMTP server listens on localhost:8024, so you either need to append ':8024' on the Value above, or set
lmtp_tcp_port = 8024in Postfix's main.cf. I ended up doing the latter, but both work.
I set up transport_maps and local_recipient_maps just as outlined in the wiki page, fired everything up, and then sent a message to the VM's Postfix while tailing the logs. I kept seeing "Connection refused" messages from Postfix and it never hit Mailman's LMTP server.
This was highly confusing because I could see in the Postfix logs that it was finding the right IP address for its own hostname and it was trying the right port number. Mailman's lmtp.log file claimed it was listening on the right IP and port, and I could even telnet to it just fine. So what was Postfix doing?
It seems that Karmic is playing tricks with /etc/hosts. I've got DNS set up to correctly resolve the VM's hostname, but there was actually an entry in /etc/hosts that set xxx.example.com to 127.0.1.1. I'm not entirely sure which process looks at what, when, but clearly this inconsistency was a key part of the problem. The other thing that surprised me was that Postfix was also consulting /var/spool/postfix/etc/hosts, and I had to change both /etc/hosts and that file as well, so that the IP addresses jived with DNS. This is unsatisfactory, but it seemed critical to getting things working.
The last piece of the puzzle was to not use Postfix's standard lmtp server in master.cf, but to define a new one like so:
mailman3 unix - - - - - lmtp -o disable_dns_lookups=yes
and then change the mailman_lists transport map to
# Key # Value test@xxx.example.com mailman3:xxx.example.com
After a restart, everything suddenly worked exactly as expected.
Robert was having a different problem, but hopefully he will follow up here with his experience and let us know if any of the above helps. If any Postfix experts have words of wisdom to make this better, please let me know.
I probably need to work on better dropping of privileges for qrunners so that you can 'bin/mailman start' as root, and once the LMTP runner binds to port 24, it can drop privs to 'mailman'. I'll put that on the list for something to do after the next alpha.
I'd also like to try to resurrect William Mead's LMTP branch. Sadly, it won't merge cleanly into trunk any more (not the least of which because it's in the wrong bzr format). If anybody would like to contribute that before I get to it, I'd be grateful.
The good news is that I think Mailman 3 is getting more real every day. My plan over the next few weeks is:
- release alpha 4
- get i18n translations working
- complete the split of the pipermail project
- hopefully get Patrick and friends going on the web u/i
- announce something <wink>
- create a live test list that anybody can subscribe to
If you've been waiting to play with Mailman 3, wait no longer!
-Barry
Hi!
- Barry Warsaw <barry@python.org>:
I'm getting very near ready to release another alpha of Mailman 3, and on the prompting of a private message from Robert Niederreiter I took some time to fire up a VM and actually try Postfix+LMTP delivery in an experimental production system. I'd like to get some feedback from the Postfix experts in the crowd so that we can update the wiki page here:
http://wiki.list.org/display/DEV/LMTP+processFTR: the VM is running Ubuntu Karmic Koala 9.10 server with Postfix 2.6.5 and Mailman 3.0 from the lp:mailman bzr trunk.
To start with, I mostly followed the instructions in the wiki. I tried using the standard lmtp transport in master.cf, creating /etc/postfix/mailman_lists as described in the wiki (not the dedicated list server example, but the earlier one on the page). I actually used:
# Key # Value test@xxx.example.com lmtp:xxx.example.com
where 'xxx.example.com' is replaced by my VM's host name.
Additionally you probably want to add square brackets around the hostname.
Quoting "man 5 transport":
...the [] suppress MX lookups. This prevents mail routing loops when your machine is primary MX host for example.com.
The first gotcha is that unless you start Mailman as root, you can't bind its LMTP server to port 24, which is Postfix's default. By default Mailman's LMTP server listens on localhost:8024, so you either need to append ':8024' on the Value above, or set
lmtp_tcp_port = 8024in Postfix's main.cf. I ended up doing the latter, but both work.
I think an option in MM3 should make this configurable.
I set up transport_maps and local_recipient_maps just as outlined in the wiki page, fired everything up, and then sent a message to the VM's Postfix while tailing the logs. I kept seeing "Connection refused" messages from Postfix and it never hit Mailman's LMTP server.
This was highly confusing because I could see in the Postfix logs that it was finding the right IP address for its own hostname and it was trying the right port number. Mailman's lmtp.log file claimed it was listening on the right IP and port, and I could even telnet to it just fine. So what was Postfix doing?
It seems that Karmic is playing tricks with /etc/hosts. I've got DNS set up to correctly resolve the VM's hostname, but there was actually an entry in /etc/hosts that set xxx.example.com to 127.0.1.1. I'm not entirely sure which process looks at what, when, but clearly this inconsistency was a key part of the problem. The other thing that surprised me was that Postfix was also
On a sidenote: It also confuses a dhcpd server running on a machine that promotes 127.0.1.1 to be xxx.example.com. I've opened a bug ticket for that two Ubuntu releases ago, but it seems they only keep carrying it further from release to release instead of addressing it. At least that's the impression I get without knowing the real cause.
consulting /var/spool/postfix/etc/hosts, and I had to change both /etc/hosts
That's because LaMont Jones delivers the Debian/Ubuntu Postfix package chrooted. If Postfix runs chrooted it uses /var/spool/postfix/etc/hosts instead of /etc/hosts.
and that file as well, so that the IP addresses jived with DNS. This is unsatisfactory, but it seemed critical to getting things working.
The last piece of the puzzle was to not use Postfix's standard lmtp server in
lmtp client... SCNR
master.cf, but to define a new one like so:
mailman3 unix - - - - - lmtp -o disable_dns_lookups=yes
and then change the mailman_lists transport map to
# Key # Value test@xxx.example.com mailman3:xxx.example.com
After a restart, everything suddenly worked exactly as expected.
suddenly... :)
Robert was having a different problem, but hopefully he will follow up here with his experience and let us know if any of the above helps. If any Postfix experts have words of wisdom to make this better, please let me know.
If MM relies on Postfix defaults and the postmaster changes them in Postfix the MM part might end up being unusable.
I recommend to take full control of the settings in the Postfix transport map. Let MM create its own Postfix style transport map. Set the "Postfix service name" (in your example its mailman3), put the hostname in square brackets and set the port.
I probably need to work on better dropping of privileges for qrunners so that you can 'bin/mailman start' as root, and once the LMTP runner binds to port 24, it can drop privs to 'mailman'. I'll put that on the list for something to do after the next alpha.
I like that idea.
I'd also like to try to resurrect William Mead's LMTP branch. Sadly, it won't merge cleanly into trunk any more (not the least of which because it's in the wrong bzr format). If anybody would like to contribute that before I get to it, I'd be grateful.
The good news is that I think Mailman 3 is getting more real every day. My plan over the next few weeks is:
- release alpha 4
- get i18n translations working
- complete the split of the pipermail project
- hopefully get Patrick and friends going on the web u/i
I will update to the latest branch that should fix the language file problem we had discovered and then we will work on the REST client/server.
p@rick
-- state of mind Digitale Kommunikation
Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666
Amtsgericht München Partnerschaftsregister PR 563
On Nov 29, 2009, at 3:22 AM, Patrick Ben Koetter wrote:
Hi!
- Barry Warsaw <barry@python.org>:
I'm getting very near ready to release another alpha of Mailman 3,
and on the prompting of a private message from Robert Niederreiter I took some
time to fire up a VM and actually try Postfix+LMTP delivery in an
experimental production system. I'd like to get some feedback from the Postfix
experts in the crowd so that we can update the wiki page here:http://wiki.list.org/display/DEV/LMTP+process
FTR: the VM is running Ubuntu Karmic Koala 9.10 server with Postfix
2.6.5 and Mailman 3.0 from the lp:mailman bzr trunk.To start with, I mostly followed the instructions in the wiki. I
tried using the standard lmtp transport in master.cf, creating /etc/postfix/ mailman_lists as described in the wiki (not the dedicated list server example,
but the earlier one on the page). I actually used:# Key # Value test@xxx.example.com lmtp:xxx.example.com
where 'xxx.example.com' is replaced by my VM's host name.
Additionally you probably want to add square brackets around the
hostname.Quoting "man 5 transport":
...the [] suppress MX lookups. This prevents mail routing loops
when your machine is primary MX host for example.com.
This could have been Robert's problem. I'd forgotten that we actually
do create the transport map automatically, if you set the
configuration to use Postfix's LMTP delivery (this is the default).
In that case, we generate a file called data/postfix_lmtp (and use
postmap to create the .db file) which should be usable directly as the
transport_map. I've changed it to include the brackets around the
host name.
This actually greatly simplifies things, since all you need to do is
set transport_maps and local_recipients and you're done.
The first gotcha is that unless you start Mailman as root, you
can't bind its LMTP server to port 24, which is Postfix's default. By default
Mailman's LMTP server listens on localhost:8024, so you either need to append ': 8024' on the Value above, or setlmtp_tcp_port = 8024
in Postfix's main.cf. I ended up doing the latter, but both work.
I think an option in MM3 should make this configurable.
It does. Also putting the port in the Mailman generated transport_map
means the right thing will always happen (modulo the permission
dropping part which won't make it into 3.0a4).
I set up transport_maps and local_recipient_maps just as outlined
in the wiki page, fired everything up, and then sent a message to the VM's
Postfix while tailing the logs. I kept seeing "Connection refused" messages from
Postfix and it never hit Mailman's LMTP server.This was highly confusing because I could see in the Postfix logs
that it was finding the right IP address for its own hostname and it was trying
the right port number. Mailman's lmtp.log file claimed it was listening on
the right IP and port, and I could even telnet to it just fine. So what was
Postfix doing?It seems that Karmic is playing tricks with /etc/hosts. I've got
DNS set up to correctly resolve the VM's hostname, but there was actually an
entry in /etc/hosts that set xxx.example.com to 127.0.1.1. I'm not entirely
sure which process looks at what, when, but clearly this inconsistency was a
key part of the problem. The other thing that surprised me was that Postfix
was alsoOn a sidenote: It also confuses a dhcpd server running on a machine
that promotes 127.0.1.1 to be xxx.example.com. I've opened a bug ticket
for that two Ubuntu releases ago, but it seems they only keep carrying it
further from release to release instead of addressing it. At least that's the
impression I get without knowing the real cause.
What's the bug number?
consulting /var/spool/postfix/etc/hosts, and I had to change both / etc/hosts
That's because LaMont Jones delivers the Debian/Ubuntu Postfix package chrooted. If Postfix runs chrooted it uses /var/spool/postfix/etc/ hosts instead of /etc/hosts.
Ah, the pieces click together. Thanks!
and that file as well, so that the IP addresses jived with DNS.
This is unsatisfactory, but it seemed critical to getting things working.The last piece of the puzzle was to not use Postfix's standard lmtp
server inlmtp client... SCNR
Erp. ;)
master.cf, but to define a new one like so:
mailman3 unix - - - - - lmtp -o disable_dns_lookups=yes
and then change the mailman_lists transport map to
# Key # Value test@xxx.example.com mailman3:xxx.example.com
After a restart, everything suddenly worked exactly as expected.
suddenly... :)
Yeah. Doesn't everybody call 6 hours on a Saturday "suddenly"? :)
Robert was having a different problem, but hopefully he will follow
up here with his experience and let us know if any of the above helps. If
any Postfix experts have words of wisdom to make this better, please let me know.If MM relies on Postfix defaults and the postmaster changes them in
Postfix the MM part might end up being unusable.I recommend to take full control of the settings in the Postfix
transport map. Let MM create its own Postfix style transport map. Set the "Postfix
service name" (in your example its mailman3), put the hostname in square
brackets and set the port.
Yep, I like setting the service name to mailman3. I'll note that
Ubuntu's Postfix-Dovecot package already has a service for mailman
(i.e. 2.x) which uses postfix-to-mailman.py.
I probably need to work on better dropping of privileges for
qrunners so that you can 'bin/mailman start' as root, and once the LMTP runner binds
to port 24, it can drop privs to 'mailman'. I'll put that on the list for
something to do after the next alpha.I like that idea.
I'd also like to try to resurrect William Mead's LMTP branch.
Sadly, it won't merge cleanly into trunk any more (not the least of which because
it's in the wrong bzr format). If anybody would like to contribute that before
I get to it, I'd be grateful.The good news is that I think Mailman 3 is getting more real every
day. My plan over the next few weeks is:
- release alpha 4
- get i18n translations working
- complete the split of the pipermail project
- hopefully get Patrick and friends going on the web u/i
I will update to the latest branch that should fix the language file
problem we had discovered and then we will work on the REST client/server.
Excellent! I forgot that one other thing must happen before 3.0 can
be released; we need a migration script from 2.1.x to 3.0.
I'll update the wiki page and release 3.0a4 a little later today.
Then perhaps other folks can try the LMTP connection on other *nix
distributions.
Thanks Patrick, -Barry
- Barry Warsaw <barry@python.org>:
It seems that Karmic is playing tricks with /etc/hosts. I've got DNS set up to correctly resolve the VM's hostname, but there was actually an entry in /etc/hosts that set xxx.example.com to 127.0.1.1. I'm not entirely sure which process looks at what, when, but clearly this inconsistency was a key part of the problem. The other thing that surprised me was that Postfix was also
On a sidenote: It also confuses a dhcpd server running on a machine that promotes 127.0.1.1 to be xxx.example.com. I've opened a bug ticket for that two Ubuntu releases ago, but it seems they only keep carrying it further from release to release instead of addressing it. At least that's the impression I get without knowing the real cause.
What's the bug number?
<https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/340383>
consulting /var/spool/postfix/etc/hosts, and I had to change both /etc/hosts
That's because LaMont Jones delivers the Debian/Ubuntu Postfix package chrooted. If Postfix runs chrooted it uses /var/spool/postfix/etc/ hosts instead of /etc/hosts.
Ah, the pieces click together. Thanks!
BTW: Postfix notes that /etc/hosts and /var/spool/postfix/etc/hosts differ on reload (at least on Debian/Ubuntu) and it sends a simliar note to syslog.
I know experience tells us not to look at logs because logging and writing documentation is something you only get when you pay for it... ;) However with Postfix it pays to read the log. Wietse did a good job on this.
After a restart, everything suddenly worked exactly as expected.
suddenly... :)
Yeah. Doesn't everybody call 6 hours on a Saturday "suddenly"? :)
:)
Robert was having a different problem, but hopefully he will follow up here with his experience and let us know if any of the above helps. If any Postfix experts have words of wisdom to make this better, please let me know.
If MM relies on Postfix defaults and the postmaster changes them in Postfix the MM part might end up being unusable.
I recommend to take full control of the settings in the Postfix transport map. Let MM create its own Postfix style transport map. Set the "Postfix service name" (in your example its mailman3), put the hostname in square brackets and set the port.
Yep, I like setting the service name to mailman3. I'll note that Ubuntu's Postfix-Dovecot package already has a service for mailman (i.e. 2.x) which uses postfix-to-mailman.py.
If we came up with a reasonable set of default settings for the mailman3 service I take these to the Postfix developer list and try to have them added to the Postfix default master.cf file.
I probably need to work on better dropping of privileges for qrunners so that you can 'bin/mailman start' as root, and once the LMTP runner binds to port 24, it can drop privs to 'mailman'. I'll put that on the list for something to do after the next alpha.
I like that idea.
I'd also like to try to resurrect William Mead's LMTP branch. Sadly, it won't merge cleanly into trunk any more (not the least of which because it's in the wrong bzr format). If anybody would like to contribute that before I get to it, I'd be grateful.
The good news is that I think Mailman 3 is getting more real every day. My plan over the next few weeks is:
- release alpha 4
- get i18n translations working
- complete the split of the pipermail project
- hopefully get Patrick and friends going on the web u/i
I will update to the latest branch that should fix the language file problem we had discovered and then we will work on the REST client/server.
Excellent! I forgot that one other thing must happen before 3.0 can be released; we need a migration script from 2.1.x to 3.0.
May I complement the list?
- logo contest
- send the web interface off to a usability lab
p@rick
-- state of mind Digitale Kommunikation
Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666
Amtsgericht München Partnerschaftsregister PR 563
On Nov 29, 2009, at 1:39 PM, Patrick Ben Koetter wrote:
<https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/340383>
Thanks, I've added this to the wiki page.
consulting /var/spool/postfix/etc/hosts, and I had to change both /etc/hosts
That's because LaMont Jones delivers the Debian/Ubuntu Postfix
package chrooted. If Postfix runs chrooted it uses /var/spool/postfix/etc/
hosts instead of /etc/hosts.Ah, the pieces click together. Thanks!
BTW: Postfix notes that /etc/hosts and /var/spool/postfix/etc/hosts
differ on reload (at least on Debian/Ubuntu) and it sends a simliar note to
syslog.
Yeah, that's how I found out about it in the first place. ;)
I know experience tells us not to look at logs because logging and
writing documentation is something you only get when you pay for it... ;)
However with Postfix it pays to read the log. Wietse did a good job on this.
Indeed!
Yep, I like setting the service name to mailman3. I'll note that Ubuntu's Postfix-Dovecot package already has a service for mailman (i.e. 2.x) which uses postfix-to-mailman.py.
If we came up with a reasonable set of default settings for the
mailman3 service I take these to the Postfix developer list and try to have
them added to the Postfix default master.cf file.
+1. I've opened LP bug 490030 to track the changes needed in Mailman.
https://bugs.launchpad.net/mailman/+bug/490030
Excellent! I forgot that one other thing must happen before 3.0 can be released; we need a migration script from 2.1.x to 3.0.
May I complement the list?
- logo contest
Yep, it's actually on my TODO list after I get 3.0a4 out. I also need
to finish setting up i18n.
- send the web interface off to a usability lab
+1. IIRC, Terri was going to look into this, but I'm not sure what's
happened with that.
-Barry
- Barry Warsaw <barry@python.org>:
May I complement the list?
- logo contest
Yep, it's actually on my TODO list after I get 3.0a4 out. I also need to finish setting up i18n.
- send the web interface off to a usability lab
+1. IIRC, Terri was going to look into this, but I'm not sure what's happened with that.
She did and offered a contact. As soon as the web interface has come to a usable state I will try to set the testing on the tracks.
p@rick
-- state of mind Digitale Kommunikation
Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666
Amtsgericht München Partnerschaftsregister PR 563
On Nov 29, 2009, at 3:29 PM, Patrick Ben Koetter wrote:
+1. IIRC, Terri was going to look into this, but I'm not sure what's happened with that.
She did and offered a contact. As soon as the web interface has come
to a usable state I will try to set the testing on the tracks.
Cool. -Barry
Patrick Ben Koetter wrote:
- Barry Warsaw <barry@python.org>:
- send the web interface off to a usability lab +1. IIRC, Terri was going to look into this, but I'm not sure what's happened with that. She did and offered a contact. As soon as the web interface has come to a usable state I will try to set the testing on the tracks.
Yup, I've got the appropriate people interested, but they're waiting for a design to test before we hash out the details of a lab study. Really looking forwards to working with this team -- I've been very impressed with what they do and the insights they garner.
Incidentally, how's that web interface coming? I don't want to make any promises yet, but if work permits I'd like to take a bit of an open source holiday and hack for a week mid-december... I was tentatively thinking archives, but I could be convinced to poke somewhere else. :)
Terri
On Nov 29, 2009, at 9:58 PM, Terri Oda wrote:
Yup, I've got the appropriate people interested, but they're waiting
for a design to test before we hash out the details of a lab study.
Really looking forwards to working with this team -- I've been very
impressed with what they do and the insights they garner.Incidentally, how's that web interface coming? I don't want to make
any promises yet, but if work permits I'd like to take a bit of an
open source holiday and hack for a week mid-december... I was
tentatively thinking archives, but I could be convinced to poke
somewhere else. :)
Either would rock. I have a branch that rips Pipermail out of the
Mailman 3 source and into a separate project on Launchpad:
https://edge.launchpad.net/pipermail
It's not functional yet because I've had to do some refactoring (e.g.
of the i18n stuff), but it's getting there. The branch should be
owned by ~mailman-coders so you should be able to read and write to
it. Of course, if you don't think Pipermail is worth the effort... :)
-Barry
participants (3)
-
Barry Warsaw -
Patrick Ben Koetter -
Terri Oda