Mailman-Developers November 2009

mailman-developers@python.org

12 participants
11 discussions

Re: [Mailman-Developers] Reply-To munging considered *carefully*
by Stephen J. Turnbull 09 Jan '10

09 Jan '10

Michael B. Trausch writes: > My main, central, and driving point is a desire to create a system that > is both idiot friendly and expert friendly, and my desire is to have all > known common (end-user) and uncommon (technical user) use cases be > handled in a manner that is consistent and portable. That's what we want, too, Brother Michael. The difference between you and us is that we believe in the RFC process, we believe in decentralization, and we believe that if we design a better way to do it in that context, software developers will adopt it and users will get the benefits merely by upgrading their favorite software (or changing, if another implementation becomes more attractive in the process).

7 6

Proposed: remove address-obfuscation code from Mailman 3
by Rich Kulawiec 07 Dec '09

07 Dec '09

Summary: Spammers now have so many ways of "harvesting" addresses from so many systems, and so many ways of exchanging those with each other, that any email address which is actually used WILL eventually be harvested. (Where what "eventually" means varies widely, of course, but can be expected to steadily decrease.) Pretending that address obfuscation in mailing list [or newsgroup] archives will have any meaningful effect on this process gives users a false sense of security and has zero anti-spam value. Summary of summary: It's pointless. Explanation: Spammers maintain extensive databases of email addresses. Some of those databases are merely lists of addresses; others are more sophisticated and include data such as "harvested-date", "havesting-method", "last-seen date", "last-seen-context", "last-known-valid date" and more. Some of these databases are private; others are available for sale/lease. Some are maintained by spammers themselves, others by spammer support services don't directly engage in spamming. The harvesting engines used to acquire email addresses are myriad, as are the methods by which spammers acquire the raw data to use as input to them. *Some* of those methods, and there are many more, include: - subscribing to mailing lists - acquiring Usenet news (NNTP) feeds - querying mail servers - acquiring corporate email directories - insecure LDAP servers - insecure AD servers - use of backscatter/outscatter - use of auto-responders - use of mailing list mechanisms - use of abusive "callback" mechanisms - dictionary attacks - construction of plausible addresses (e.g. "firstname.lastname") - purchase of addresses in bulk on the open market. - purchase of addresses from vendors, web sites, etc. - purchase of addresses from registrars, ISPs, web hosts, etc. - domain registration (some registrars ARE spammers) [1] - misplaced/lost/sold media (hard disk, tape, CD, DVD, USB stick, etc.) and perhaps most significantly: - harvesting of the mail, address books and any other files present on any of the hundreds of millions of compromised Windows systems [2] Consider for example: the first time a newly-created address is used by someone (who is sending a message to it), it's now present on their system: in their saved outbound mail, or perhaps in their address book (if they have one), or in some cache. Any sensible malware resident on their system will of course pick it up and eventually hand it over to a harvesting agent. (Competent malware will harvest it in real time *and* associate it with the sender's address.) And if that particular system happens to be clean? Doesn't help much, because the more times that address is used, the more systems it's present on. And the more systems it's present on, the greater the probability that one of them is already compromised or will be soon. Thus even if we eliminate the originating end-user system as a possible source, we still have to consider the outbound mail server used by that end-user system, which is also a candidate for compromise. And then the inbound mail server used by the recipient, and then the recipient end-user system. And if there's some filtering appliance or intermediate system in place at either end, then it's there too. If the message is forwarded to a third party, then another set of systems is in play. If mail server logs are rolled up and moved to some central location, then it's there too. If backups are made, then it's present there, and subsequently may be present on any system where the backups are read/restored. And finally, if the destination of a mail message isn't an individual user, but an entire mailing list, then we must multiply the number of possible harvesting points by at least the number of people on the mailing list plus a factor for mail servers/gateways/filters/etc. (modulo overlaps). This in turns means that messages to sent to lists of any appreciable size (say, 1000 members) will turn up on considerably more than 1000 systems -- and the chances that all 1000-plus are secure are microscopic. Please note that the previous paragraph's recitation only covered the last vector I enumerated in the [indented] list above: compromised systems. That laundry list of methods also affords many other opportunities for addresses to find their way into spammers' hands. As just one pointed example out of a great many more that could be cited: how do you know that the address user(a)example.com which has just subscribed to the list you run is a real person and not just the front-end for an address-harvester that will pick up every address used to send traffic to the list? And so on. There are far too many others to enumerate, all of which have discussed at great length in anti-spam forums for many years, and are depressingly familiar to experienced practitioners working in the field. The bottom line is that any email address which is actually used [3], *especially* any email address used to send traffic to a mailing list, is going to be harvested. It's only a matter of when, not if, and "when" is getting sooner all the time. Incidentally, everyone (including me) can produce anecdotal tales of addresses that have remained surprisingly under-targeted by spammers over long periods of time. But this is clearly not the way to bet: it is in spammers' interests to ferret out as many addresses as possible and to use them as soon and as often as possible. Note, however, that some addresses are *deliberately* un-/under-targeted, so lack of substantial spam traffic to a given address is NOT an indicator that the address hasn't been harvested. That's because along with target lists, spammers maintain "suppression" lists, which they use to avoid hitting the addresses of people they think are likely to cause issues for them. [4] And obviously, people with postmaster or mailing list roles would be good candidates for membership on those lists. I know that if I were in their shoes, I'd add everyone who's ever sent a message to the mailman-* mailing lists to mine: a quick check indicates that it's on the order of only 10K addresses. Skipping those would be inconsequential when sending spam to a few hundred million addresses, and I trust it's obvious why spammers would benefit from doing so. With all this in mind, it's clearly pointless to pretend that address obfuscation in archives provides any protection at all. [5] It would be better to remove the code entirely than to continue to maintain the facade that it actually has any anti-spam value. Everyone should simply presume that all email addresses are in the hands of spammers and prepare defenses accordingly -- because even if that's not quite true yet, it will be soon enough. Notes: [1] I deliberately didn't mention mass WHOIS queries. While some efforts in this direction were made by spammers years ago, they've found it far more efficient and cost-effective to simply buy WHOIS data in bulk. There's always someone who wants to sell, and a CD/DVD or USB stick will suffice. This is why attempts by registrars to rate-limit queries or restrict access are not only foolish, but disengenuous: spammers already *have* the data, and can acquire updates at will, and they are clearly doing so via processes that lead back to registrars themselves. [2] The exact number of such systems is not only unknown, but unknowable, since any compromised system which (a) doesn't make its presence known (b) to a suitable detector will remain undetected indefinitely. However, two things are clear: (1) any estimate under 100 million should be laughed out of the room, and (2) there is no reason to suspect that the number is decreasing, and there are numerous reasons to suspect that it's increasing. Note, incidentally, that some detectors have reported observing 200,000 new such systems in a single day; and further note that it's now quite routine for individual botnets with several million *known* members to turn up. [3] Addresses which aren't used may remain out of spammer view for considerable time, depending on the care with which they're selected and maintained. However, this obviously excludes addresses used for participation in mailing lists. [4] For the purpose of this discussion, I'm just talking about suppression lists which enumerate individual email addresses. It's well-known that spammers also maintain suppression lists of MX's, domains, network allocations, ASNs, etc., in an attempt to avoid hitting spamtraps and/or hitting the mailboxes of those who might be in a position to file complaints or take action against them. [5] The only people left who are impeded in the slightest by obfuscation code are NON-spammers: that is, people who are trying to contact someone who has previously sent a message to some mailing list. ---Rsk

7 16

Mailman 3.0 alpha 4
by Barry Warsaw 04 Dec '09

04 Dec '09

I'm happy to announce the release of Mailman 3.0 alpha 4. Things are getting quite real now. It's pretty easy to hook Mailman up to Postfix for both incoming and outgoing mail[1] and you can create mailing lists and populate them with addresses. Currently, you must do this through the command line though. Please download the latest tarball or grab the bzr branch, and put it through some paces. Now is a good time to comment on the design and feature set for Mailman 3 as we are still in alpha. I'm still planning on releasing a first beta by the end of the year -- your testing, feedback, input, and contributions can help with that! You can download the tarball either from the Cheeseshop, or Launchpad: http://pypi.python.org/pypi/mailman/3.0.0a4 https://launchpad.net/mailman Please note: Mailman 3.0a4 is not production ready yet. Enjoy, -Barry

2 1

MM 3 alpha 4: UnicodeWarning
by Patrick Ben Koetter 02 Dec '09

02 Dec '09

I just sent a message to my freshly installed MM 3 alpha 4 (on Ubuntu 9.04) and got the following warning: Nov 29 23:53:51 mailman postfix/lmtp[8620]: D409D32E12: to=<test(a)mailman.state-of-mind.de>, relay=127.0.0.1[127.0.0.1]:8025, delay=1.8, delays=0.07/0.01/0/1.7, dsn=2.0.0, status=sent (250 Ok) Nov 29 23:53:51 mailman postfix/qmgr[8613]: D409D32E12: removed /root/mailman/src/mailman/rules/administrivia.py:84: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if line == '': The delivery worked though. p@rick -- state of mind Digitale Kommunikation http://www.state-of-mind.de Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666 Amtsgericht München Partnerschaftsregister PR 563

4 6

Mailman 3 and LMTP
by Barry Warsaw 30 Nov '09

30 Nov '09

Hi everybody, I'm getting very near ready to release another alpha of Mailman 3, and on the prompting of a private message from Robert Niederreiter I took some time to fire up a VM and actually try Postfix+LMTP delivery in an experimental production system. I'd like to get some feedback from the Postfix experts in the crowd so that we can update the wiki page here: http://wiki.list.org/display/DEV/LMTP+process FTR: the VM is running Ubuntu Karmic Koala 9.10 server with Postfix 2.6.5 and Mailman 3.0 from the lp:mailman bzr trunk. To start with, I mostly followed the instructions in the wiki. I tried using the standard lmtp transport in master.cf, creating /etc/postfix/mailman_lists as described in the wiki (not the dedicated list server example, but the earlier one on the page). I actually used: # Key # Value test(a)xxx.example.com lmtp:xxx.example.com where 'xxx.example.com' is replaced by my VM's host name. The first gotcha is that unless you start Mailman as root, you can't bind its LMTP server to port 24, which is Postfix's default. By default Mailman's LMTP server listens on localhost:8024, so you either need to append ':8024' on the Value above, or set lmtp_tcp_port = 8024 in Postfix's main.cf. I ended up doing the latter, but both work. I set up transport_maps and local_recipient_maps just as outlined in the wiki page, fired everything up, and then sent a message to the VM's Postfix while tailing the logs. I kept seeing "Connection refused" messages from Postfix and it never hit Mailman's LMTP server. This was highly confusing because I could see in the Postfix logs that it was finding the right IP address for its own hostname and it was trying the right port number. Mailman's lmtp.log file claimed it was listening on the right IP and port, and I could even telnet to it just fine. So what was Postfix doing? It seems that Karmic is playing tricks with /etc/hosts. I've got DNS set up to correctly resolve the VM's hostname, but there was actually an entry in /etc/hosts that set xxx.example.com to 127.0.1.1. I'm not entirely sure which process looks at what, when, but clearly this inconsistency was a key part of the problem. The other thing that surprised me was that Postfix was also consulting /var/spool/postfix/etc/hosts, and I had to change both /etc/hosts and that file as well, so that the IP addresses jived with DNS. This is unsatisfactory, but it seemed critical to getting things working. The last piece of the puzzle was to not use Postfix's standard lmtp server in master.cf, but to define a new one like so: mailman3 unix - - - - - lmtp -o disable_dns_lookups=yes and then change the mailman_lists transport map to # Key # Value test(a)xxx.example.com mailman3:xxx.example.com After a restart, everything suddenly worked exactly as expected. Robert was having a different problem, but hopefully he will follow up here with his experience and let us know if any of the above helps. If any Postfix experts have words of wisdom to make this better, please let me know. I probably need to work on better dropping of privileges for qrunners so that you can 'bin/mailman start' as root, and once the LMTP runner binds to port 24, it can drop privs to 'mailman'. I'll put that on the list for something to do after the next alpha. I'd also like to try to resurrect William Mead's LMTP branch. Sadly, it won't merge cleanly into trunk any more (not the least of which because it's in the wrong bzr format). If anybody would like to contribute that before I get to it, I'd be grateful. The good news is that I think Mailman 3 is getting more real every day. My plan over the next few weeks is: * release alpha 4 * get i18n translations working * complete the split of the pipermail project * hopefully get Patrick and friends going on the web u/i * announce something <wink> * create a live test list that anybody can subscribe to If you've been waiting to play with Mailman 3, wait no longer! -Barry

3 8

Mailman and Submission port
by Patrick Ben Koetter 30 Nov '09

30 Nov '09

MM developers, I'd like to propose a change in MM3s default SMTP client port from port 25 (transport) to port 587 (submission). Why? From my point of view mailman rather is a mail component that introduces messages into a mail system than one that sits between MTAs and assists in transporting messages that pass by. RFC 4409 <http://www.rfc-editor.org/rfc/rfc4409.txt> explicitly defines a submission port (587) for mail systems whose purpose is to accept message from MUAs: However, SMTP is now also widely used as a message *submission* protocol, that is, a means for Message User Agents (MUAs) to introduce new messages into the MTA routing network. The process that accepts message submissions from MUAs is termed a Message Submission Agent (MSA). Apart from doing 'the right thing' what would be the benefit? The RFC gives some ideas in a later section: (...) Even when submitted messages are complete, local site policy may dictate that the message text be examined or modified in some way, e.g., to conceal local name or address spaces. Such completions or modifications have been shown to cause harm when performed by downstream MTAs -- that is, MTAs after the first-hop submission MTA -- and are in general considered to be outside the province of standardized MTA functionality. >From my daily work with mailman the following "modified in some way"-tasks come to my mind immediately: - apply client and content policy that differs from the port 25 anti-spam policy - add DKIM signatures because it is clear mailman messages are ORIGINATING from our network What would we have to do, to make port 587 the default port? In section 4 the RFC says, a MSA MUST do all of the following: 1. General Submission Rejection Code 2. Ensure All Domains Are Fully-Qualified 3. Require Authentication To cut it short: 1. and 2. are trivial (at least in Postfix and I don't know the others MTAs well enough to tell for them too). 3. requires to add SMTP AUTH functionality to Mailman's SMTP client. How should we implement SMTP AUTH in the MM SMTP client? I propose for a start plaintext (PLAIN, LOGIN) and shared-secret mechanisms (CRAM-MD5, DIGEST-MD5) should be added to the SMTP client. Those are the ones used most widely in every day SMTP AUTH. Later implementations could add GSSAPI and EXTERNAL. If plaintext mechanisms are added we should also consider to add STARTTLS functionality to MM's SMTP client to shield credentials while they are sent in a plaintext authentication session. p@rick -- state of mind Digitale Kommunikation http://www.state-of-mind.de Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666 Amtsgericht München Partnerschaftsregister PR 563

5 9

Mail of subscribe to mailing list with multiple users
by peibel80 19 Nov '09

19 Nov '09

Hi, I´m new with Maiman. I need make mail of subscription with multiple users, Is possible make a mail with name-request(a)sudominio.com with several lines of subscribe [contraseña] [digest|nodigest] [address=] command if is yes, I need a example, if is not possible, as can be done? thanks, Sorry for my inglish -- View this message in context: http://old.nabble.com/Mail-of-subscribe-to-mailing-list-with-multiple-users… Sent from the Mailman - Dev mailing list archive at Nabble.com.

2 1

GNU Mailman roadmap
by Barry Warsaw 12 Nov '09

12 Nov '09

As you know, Mailman 2.1 has long been in maintenance-only mode. Mailman 2.2 was where we were going to add new features and update the user interface, without changing the basic model. Mailman 3 was where we were going to fix the model and modernize the architecture to allow for better embedded use. Mark has been doing an incredible job fixing Mailman 2.1, and forward porting these fixes to Mailman 2.2. I have been working on Mailman 3 and have released several alphas. The current state of affairs is not ideal though. Neither 2.2 nor 3.0 has been released, there is confusion in the community as to which version to develop patches for, and frustration on our part that we have divided efforts and not as much community participation as we'd like. Mark and I have decided therefore to combine our efforts under Mailman 3, and we invite you to join us. Working together, I feel confident that we can have a solid release of Mailman 3 very soon, hopefully by the end of the year. Patrick Koetter and his group have expressed interest and resources in helping jump start the new Mailman user interface, which will be built on top of Mailman 3's REST interface. What do /you/ want to work on? :) Here's the plan: Mark is going to put a 2.1.13 bug fix release out soon and will continue to fix only the most important bugs on the 2.1 branch. He'll forward port those fixes to the 2.2 branch for the few people who are running it from source, but there will never be a Mailman 2.2 release. For all practical purposes, Mailman 2.2 is dead. Mark will be joining me to focus all new development work on Mailman 3.0. I hope this brings clarity to where we're going, and I hope that the renewed and concentrated efforts will encourage you to pull down the Mailman 3.0 code or alphas and begin testing and developing for it. Enjoy, -Barry

4 5

Network is unreachable
by Gerheim 11 Nov '09

11 Nov '09

Hi, I've terminated to configure my server but when I create a list, the welcome message stops on queue. Analysing /var/log/mailman/smtp-failure I get: Oct 29 09:04:29 2009 (18551) delivery to test(a)xxx.xxx.xxx failed with code -1: (101, 'Network is unreachable') Could anyone help? Thanks Wallace Gerheim

3 2

FR: obscure email address in archives
by Joost Schuttelaar 11 Nov '09

11 Nov '09

When the obscure email address feature in the admin settings is enabled, e-mail addresses are transformed into the format 'xxx AT xxx' in the archives. This may have worked 5 years ago to prevent spam harvesting. But it does not work anymore. I've patched (well... butchered) my own install by modifying HyperArch.py to simply output 'EMAIL HIDDEN' instead of the obscured e- mail address, by replacing the regexes to a constant string at line 284. I think a 'proper' feature like this would be a very welcome addition. Right now, it would be trivial to write a script to harvest tens of thousands of e-mail addresses from Mailman users... -- Joost Schuttelaar The Hague, NL

3 2

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

Mailman-Developers November 2009