ANNOUNCE: GNU Mailman 3.0a1 (Leave That Thing Alone)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
After far too long, I'm finally happy to announce the availability of
GNU Mailman version 3.0 alpha 1, code name "Leave That Thing Alone".
Hopefully, I needn't point out that this version is not suitable for
production. Instead, I'm making it available to provide an early
snapshot of where Mailman 3 is going, and to give interested
developers, users and integrators something real to play with. I am
looking forward to your feedback and contributions, and I hope you
will at least download the package and give what works a try.
For now, I would prefer if all discussion about 3.0a1 were held on the
mailman-developers mailing list. Please don't submit bug reports to
SourceForge; it's far too early for that. Ideally, you'll be
interested enough to create your own branches and push them to
Launchpad for all of us to look at.
For detailed information on 3.0a1, please read docs/ALPHA.txt. This
will explain what works and what doesn't, how to run the test suite
and how to actually create, populate, and deliver to some mailing
lists. Much does not yet work, but everything that does work is very
well documented using Python's doctest framework.
Please note that Python 2.5 is required, but a C compiler is no longer
needed. You will need some third party packages, but everything you
need is available in the Python Cheeseshop.
You can download both the tarball and egg file for 3.0a1 from the
Cheeseshop as well:
http://pypi.python.org/pypi/mailman/3.0.0a1
You'll probably want the source tarball.
I hope you will find this first snapshot useful and encouraging.
Please participate!
Enjoy,
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkf8JbMACgkQ2YZpQepbvXGtxgCguOrVEuZ61VnCe86nwkWHXGBm b7sAoLn2fUbRLDmRdWusYxfsAu21FAxw =eiPc -----END PGP SIGNATURE-----

--On 8 April 2008 22:10:58 -0400 Barry Warsaw <barry@list.org> wrote:
After far too long, I'm finally happy to announce the availability of GNU Mailman version 3.0 alpha 1, code name "Leave That Thing Alone".
This is great news.
I've downloaded the code, and had a look at the LMTP server code, which I'm looking forward to. I'm hoping that it will permit integration with Exim in a way (LMTP call forwards) that gives us something sensible to do with messages from forbidden posters.
Unfortunately, it doesn't. But, if the following seems sensible, then I'm willing to implement it:
The LMTP server is implemented as a subclass of Python's built in SMTPServer, which seems to have only one method: process_message. That must be run too late to give any useful information to a callforward. However, on closer inspection, there's a "channel" class which inherits from smtpd.SMTPChannel. This seems to override some useful looking methods:
def smtp_LHLO(self, arg):
def smtp_HELO(self, arg):
Ah, and the source suggests to me that overriding smtp_RCPT is feasible. That's the place to check whether the list exists (and whether the sender is permitted to post - but let's do one thing at a time).
Validating the recipient
All that's required is that the code which checks the existence of a list is moved from process_message to smtp_RCPT, and it's modified to "self.__rcpttos.append(address)" when the address is valid, but not otherwise. Thus, we build rcpttos instead of reading from it. LMTP requires a reply after DATA for only those addresses that haven't been rejected at RCPT. If I'm reading the code correctly, then a 550 error would be generated if the queue disappeared between RCTP and DATA (by virtue of the exception being thrown by queue.enqueue().
Validating the sender
Now, as to the question of determining whether the message sender is permitted to post to a list, that's a bit more complicated. As far as I can see, all the logic for this is in moderate.py - in the "process" function. But, the decision is all tangled up with with the actions.
I propose an additional function, to be called from process, or by the lmtp server in smtp_RCPT: perhaps post_action(list, sender) which returns the name of the action to take when "sender" posts to the list "list". I guess it will return one of "accept", "discard", "reject", or "hold", and the lmtp server should reject (not bounce) the message if "reject" or "discard" is returned. It might be nice to introduce a distinction between bounce and reject. For example, a site admin might decide to permit bounce messages to be generated for local mail domains.
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 9, 2008, at 2:42 PM, Ian Eiloart wrote:
Hi Ian, I'm really psyched that you've taken a look and are interested
in fleshing this part of LMTP out. I didn't have time for it before
getting 3.0a1 out, but I did think about it, so let me outline how I
was thinking of doing it. I'm definitely open to other approaches if
it makes the feature work better.
IIRC, this is going to be called when the "\r\n.\r\n" terminator to
the DATA command is seen. That might be an interesting place to hook
in content filtering.
We probably want to refactor the code that splits the recipient
address into a separate method, this would validate the fqdn list name
and subaddress, but it wouldn't instantiate a Switchboard object.
Might as well skip that if the message isn't going to get delivered.
If everything looks good, store the fqdn listname and subaddress in
self._rcpttos (no need for double underscore) and then
process_message() can iterate over that and create the Switchboard
instance. If the recipient is not valid, store a marker object
instead so that it can be compared against in the self._rcpttos loop
in process_message(). When that loop sees the marker object, it
appends an ERR_550 without bothering to create the Switchboard instance.
That's not the way I was thinking about doing it. Although you can't
really tell, pipeline/moderate.py is obsolete and will likely go
away. It doesn't fit the current model where moderation occurs by
rule-chain processing in the incoming queue, and the pipeline queue
runs handlers which only modify already accepted messages. I really
should have moved moderate.,py to an attic because there are bits I
still want to convert, but just haven't gotten to yet.
Originally, I was thinking we could just define an IChain which could
be used in the lmtp runner. This new chain would be specifically
defined to run a smaller set of rules that would validate the sender,
or do other early filtering. But there's a problem with that because
the interface to IChain and IRule takes three arguments, the mailing
list, the message, and the message metadata dictionary.
One problem is that until you get to process_message() -- i.e. you've
seen the DATA terminating dot), you won't have a message object, and
many of the rules you're going to want require a message object. You
could get around that by defining some custom rules that ignored the
'msg' argument, and you could put everything you know so far in the
msgdata dictionary. These rules would only work then in the lmtp
runner.
For the mlist argument, you probably need to run the rule-chain for
each RCPT TO. Because you've already determined that it's a valid
mailing list above, you get the mailing list object and use that in
the rule chain. It would therefore be possible to reject the message
for list A but accept it for list B. I think you want to return a 553
if you're going to reject the recipient.
Upshot is that in thinking about this, I think it's doable using the
rule-chain architecture during the smpt_RCPT call.
Does the above make sense? If you decide to work on it, I'd be happy
to help answer ore specific questions, and of course review the code.
I'm not opposed to making some architectural changes to suit this
better, but I'd want to think about them carefully, and would prefer
to fit it into the current model. We'd also need doctest for the
feature and any other changes you make.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkf+0oEACgkQ2YZpQepbvXH8SwCeL86BhmoIrur6Kbbapt8pMkJt gsoAn0v95kRIfMczhTJb9qu+Ighbha+K =JNvC -----END PGP SIGNATURE-----

--On 10 April 2008 22:52:49 -0400 Barry Warsaw <barry@list.org> wrote:
Yes, that's the earliest opportunity to filter content. It also allows the LTMP server to accept the message for some lists, but not others. It doesn't help the local SMTP server, though, since it's too late for an SMTP server to selectively reject.
Some admins may prefer this though. For example, I monitor my SMTP server queues more closely than my Mailman queues.
Hmm, I don't know what a Switchboard object is.
No, I don't think that's necessary. If we don't like the argument to RCPT we should say so with a 4xx or 5xx error, and then add NOTHING to self._rcpttos. The RFC only wants us to give a reply for addresses that made it past RCPT. It's crucial to get this right, because the returned list of codes isn't indexed, and therefore has to be matched up with the list of RCTP commands that didn't get a rejection.
If you look at the example here: <http://www.apps.ietf.org/rfc/rfc2033.html#sec-4.2>, there are three RCPT commands, but only two post-DATA responses.
Of course, we could say yes to everything at RCTP, but then our SMTP server callouts are pointless.
Or, could you pass an empty message? I guess not if a list is configured to reject implicit recipients, for example.
That's good news.
Does the above make sense?
To an extent. I know now that I was wise to ask before ploughing on!
The first part seems quite easy to do, and has the significant gain for me that it allows me to put my Mailman server on a different host from my MTA. Currently, my MTA is looking for the existence of the Mailman files for routing purposes. Calling out to the LMTP server is a much more flexible solution.
The second part - screening senders seems harder at the moment, because I don't understand how the chains work. I'll take another look, but a brief explanation of the concept, and how they're called would be useful.
Hmm, well if we can't do stuff at RCPT time, then I think the architecture is flawed. Whether its reasonably easy to fix, I can't tell.
I think it's really important to separate out the code for determining whether a sender can post to a list. In my experience, it's the only test that (a) users really understand and (b) is really useful.
It's also important to because it helps us to decide whether we are safe to generate a bounce message, in the event that the message content is bad. I'd argue that it usually is safe when the sender is a member of the list, but usually not otherwise. That still leaves us with a problem for open lists, but no worse a problem than we already have.
Oh, I guess I'll have to do some proper learning, then!
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 11, 2008, at 9:30 AM, Ian Eiloart wrote:
So you mean that most SMTPd's will have accepted the message by they
time they issue the DATA to the LMTP? If that's the case, then does
it even make sense to do content filtering at that step in LMTP, given
that we'll still be doing the full-blown rule matching in the incoming
runner?
The Switchboard object represents a queue's directory. It's the class
that handles enqueuing and dequeuing message pickles from that
directory.
Yep. I wasn't thinking clearly when I wrote the above. You're
absolutely right.
Actually, this occurred to me to, and I do think it would work. You
have limited information at that point, but enough to create a bare-
bones Message instance that can satisfy the interface. You just have
to be careful not to put any rules into that chain that require
information you don't have at that point, but that should be fairly
easy to do.
All this is in the doctests (that's the beauty of testable
documentation! :), but briefly:
You have rules, which have a check() method. This method takes a
mailing list, a message, and a metadata dictionary and it returns a
boolean specifying whether the rule matched or not. Rules are
organized into chains, where each element of the chain is actually a
"chain link". The chain link ties a rule with an action.
Processing a chain is then just iterating over all the chain links,
executing the rule in the link, and if the rule returns True,
processing the action. The actions are things like "jump to another
chain", "take a detour through another chain", "stop processing this
chain", "run some function".
So the idea is that we'll define a rule (or rules -- they should be as
narrow as possible) that implements the "verify if this RCPT TO is
acceptable", and an action that has a callable that says "set the RCPT
TO return code", which is probably stored in the metadata dictionary
There's probably a built-in such chain, and it may be YAGNI to allow
per-list overrides.
So, you get the RCPT TO, you process the chain, grab the return code
out of the metadata dictionary. If it's a 250, you remember the
recipient for later, and if not, you chuck it.
See above, I think we can.
It probably does make sense to refactor this out. I can imagine the
REST interface is going to want to ask the same question.
There's something else that Mailman 3 allows us to do. We can ask
whether we know anything about the sender or not. So she may not be a
member of List A, but maybe she's a member of List B. So when she
posts to List A, we at least know that the sender is a real person.
Maybe we send her a bounce, maybe we don't, but I think knowing that
the site has a relationship with the sender can come in handy.
I hope the above helps! Let me know if any the doctests don't make
sense as documentation.
Good luck!
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgA0rQACgkQ2YZpQepbvXEbPQCeNBockJ6jmb9nugeGxT3t6017 Oo0An0SnGNW8Sn4+O9yfbZ9NGPO/sRFh =ljz2 -----END PGP SIGNATURE-----

Barry Warsaw wrote:
I don't think that's what Ian is saying. The key word is 'selectively'. I.e. the SMTP server can still reject the incoming message, but once it has seen the DATA, it can only reject the entire message; it can't reject a subset of RCPTs.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 12, 2008, at 3:06 PM, Mark Sapiro wrote:
Ah, right, on the SMTP side. LMTP can reject individual RCPTs, but
the SMTP upstream of it can't.
Thanks,
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgBC9cACgkQ2YZpQepbvXGxpwCgnIYYZNTyI0JGRyJ6xCbfCwxU FJIAnju+uzmG6QKjPHbFRIvGM0Ol7vMH =fPWI -----END PGP SIGNATURE-----

--On 12 April 2008 12:06:06 -0700 Mark Sapiro <mark@msapiro.net> wrote:
Yes, that much is true in theory. In practice, it's even worse than that. Exim really has accepted the email before it attempts delivery. That's going to be true for any MTA that has a queueing mechanism (qmail, for example), and perhaps others, too.
-- Ian Eiloart IT Services, University of Sussex x3148

Mark Sapiro wrote:
There is http://www.courier-mta.org/draft-varshavchik-exdata-smtpext.txt .
Not very widely implemented, though, AFAICT. :-(

--On 12 April 2008 11:18:09 -0400 Barry Warsaw <barry@list.org> wrote:
Exim will have, almost certainly. Although it's possible for Exim to run any perl script before sending the response to DATA, or even call any external program, there are three reasons why it probably won't:
- It requires some coding to create the script.
- The script would have to substantially replicate Exim's routing and delivery code.
- It's relatively expensive to run such scripts.
- It's already too late to selectively reject the message when there is more than one recipient, because we've passed the SMTP RCTP stage. If only email were all done by LMTP!
I don't know about other MTAs.
It's of very limited use. I'm not convinced that my users use it at all. If a message passes the sender filter, then it's probably OK to generate a bounce message. The exception would be for an open list, but an open list that gets lots of spam probably isn't going to last very long.
My view is that content filtering for spam belongs in the MTA, which is much better equipped to deal with it. Rejecting spam at SMTP time has the huge advantage that spambots can learn to leave you alone. I don't know if they do, though. Otherwise, you can teergrube them (add delays to the smtp replies, to slow them down).
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 14, 2008, at 8:07 AM, Ian Eiloart wrote:
Okay. BTW, wouldn't you think that the majority of messages sent to a
Mailman list will have exactly one RCPT? Cross-posting is relatively
rare for most sites, isn't it?
I agree. Okay, let's code for the common/useful case and leave the
rest for YAGNI.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgDWiwACgkQ2YZpQepbvXEerQCgh7XTEkUaTfUz6HiApyqkvWIo 7e4AnRY1Sj/LH46xPeompxWL4WqF7HJc =xpgz -----END PGP SIGNATURE-----

--On 14 April 2008 09:20:43 -0400 Barry Warsaw <barry@list.org> wrote:
Check the headers of this message! Two recipients.
OK, that might be OK if python.org and list.org are handled by different servers. On our site, though, we have thousands of users and hundreds of lists in the same domain. Most of our list members are in that same domain. So, replies to list messages will frequently have more than one RCPT.
Oh, and we make extensive use of umbrella lists, which might also make a difference. In fact, we have a complex heirarchy of lists which we'd like to develop further.
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 14, 2008, at 10:44 AM, Ian Eiloart wrote:
They are ;)
Okay, that makes sense. BTW, I'd love to get your feedback on the
roster concept in MM3. I /think/ it will make composable mailing list
members much easier to deal with, but the ideas could probably still
use some fleshing out.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgDe+QACgkQ2YZpQepbvXE/2QCdEDgvSaqiRe8B9lAtc7V+WPDN YywAn0loB4ak20Oy2/qki/IwovCc39Fm =1z35 -----END PGP SIGNATURE-----

Just another data point: I don't have experience at most sites, but here cross-posting is extremely common.
For example, we have hundreds of lists just for different staff cross-sections, and hundreds for student interest groups with some common themes among them. If someone wants to address people in two different working groups who are collaborating on a short project, their mail will quite likely go to each of those groups. It's not worth creating a new list for the few days that the project requires.
Another very common use case: one student group puts on an art show, and they want to call the attention of other student groups who have an arty focus. That announcement might go to 20 lists.
-- -D. dgc@uchicago.edu NSIT University of Chicago

Barry Warsaw writes:
It's not just cross-posting, though. Consider this message, except assume that you posted from your Python address. Open source lists may be unusual, but (as one additional data point) the lists at my work generally get lots of wide replies containing local users' mailboxes.

--On 8 April 2008 22:10:58 -0400 Barry Warsaw <barry@list.org> wrote:
After far too long, I'm finally happy to announce the availability of GNU Mailman version 3.0 alpha 1, code name "Leave That Thing Alone".
This is great news.
I've downloaded the code, and had a look at the LMTP server code, which I'm looking forward to. I'm hoping that it will permit integration with Exim in a way (LMTP call forwards) that gives us something sensible to do with messages from forbidden posters.
Unfortunately, it doesn't. But, if the following seems sensible, then I'm willing to implement it:
The LMTP server is implemented as a subclass of Python's built in SMTPServer, which seems to have only one method: process_message. That must be run too late to give any useful information to a callforward. However, on closer inspection, there's a "channel" class which inherits from smtpd.SMTPChannel. This seems to override some useful looking methods:
def smtp_LHLO(self, arg):
def smtp_HELO(self, arg):
Ah, and the source suggests to me that overriding smtp_RCPT is feasible. That's the place to check whether the list exists (and whether the sender is permitted to post - but let's do one thing at a time).
Validating the recipient
All that's required is that the code which checks the existence of a list is moved from process_message to smtp_RCPT, and it's modified to "self.__rcpttos.append(address)" when the address is valid, but not otherwise. Thus, we build rcpttos instead of reading from it. LMTP requires a reply after DATA for only those addresses that haven't been rejected at RCPT. If I'm reading the code correctly, then a 550 error would be generated if the queue disappeared between RCTP and DATA (by virtue of the exception being thrown by queue.enqueue().
Validating the sender
Now, as to the question of determining whether the message sender is permitted to post to a list, that's a bit more complicated. As far as I can see, all the logic for this is in moderate.py - in the "process" function. But, the decision is all tangled up with with the actions.
I propose an additional function, to be called from process, or by the lmtp server in smtp_RCPT: perhaps post_action(list, sender) which returns the name of the action to take when "sender" posts to the list "list". I guess it will return one of "accept", "discard", "reject", or "hold", and the lmtp server should reject (not bounce) the message if "reject" or "discard" is returned. It might be nice to introduce a distinction between bounce and reject. For example, a site admin might decide to permit bounce messages to be generated for local mail domains.
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 9, 2008, at 2:42 PM, Ian Eiloart wrote:
Hi Ian, I'm really psyched that you've taken a look and are interested
in fleshing this part of LMTP out. I didn't have time for it before
getting 3.0a1 out, but I did think about it, so let me outline how I
was thinking of doing it. I'm definitely open to other approaches if
it makes the feature work better.
IIRC, this is going to be called when the "\r\n.\r\n" terminator to
the DATA command is seen. That might be an interesting place to hook
in content filtering.
We probably want to refactor the code that splits the recipient
address into a separate method, this would validate the fqdn list name
and subaddress, but it wouldn't instantiate a Switchboard object.
Might as well skip that if the message isn't going to get delivered.
If everything looks good, store the fqdn listname and subaddress in
self._rcpttos (no need for double underscore) and then
process_message() can iterate over that and create the Switchboard
instance. If the recipient is not valid, store a marker object
instead so that it can be compared against in the self._rcpttos loop
in process_message(). When that loop sees the marker object, it
appends an ERR_550 without bothering to create the Switchboard instance.
That's not the way I was thinking about doing it. Although you can't
really tell, pipeline/moderate.py is obsolete and will likely go
away. It doesn't fit the current model where moderation occurs by
rule-chain processing in the incoming queue, and the pipeline queue
runs handlers which only modify already accepted messages. I really
should have moved moderate.,py to an attic because there are bits I
still want to convert, but just haven't gotten to yet.
Originally, I was thinking we could just define an IChain which could
be used in the lmtp runner. This new chain would be specifically
defined to run a smaller set of rules that would validate the sender,
or do other early filtering. But there's a problem with that because
the interface to IChain and IRule takes three arguments, the mailing
list, the message, and the message metadata dictionary.
One problem is that until you get to process_message() -- i.e. you've
seen the DATA terminating dot), you won't have a message object, and
many of the rules you're going to want require a message object. You
could get around that by defining some custom rules that ignored the
'msg' argument, and you could put everything you know so far in the
msgdata dictionary. These rules would only work then in the lmtp
runner.
For the mlist argument, you probably need to run the rule-chain for
each RCPT TO. Because you've already determined that it's a valid
mailing list above, you get the mailing list object and use that in
the rule chain. It would therefore be possible to reject the message
for list A but accept it for list B. I think you want to return a 553
if you're going to reject the recipient.
Upshot is that in thinking about this, I think it's doable using the
rule-chain architecture during the smpt_RCPT call.
Does the above make sense? If you decide to work on it, I'd be happy
to help answer ore specific questions, and of course review the code.
I'm not opposed to making some architectural changes to suit this
better, but I'd want to think about them carefully, and would prefer
to fit it into the current model. We'd also need doctest for the
feature and any other changes you make.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkf+0oEACgkQ2YZpQepbvXH8SwCeL86BhmoIrur6Kbbapt8pMkJt gsoAn0v95kRIfMczhTJb9qu+Ighbha+K =JNvC -----END PGP SIGNATURE-----

--On 10 April 2008 22:52:49 -0400 Barry Warsaw <barry@list.org> wrote:
Yes, that's the earliest opportunity to filter content. It also allows the LTMP server to accept the message for some lists, but not others. It doesn't help the local SMTP server, though, since it's too late for an SMTP server to selectively reject.
Some admins may prefer this though. For example, I monitor my SMTP server queues more closely than my Mailman queues.
Hmm, I don't know what a Switchboard object is.
No, I don't think that's necessary. If we don't like the argument to RCPT we should say so with a 4xx or 5xx error, and then add NOTHING to self._rcpttos. The RFC only wants us to give a reply for addresses that made it past RCPT. It's crucial to get this right, because the returned list of codes isn't indexed, and therefore has to be matched up with the list of RCTP commands that didn't get a rejection.
If you look at the example here: <http://www.apps.ietf.org/rfc/rfc2033.html#sec-4.2>, there are three RCPT commands, but only two post-DATA responses.
Of course, we could say yes to everything at RCTP, but then our SMTP server callouts are pointless.
Or, could you pass an empty message? I guess not if a list is configured to reject implicit recipients, for example.
That's good news.
Does the above make sense?
To an extent. I know now that I was wise to ask before ploughing on!
The first part seems quite easy to do, and has the significant gain for me that it allows me to put my Mailman server on a different host from my MTA. Currently, my MTA is looking for the existence of the Mailman files for routing purposes. Calling out to the LMTP server is a much more flexible solution.
The second part - screening senders seems harder at the moment, because I don't understand how the chains work. I'll take another look, but a brief explanation of the concept, and how they're called would be useful.
Hmm, well if we can't do stuff at RCPT time, then I think the architecture is flawed. Whether its reasonably easy to fix, I can't tell.
I think it's really important to separate out the code for determining whether a sender can post to a list. In my experience, it's the only test that (a) users really understand and (b) is really useful.
It's also important to because it helps us to decide whether we are safe to generate a bounce message, in the event that the message content is bad. I'd argue that it usually is safe when the sender is a member of the list, but usually not otherwise. That still leaves us with a problem for open lists, but no worse a problem than we already have.
Oh, I guess I'll have to do some proper learning, then!
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 11, 2008, at 9:30 AM, Ian Eiloart wrote:
So you mean that most SMTPd's will have accepted the message by they
time they issue the DATA to the LMTP? If that's the case, then does
it even make sense to do content filtering at that step in LMTP, given
that we'll still be doing the full-blown rule matching in the incoming
runner?
The Switchboard object represents a queue's directory. It's the class
that handles enqueuing and dequeuing message pickles from that
directory.
Yep. I wasn't thinking clearly when I wrote the above. You're
absolutely right.
Actually, this occurred to me to, and I do think it would work. You
have limited information at that point, but enough to create a bare-
bones Message instance that can satisfy the interface. You just have
to be careful not to put any rules into that chain that require
information you don't have at that point, but that should be fairly
easy to do.
All this is in the doctests (that's the beauty of testable
documentation! :), but briefly:
You have rules, which have a check() method. This method takes a
mailing list, a message, and a metadata dictionary and it returns a
boolean specifying whether the rule matched or not. Rules are
organized into chains, where each element of the chain is actually a
"chain link". The chain link ties a rule with an action.
Processing a chain is then just iterating over all the chain links,
executing the rule in the link, and if the rule returns True,
processing the action. The actions are things like "jump to another
chain", "take a detour through another chain", "stop processing this
chain", "run some function".
So the idea is that we'll define a rule (or rules -- they should be as
narrow as possible) that implements the "verify if this RCPT TO is
acceptable", and an action that has a callable that says "set the RCPT
TO return code", which is probably stored in the metadata dictionary
There's probably a built-in such chain, and it may be YAGNI to allow
per-list overrides.
So, you get the RCPT TO, you process the chain, grab the return code
out of the metadata dictionary. If it's a 250, you remember the
recipient for later, and if not, you chuck it.
See above, I think we can.
It probably does make sense to refactor this out. I can imagine the
REST interface is going to want to ask the same question.
There's something else that Mailman 3 allows us to do. We can ask
whether we know anything about the sender or not. So she may not be a
member of List A, but maybe she's a member of List B. So when she
posts to List A, we at least know that the sender is a real person.
Maybe we send her a bounce, maybe we don't, but I think knowing that
the site has a relationship with the sender can come in handy.
I hope the above helps! Let me know if any the doctests don't make
sense as documentation.
Good luck!
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgA0rQACgkQ2YZpQepbvXEbPQCeNBockJ6jmb9nugeGxT3t6017 Oo0An0SnGNW8Sn4+O9yfbZ9NGPO/sRFh =ljz2 -----END PGP SIGNATURE-----

Barry Warsaw wrote:
I don't think that's what Ian is saying. The key word is 'selectively'. I.e. the SMTP server can still reject the incoming message, but once it has seen the DATA, it can only reject the entire message; it can't reject a subset of RCPTs.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 12, 2008, at 3:06 PM, Mark Sapiro wrote:
Ah, right, on the SMTP side. LMTP can reject individual RCPTs, but
the SMTP upstream of it can't.
Thanks,
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgBC9cACgkQ2YZpQepbvXGxpwCgnIYYZNTyI0JGRyJ6xCbfCwxU FJIAnju+uzmG6QKjPHbFRIvGM0Ol7vMH =fPWI -----END PGP SIGNATURE-----

--On 12 April 2008 12:06:06 -0700 Mark Sapiro <mark@msapiro.net> wrote:
Yes, that much is true in theory. In practice, it's even worse than that. Exim really has accepted the email before it attempts delivery. That's going to be true for any MTA that has a queueing mechanism (qmail, for example), and perhaps others, too.
-- Ian Eiloart IT Services, University of Sussex x3148

Mark Sapiro wrote:
There is http://www.courier-mta.org/draft-varshavchik-exdata-smtpext.txt .
Not very widely implemented, though, AFAICT. :-(

--On 12 April 2008 11:18:09 -0400 Barry Warsaw <barry@list.org> wrote:
Exim will have, almost certainly. Although it's possible for Exim to run any perl script before sending the response to DATA, or even call any external program, there are three reasons why it probably won't:
- It requires some coding to create the script.
- The script would have to substantially replicate Exim's routing and delivery code.
- It's relatively expensive to run such scripts.
- It's already too late to selectively reject the message when there is more than one recipient, because we've passed the SMTP RCTP stage. If only email were all done by LMTP!
I don't know about other MTAs.
It's of very limited use. I'm not convinced that my users use it at all. If a message passes the sender filter, then it's probably OK to generate a bounce message. The exception would be for an open list, but an open list that gets lots of spam probably isn't going to last very long.
My view is that content filtering for spam belongs in the MTA, which is much better equipped to deal with it. Rejecting spam at SMTP time has the huge advantage that spambots can learn to leave you alone. I don't know if they do, though. Otherwise, you can teergrube them (add delays to the smtp replies, to slow them down).
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 14, 2008, at 8:07 AM, Ian Eiloart wrote:
Okay. BTW, wouldn't you think that the majority of messages sent to a
Mailman list will have exactly one RCPT? Cross-posting is relatively
rare for most sites, isn't it?
I agree. Okay, let's code for the common/useful case and leave the
rest for YAGNI.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgDWiwACgkQ2YZpQepbvXEerQCgh7XTEkUaTfUz6HiApyqkvWIo 7e4AnRY1Sj/LH46xPeompxWL4WqF7HJc =xpgz -----END PGP SIGNATURE-----

--On 14 April 2008 09:20:43 -0400 Barry Warsaw <barry@list.org> wrote:
Check the headers of this message! Two recipients.
OK, that might be OK if python.org and list.org are handled by different servers. On our site, though, we have thousands of users and hundreds of lists in the same domain. Most of our list members are in that same domain. So, replies to list messages will frequently have more than one RCPT.
Oh, and we make extensive use of umbrella lists, which might also make a difference. In fact, we have a complex heirarchy of lists which we'd like to develop further.
-- Ian Eiloart IT Services, University of Sussex x3148

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 14, 2008, at 10:44 AM, Ian Eiloart wrote:
They are ;)
Okay, that makes sense. BTW, I'd love to get your feedback on the
roster concept in MM3. I /think/ it will make composable mailing list
members much easier to deal with, but the ideas could probably still
use some fleshing out.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkgDe+QACgkQ2YZpQepbvXE/2QCdEDgvSaqiRe8B9lAtc7V+WPDN YywAn0loB4ak20Oy2/qki/IwovCc39Fm =1z35 -----END PGP SIGNATURE-----
participants (6)
-
Barry Warsaw
-
David Champion
-
Ian Eiloart
-
Julian Mehnle
-
Mark Sapiro
-
Stephen J. Turnbull