Mailman 3 May 2006 - Mailman-Developers

Topic regexps
by Mark Sapiro 03 Jun '06

03 Jun '06

There is a problem with Topic regexps. The regexp entry is compiled in re.VERBOSE mode. This can be good, but also causes problems as the 'help' doesn't mention this. Further, the description says 'Topic keywords, one per line, to match against each message.' The implication is if I put one two three in the regexp box, that this topic will match keywords 'one', 'two' or 'three', but actually it matches only 'onetwothree'. I see several ways to address this. 1) change the processing of this field to effectively join the lines with '|' - presumably this will break existing multiline entries, but possibly they (at least ones which already contain '|') can be converted. 2) do 1) as a list or mm_cfg option which defaults to not doing it. 3) change the description and help for the regexp to correctly reflect what happens. What do others think? -- Mark Sapiro <msapiro(a)value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

6 12

sender-based authorisation
by David Lee 22 May '06

22 May '06

Mailman: Proposed sender-based passwords ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note: this is based on Mailman 2.1.8 which, at the time of writing, is the current version. A discussion "approval password linked to sender rather than list?" on mailman-users around 15 May 2006 agreed on the desirability of a variant of the "Approved: password" mechanism, whose permission-granting password would be associated with the email's sender rather than with the list. The motivation is large sites with many "announcement" lists (in our case over 2,000), where the entire membership of each such list is moderated, and where several legitimate senders might regularly need to post single emails to several lists. These lists might well exist in clusters. But the option of allowing unmoderated access for those legitimate senders opens up a major problem with spoofed email. So everyone needs to be moderated. But then the potentially useful moderation-bypass "Approved: listpw" mechanism has the problems of requiring many people to share a particular list's password, and of the passwords across a cluster of lists having to be common. This "shared secret" problem grows even more alarmingly if the list clusters overlap which would force reduction to a single password across many lists, and widespread knowledge (hence leakage potential) of that password. Introducing an "Approval:"-like mechanism, whose password is associated with the sender ("From:"), is intended address all this. Virtual host considerations ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Mailman is beginning to allow some virtual hosting (i.e. multiple domains ("...@dom1", "...@dom2"). Although this is currently limited (cannot have the same listname at multiple domains) it is clearly on the roadmap for the future. Accordingly this password proposal attempts to take full account of that direction. New "sender" role ~~~~~~~~~~~~~~~~~ Currently, according to "Mailman/SecurityManager.py": # There are current 5 roles defined in Mailman, [...] # user, list-creator, list-moderator, list-admin, site-admin. That is, five contexts in which passwords are used and supported. Clarification: "user" means the intersection of a particular email address and a particular list: the (in)famous "monthly password reminder" entity. Put another way: the same person (email address) on 'n' different lists is actually 'n' unrelated "users". A new "sender" role is proposed, available across all lists at all virtual hosts. It is implemeted as a site-wide database of email addresses and their passwords, with the same variety of options for encryption and storage as in (or potentially in) the other databases. Across virtual hosts it similarly remains the same, common, sender. Clarification: Unlike a "user", this is a single stored-once entity, representing a single email address across the entire installation. In each list a new list-admin function maintains a set of email addresses, herein called "authorised_senders", that can post through the list. On the GUI, this would present a little like various "*_these_nonmembers". An incoming email can specify "Authorised: sender-password" in the manner of the existing "Approved: list-password" mechanism (headers, first real blank-separated line, etc.). [Note: I have chosen a separate word "Authorised:". Perhaps we can conflate this new concept onto "Approved:" if there would be no loss of functionality. If so, then "authorised_senders" should perhaps be called "approved_senders".] Its processing confirms the "From:" in the list's "authorised_senders", then does an authorisation check against the site-wide "sender" (role) database. (If we conflate "Authorised:" and "Approved:", the password check would potentially need to be done on both the list and the "sender" databases.) If an incoming "Authorised:" email is destined for several lists, perhaps at several virtual hosts, its "From:" is processed within each list as described above against the site-wide "sender" database. Maintenance of "sender" ~~~~~~~~~~~~~~~~~~~~~~~ The new "sender" database requires maintenance. Insertion: How are email addresses and their passwords added? How is it requested? By whom or by what is it implemented? What should be done to detect and prevent malicious (spam-like, DOS-like) mass sign-up? Deletion: How can an email address be removed from the database? When it is removed, how is that reflected back into all the lists which had it in their "authorised_senders" table? (Analogous to removing a file in a filesystem but leaving behind dangling symbolic links.) Should those lists be left alone (but now containing redundant entries) or should their tables be amended? Suppose the deletion from "senders" was accidental or transient: how can any removed entries in the several "authorised_senders" tables be efficently restored? Changes: How does someone maintain (change) their email address and password? If a person changes their email address how is that reflected back to the lists? (See "Deletion".) Intuitively, the database key would have been the email address. But perhaps the real key should rather be some opaque, unique (UUID-like?) entity, with the email address simply being an attribute. -- : David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :

4 7

Why does Mailman delete post recipients from Cc: header?
by Mark Sapiro 20 May '06

20 May '06

In response to a question on mailman-users, I looked at AvoidDups.py, and I see that if an address is in the Cc: header of a post and is a candidate recipient and is a list member with DontReceiveDupes set, the address is removed from the recipient list and from the Cc: header of the post. I understand why it is removed from the recipient list - that's the whole point - but does anyone know/remember why it is also removed from the Cc: header? This seems unnecessary and inconsistent since it isn't removed from To:, Resent-To: or Resent-Cc:. Also, there is a docstring in the module that refers to an in memory Message-Id: list for avoiding cross-posted dups which I think was in the original patch, but was never incorporated. Perhaps this should be rewritten. -- Mark Sapiro <msapiro(a)value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 2

Re: [Mailman-Developers] Mailman "true virtual hosting" and Google SoC
by Heikki Lampén 13 May '06

13 May '06

On 4/24/06, Barry Warsaw <barry(a)python.org> wrote: > On Mon, 2006-04-24 at 16:21 +0300, Heikki Lampén wrote: > > I'm wondering about this "true virtual hosting" issue and what is > > going to happen to it. > > > > I see there's some talk about Google SoC in the wiki [0]. > > Unfortunately I don't see Mailman in the Google SoC participant list > > [1]. Has Mailman applied as a mentoring organization? The deadline is > > 1st of May. > > True, although I was thinking that if someone wants to do this, we could > do it under the auspices of the Python Software Foundation. There > really is no "Mailman organization" and the latter's SoC project page > does list Mailman as a possibility. Sounds good. > > Anyway, it'd be awesome if SoC could finish what the vhost branch has > > began. Or is it totally out of the question that such a patch would be > > integrated to 2.1? Even as a branch (that'd be kept up to date with > > 2.1) it could be very useful. > > It's not out of the question, although no way can it go into the > mainline 2.1 branch. Ideally, the work would be done on the Mailman > trunk for 2.2, although I wouldn't be opposed to an "unofficial" 2.1 > patch branch if the volunteers were so motivated. It's too big of > project to go into 2.1 proper though. Fair enough. > > Why or why not should the vhost branch development happen on the same > > subversion repository? In CVS there was the awful branching problem. I > > think Subversion is the most attracting version control system. Then > > again what I think doesn't matter but what you guys think is what > > matters. Of course moving off to another platform is always extra work > > and has to be motivated properly. > > My hesitance is much more the code disruption, upgrade testing, and > support costs involved in pulling this into 2.1. We just can't be > adding new features to 2.1 at this point. What we /really/ need to do > is get 2.2 moving so that we can release it fairly soon (this year, I > hope). > > > Also the usage of JIRA and Confluence should make it even more > > interesting as organizing the project becomes a whole lot easier that > > way. Maybe add a project for the vhost branch. > > > > Maybe I'm just being impatient but as I see it, this vhost stuff > > should really interest a lot of people. > > Definitely. Everyone wants to see it, and it will definitely happen for > 2.2. It would be awesome if a SoC volunteer wanted to take it on and I > would be more than happy to mentor the work. Are you a student and are > you interested in working on it? I'm unfortunately already employed during summer time (would've had been an interesting project). Maybe someone on the list is or knows a promising victim? -- heze heikki.lampen(a)gmail.com

2 1

thoughts on bounce processing
by James Ralston 11 May '06

11 May '06

Per Barry's request, I thought I'd describe some of the issues our sites has encountered with Mailman lists and bounce messages. At our site, we currently 15,775 subscribers (8,296 unique) across 549 lists. While a few of the lists are huge (hundreds or thousands of subscribers), the median list size is only 4 subscribers. The primary issue we are encountering is that the vast majority of our lists (even the ones with hundreds of subscribers) are announcement-type lists, not discussion-type lists. The list owners are using the lists to reach coworkers, specific customers, course attendees, etc. The list admins know exactly which addresses are on their lists, because they've added them all themselves. Situations like this happen all the time: 1. A VIP asks an administrative assistant to send out a message to a group of people: team members, specific customers, etc. 2. The AA sends the message to the appropriate Mailman list. Unbeknown to the AA, one of the addresses bounces. 3. The person whose address bounced doesn't get the message, and (e.g.) misses an important meeting, fails to provide some important information, etc. 4. The VIP comes down hard on the AA for failing to notify everyone. 5. The AA comes to us full a fire and brimstone, complaining the Mailman doesn't permit people to perform necessary business functions. The issue is that people in this situation *want* to see the individual bounces, and they want to see them *immediately*, because the bounces contain important information that they might need to act upon. The closest they can get to that right now is to configure the list so that a single bounce immediately removes the subscriber from the list. But this isn't always the correct thing to do. (For example, lots of people seem to have broken vacation programs that respond to *everything*, and they continually get unsubscribed from lists.) The more I think about this, the more I think that silently discarding bounces is an error. While some bounces are irrelevant (e.g. vacation messages), in the majority of cases, bounces are information that some entity should act upon. There should be two (and only two) options: 1. Mailman processes all bounces. 2. All bounces are forwarded to the list administrator(s). I've attached my first attempt at implementing this behavior. (I say "first attempt" because I'd really like to get both the listname and the address that bounced into the Subject header, but to do that requires more complicated logic than what I've implemented so far.) If list admins *really* want to (effectively) discard all bounces, they can e.g. set bounce_info_stale_after to 1 and then set bounce_score_threshold to something impossibly high (e.g. 500,000). Thoughts? James

4 5

Re: [Mailman-Developers] Sender field
by James Ralston 11 May '06

11 May '06

On 2006-04-27 at 22:46-05 Brad Knowles <brad(a)stop.mail-abuse.org> wrote: > At 10:40 AM -0500 2006-04-27, Neal Groothuis wrote: > > > Again from RFC 2822 3.6.2, the Sender: header should contain the > > address of the agent responsible for transmitting the message, > > meaning that a person who sends mail to the address in that header > > should expect to reach said agent, not suggest to Mailman that a > > message bounced. > > Right. Mailman is the agent responsible for transmitting the > message, and this needs to be reflected. This is not correct. Quoting RFC2822 section 3.6.2: 3.6.2. Originator fields The originator fields indicate the mailbox(es) of the source of the message. The "From:" field specifies the author(s) of the message, that is, the mailbox(es) of the person(s) or system(s) responsible for the writing of the message. The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message. For example, if a secretary were to send a message for another person, the mailbox of the secretary would appear in the "Sender:" field and the mailbox of the actual author would appear in the "From:" field. The Sender header should be employed by the orignator of the message, and only the originator. Mailman is not the originator of a message sent to a list; it is merely a relay agent. I will grant that the phrasing of the RFC is suboptimal here--it uses "transmission" when a better word choice would have been "submission". But the immediately proceeding example (of a secretary sending mail on behalf of another person) clarifies the intent beyond any claim of ambiguity. > [Outlook's behavior] is an MUA problem. See FAQ 2.3. No, it's not. As much as it pains me to say it, Outlook's behavior matches perfectly the intent of the RFC. It is Mailman's behavior of rewriting the Sender header that is the problem. > However, we want to make sure to capture any potential messages that > may be routed to the "Sender:" field and have them automatically > processed through the part of the system that is designed to do that > sort of thing. Mailman's "processing" behavior is to treat a reply to the Sender as a bounce. This is incorrect behavior, because many mail clients will include address of the Sender header in a "reply-to-all" function, causing Mailman to treat the reply as a bounce. So, in summary, the disadvantages of Mailman's behavior of rewriting the Sender header is that doing so is not in the intended spirit of RFC2822, causes subscription grief, and breaks Outlook. The advantage is that it helps Mailman detect bounces from a slim minority of brain-dead MTAs that send bounces to the Sender header. > The problem is that you said you wanted to implement an option to > allow people to turn it off, not to rip this feature completely out > of the system. I would argue that the best course of action is to excise Sender header rewriting entirely and provide no option to turn it on. (Mailman has way too many options already.) If, however, an option is created to control the behavior, it should definitely default to OFF (no Sender header rewriting), not on. -- James Ralston, Information Technology Software Engineering Institute Carnegie Mellon University, Pittsburgh, PA, USA

6 14

GNU FSF Assignment
by Fil 10 May '06

10 May '06

Hello, can someone point me to the procedure/documents needed to assign copyright to Barry^H^H^H^H^H the FSF ? I can't find them on Google or on the GNU project pages (??). -- Fil

3 6

why not PageTemplates for GUI?
by Dario Lopez-Kästen 09 May '06

09 May '06

hello, I am wondering why any future template suggestion a priori rule out PageTemplates? I find them really handy to use, at least in zope-contexts. Is it becuase it is zope or other reasons? thanks, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley

2 3

logger umask
by Tokio Kikuchi 04 May '06

04 May '06

Hi, I think our new logger should set group writable to the new logs. I tried a fresh install and get cgi error after mailmanctl start / mailmanctl errot after cgi execution, vice versa. The patch should look like this: Index: Mailman/loginit.py =================================================================== --- Mailman/loginit.py (revision 7892) +++ Mailman/loginit.py (working copy) @@ -103,6 +103,8 @@ # There was also a 'debug' logger, but that was mostly unused, so instead # we'll use debug level on existing loggers. # + # First set umask because both mailman/www write logs + oumask = os.umask(002) # Start by creating a common formatter and the root logger. formatter = logging.Formatter(fmt=FMT, datefmt=DATEFMT) log = logging.getLogger('mailman') @@ -120,6 +122,8 @@ _handlers.append(handler) handler.setFormatter(formatter) log.addHandler(handler) + # restore original umask + os.umask(oumask) -- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

1 0

New Mailman logging
by Mark Sapiro 03 May '06

03 May '06

I think I've finally gotten a grasp of the new Mailman logging scheme and I think there's something missing. I see how the basic logging to files works and how the propagate flag to loginit.initialize() controls propagation to the root 'mailman' logger which logs to the sys.stderr stream. What I think is missing is something analogous to the old Mailman.Logging.Utils.LogStdErr() to cause writes to sys.stderr to be logged instead of or in addition to being sent to the stderr stream. The end result of this is that a few messages from bin/mailmanctl and one message from bin/qrunner will only go to the stderr stream and will not be logged. I don't see this as serious as I think all these messages can only occur in response to fatal conditions in direct response to a mailmanctl command. However, there are also sys.stderr writes in the various scripts invoked by the mail wrapper. These used to be logged as well as written to the stderr stream (where they presumably end up in a DSN). The error conditions that cause these messages occur because of bad alaises or routers in the MTA (things that invoke the wrapper without a list name) or aliases left after a list is removed. These should probably still be logged. -- Mark Sapiro <msapiro(a)value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 3

Mailman-Developers May 2006