Mailman 3 January 2002 - Mailman-Developers

RE: Mailman developer?
by Damien Morton Jan. 30, 2002

Jan. 30, 2002

We can use GIFs, but we cant use the LZW compression option. As the GIFs we produce will be small, this is hardly a problem. At any rate, any of GIF, PNG or JPEG will do. The advantage of using encryption rather than hashing is that the email renderer needs to know nothing about the system except what it is given in the querystring (and the key, of course). Using md5 hashes would be more secure, but would require more information to be shared between the pipermail cgi and the renderemail cgi; that is, a mapping of hashes to email addresses. Frankly, I cant see email harvesters going to the trouble of cracking encryption - no matter what kind. I can think of an alterative technique for preventing email harvesting; one that preseves the clickability of emaila addresses. It involves the use of javascript. Using it, an email link would look something like this: <script language="javascript"> email = decode("92731602eba1aa4f506604f8c3671ed83ea9"); document.write("<a href='mailto:"+email+"'>"+email+"</a>"); </script> decode() is some kind of decrypting function. Possibly something as simple as a substitution cipher. The disadvantage of this technique is that email addresses wouldn't be accessible to people using a non-javascript capable browser, and browser variations would tend to make this less reliable than the image viewing technique. For example, I have found that Netscape is less than reliable when it comes to document.write(). One solution to the non-javascript browser issues would be to use the <noscript> tag to deliver the email address as an image. <noscript> <img src="/render-email.py?92731602eba1aa4f506604f8c3671ed83ea9"> </noscript> One problem with all this is that pipermail renders HTML as it stores emails, rather than as they are viewed. The issue here is that the original emails are lost; if we encrypt (or hash) email addresses, then losing the key (or hash->email mapping) implies also losing the addresses. There are some comments in the pipermail/HyperMail source code about rendering to html on viewing rather than storage, but converting to this scheme would imply a backwards compatability issue: how to import emails alerady rendered to html. A backwards compatable solution would be to render to html twice - once on storage and again on viewing. The viewing renderer would be responsible for encrypting/obfuscating email addresses. Im going to join the mailman-developers list with a temporary email address. Get some more input on this issue. -----Original Message----- From: Barry A. Warsaw [mailto:barry@zope.com] Sent: Wednesday, 30 January 2002 21:18 To: Damien Morton Subject: Re: Mailman developer? >>>>> "DM" == Damien Morton <Damien.Morton(a)acm.org> writes: DM> Im under the impression that you are one of the main DM> developers and/or maintainers of Mailman. I hope you don't DM> mind me writing to you. Nope. DM> I notice that Mailman obfuscates email addresses to a DM> certain extent, but replacing the @ symbol with %40 or DM> &atmark; is hardly sufficient. An even vaguely intelligent DM> email harvester will see through this. True. DM> The feature im proposing is to render out all email DM> addresses in the archive as GIFs. I would have pipermail DM> render out <img> tags whose src is an encrypted version of the DM> email address. A companion CGI script would decrypt the email DM> address and render it out as a GIF image using PIL or DM> somesuch. Of course, because Mailman is a GNU project, we can't use gifs, but pngs or jpegs would work just as well. IIRC, PIL can generate either of those formats. | Instead of rendering this: | <a href="mailto:your.email@address">your.email@address</a> DM> You'd render this instead: <img DM> src="/render-email.py?92731602eba1aa4f506604f8c3671ed83ea9"> DM> This exmaple uses the simple rotor encryption that DM> comes with python and the key is "the quick brown fox jumped DM> over the lazy dog" I usually use the md5 module to generate unique keys in such situations. DM> Ive been looking at pipermail, and it is DM> _ugly_. Tell me about it! About it's only saving grace is that it's reasonably well integrated and it's all in Python. Other than that... ;) There have been lots of discussion over the years about ditching that code and doing it right, so I'd suggest pouring over the mailman-developers archives (yes, you'll miss being able to search it ;). It's a lot of work though, so it currently languishes for lack of a motivated champion. DM> Nonetheless, I'm fairly sure I can add this functionality DM> easily. Im running w2k, however, and I see that Mailman isnt DM> really meant for w2k. Correct. DM> As I would be working on the pipermail part of mailman only, DM> it might be easier to get only that component up and running DM> under widnows. Not sure if theres a sample archive that comes DM> with mailman, but... any suggestions welcome. Nope, but you can of course grab the raw mbox for any archive, or for a month of messages. Note that that does point to another source of leaked addresses, one that won't be directly affected by your idea. However I could see hiding raw mbox access behind a cgi POST which should effectively stop today's harvesters. DM> The downsides of this functionality are that it might DM> incur a performance penalty and that it eliminates the DM> clickable mailto: functionality persently there. System-wide caching should alleviate the performance hit (modulo cgi overhead). The loss of clickable mailto: would be a drag, although I don't know how much it would be missed in practice. DM> As far as functionality goes, I imagine that the bulk DM> of any mailman bandwidth will be from spiders, and these are DM> unlikely to traverse an image source link. Secondly, simple DM> caching should be very easy to implement. DM> As far as the clickable mailto functionality goes, I DM> have two suggestions. The first is that the DM> render-emails-as-images functionality could be a personal DM> preference of the sender of the email, and the second is that DM> that preference could be overridden by acquiring a cookie DM> through some bot and spider proof mechanism. I like the 'type DM> what you read in the image above' mechanism for detecting DM> humans. I wouldn't make it a option of the sender, but of the list (or maybe just of the site). Anyway, it's a neat idea. Mailman-developers would be the best place to discuss it, but that does present a bit of a catch 22 for you. ;) >From a practical standpoint, MM2.1 will likely go to beta this weekend, meaning feature freeze. However I encourage you to follow through and work out some patches, if you'd be willing to assign copyright to them to the FSF eventually. Post any patches on Mailman's SF project page and that will let other interested parties download it and test it out, etc. Cheers, -Barry

1 0

decoupling the web interface from the exploder.
by Darrell Fuhriman Jan. 30, 2002

Jan. 30, 2002

All right, I haven't done any Mailman hacking for a while, but I wanted to work on adding this feature. Let me explain why. Right now, it's essentially impossible to have more than one machine doing Mailman processing. (Yes, there are ways to hack around it, but they get ugly quickly.) It's also impossible to have the web interface on any machine other than the machine where list explosion happens. This is suboptimal, because you have the potential of having more than one web server, or more than one mail box. Let me give an example, from the real world, aka my setup. |---------| /---| MR/web |---\ | |----------| / |---------| \ | |----------| | internet |< >--|-----|int. mail | |----------| \ |---------| / | |----------| \---| MR/web |---/ | |---------| ^ Firewall MR == Mail Relay. All mail/web traffic is load balanced. Now, I can't simply relay all traffic for mailing lists inside the firewall, because then I would no longer have access to the web interface. Therefore I have to special case all my lists, to have them exploded on the MR machine, then relayed to the internal mail machine for delivery. That part works fine for one machine, it doesn't work for 1+N. Instead you have to do is have each of the N MR machines forward all mailing-list traffic to one machine, and put redirects in your web server to again point to that one machine. That's kinda ugly, and defeats the purpose of having multiple identical relays. What I'd really like is a way to have the web interface be a client which could get/set any mailing list configuration through some sort of backend. Then it would be trivial to just relay all lists into the internal machine(s) for delivery. So, if I understand it correctly there are three things that need to be abstracted, and backends written (ideally, for me, to use a database): 1) membership lists 2) list config 3) templates 4) archives Now (1) seems to have been done already with MemberAdaptor, and (2) shouldn't be that hard to do either, but will require changing a number of the assumptions made in MailList.py and probably elsewhere. 3) is annoying, but shouldn't be too hard to handle. Again, there just needs to be a defined interface. The most challenging one is (4), which is easy enough to do for the case of one delivery machine, but much harder for the 1+N case. It also needs to be abstracted. So, I'll probably be able to devote time over the next couple months to writing this, but I'm interested in how people feel such a beast should look, especially (4). Thoughts? Darrell

3 3

USE_ENVELOPE_SENDER is not flexible enough
by Jason R. Mastaler Jan. 30, 2002

Jan. 30, 2002

For members-only lists like this one, USE_ENVELOPE_SENDER currently allows the admin to determine membership by checking either the From: header, OR the envelope sender address. I think a more flexible alternative would be to first check the envelope sender, and then the From: header for a match. In fact, this would make sense as the default setting IMHO. Why am I interested in this? Because I like to post to public mailing lists and newsgroups with a 'dated' address that expires after a set period of time to thwart SPAM harvesters. Obviously I can't use these one-time addresses on closed lists, but I could get around this by posting with an envelope sender address that matches my subscription address, and a 'dated' address in my From: header. The closed list would be happy, and I would be happy. -- (TMDA (http://tmda.sourceforge.net/) (UCE intrusion prevention in Python)

3 8

reply-to setting per user
by Marc MERLIN Jan. 30, 2002

Jan. 30, 2002

(following up a private discussion on having mailman allow users to decide if they want reply-to munging or not, not the list master) > Another thought is to not do the munging in CookHeaders.py, but to > defer it to later in the delivery process. If you could wedge it into > SMTPDirect.py it might not be too hard, but I'd have to think about it > more. (feel free to extract this part of the mail and answer on mailman-devel if you wish) Ok, so I'm looking at ToOutgoing.py I need to - find out where msg is defined (pointer welcome) - add a 'munge-users' field - enqueue the message once as-is - set munge in msg, modify msgdata to add reply-to, and enqueue the message a 2nd time While it would be better to have a tri-state reply-to setting per user: as defined by the list / leave intact (empty or set by posting user) / set that would be hard because doing #2 is not easy since the listwide munging will happen before the actual explosion and delivery to the users. So, I think we can have: If reply-to munging is enabled on that list, ignore user settings (I'd really like to give them the option to disagree with the list owner, but I have to set some bounds, otherwise it's going to be too much work) If munging is disabled (which ideally could be done on a site wide basis), then users get to chose if they want to enable it (and the default is obviously disabled when the user setting is undefined) As an optimization, ToOutgoing.py would scan the user list and only queue the message once if no list member opted for munging or if munging is already done on a list-wide basis. Then, the outgoing qrunner when it picks the messages, would check if munge-users is set for that specific message, and only send the message to the users whose munging setting matches the value. I don't know if and when I'll be able to do that (depends on how my time gets assigned), but I thought I would run it by you guys to see if I missed anything and if there was a bored hacker who knew the mailman code better than me :-) (turns out, while I think this feature would be a great thing, I don't yet know whether it will solve my particular problem or not) Marc -- Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f(a)merlins.org for PGP key

1 0

yet another hack: print "nomail" addresses and reasons
by Dan Mick Jan. 30, 2002

Jan. 30, 2002

I've been changing my "nomail" module for withlist to incorporate the new format in 2.1 and print it. Here's the latest version (use with bin/withlist -r nomail <listname> from ~mailman): from Mailman import MemberAdaptor def statusstr(status): return ('0', 'UNKNOWN', 'BYUSER', 'BYADMIN', 'BYBOUNCE')[status] def nomail(list): statuses = (MemberAdaptor.UNKNOWN, MemberAdaptor.BYUSER, MemberAdaptor.BYADMIN, MemberAdaptor.BYBOUNCE) for status in statuses: members = list.getDeliveryStatusMembers((status,)) for member in members: print member, statusstr(status) sys.exit(0)

1 0

pending.db dumper
by Dan Mick Jan. 30, 2002

Jan. 30, 2002

Here's a short Python hack to dump the pending.db database in a slightly-more-useful (to me at least) format: sorted by expiry ('eviction') date, with key, eviction date, and item tuple on a separate line. Again, any and all comments welcome. Meant to be run from ~mailman. #!/usr/bin/env python from Mailman import Pending import time db = Pending._load() def sortfunc(i1, i2): (c1, d1) = i1 (c2, d2) = i2 if (d1 < d2): return -1 if (d1 == d2): return 0 if (d1 > d2): return 1 ev = db['evictions'] evtuples = map(None, ev.keys(), ev.values()) evtuples.sort(sortfunc) for k, d in evtuples: print k, time.ctime(d) print "\t", db[k]

1 0

Postfix mailq formatter/dumper/searcher: mq
by Dan Mick Jan. 30, 2002

Jan. 30, 2002

I use Mailman with Postfix, and every so often, I would like to examine the outgoing mail queue and kill some messages. Postfix supplies "postsuper" to do this given that you know the queue ID, but won't, say, take an address and find all mail to that address and dequeue it. Also, I sorta hate the output format of mailq. So I hacked together a Python program that: 1) formats the mailq output more nicely, in two different ways: a verbose format: id: E78C251BDD* size: 4317 datetime: Tue Jan 29 15:25:50 sender: <listname>-bounces(a)dom.ain recipients: ['luser1(a)dom.ain', 'luser2(a)dom.ain2'] and a one-per-line greppable format: id=E78C251BDD* size=4317 datetime=Tue Jan 29 15:25:50 sender=<listname>-bounces(a)dom.ain recipients=['luser1(a)dom.ain','luser2(a)dom.ain2'] Both formats can optionally include the 'remote SMTP error message'. 2) filter by looking for any one-recipient queued messages that match a regular expression: mailq -v luser1(a)dom.ain might show id: E78C251BDD size: 4317 datetime: Tue Jan 29 15:25:50 sender: <listname>-bounces(a)dom.ain recipients: ['luser1(a)dom.ain'] 3) output just the queue ID, useful in combination with 2) above, so that one can do postsuper -d `mq -i luser1(a)dom.ain` and delete all the queued messages going just to luser1(a)dom.ain. Here's a usage message: mailq [-vie] [--verbose] [--id-only] [--show-errors] [recip RE] show the queued mail (using mailq) -v/--verbose: show a multiline view of each queued mail (default is one line per queue item) -i/--id-only: show only the queue IDs -e/--show-errors: show SMTP errors for each recipient recip RE: a regular expression to filter by recipient if that recipient is the only one for a queue item Example: mailq -i foo(a)bar.com shows queue IDs for any mails queued for foo(a)bar.com only Example: mailq -v shows all queue items in verbose format and here's the code. Use as you wish; any comments on Python style are especially welcome. I'd be happy to adapt this to other mailq output formats if I find out about them; obviously it's pretty fragile wrt the Postfix output format. It's either that or get my fingers into the queue directories, which is a less-stable interface, IMO. But if someone can document the sendmail format I'll see if I can't adapt to it as well. #!/usr/bin/env python import os import sys import re import getopt class MyRec: def __init__(self, id): self.id = id self.size = 0 self.sender = self.datetime = '' self.recipients = [] def __str__(self): return self.makestr(1) def makestr(self, errors): if errors: reciplist = self.recipients else: reciplist = map(lambda x: x[0], self.recipients) return '\n'.join(( "id: %s" % self.id, "size: %d" % self.size, "datetime: %s" % self.datetime, "sender: %s" % self.sender, "recipients: %s" % repr(reciplist))) def __repr__(self): return self.__str__() def oneline(self, errors): if errors: reciplist = self.recipients else: reciplist = map(lambda x: x[0], self.recipients) return ' '.join(( "id=%s" % self.id, "size=%d" % self.size, "datetime=%s" % self.datetime, "sender=%s" % self.sender, "recipients=%s" % repr(reciplist))) def multiline(self, errors): return self.makestr(errors) + '\n' def parsemailq(): hre = re.compile('.*Queue ID.*') idre = re.compile(r""" ^(?P<id>[0-9A-Z\*]+) \s+(?P<size>[0-9]+) \s+(?P<dow>\S+) \s+(?P<mon>\S+) \s+(?P<day>[0-9]+) \s+(?P<time>\S+) \s+(?P<sender>\S+) """, re.VERBOSE) reasonre = re.compile('^\s*\(') recre = re.compile('^\s+(?P<recip>[^@]+@[^@]+)') sepre = re.compile('^$') recs = [] mq = os.popen("mailq", "r") for line in mq.readlines(): line = line.rstrip() if hre.search(line): continue mo = idre.match(line) if mo: r = MyRec(mo.group('id')) reason = '' r.size = int(mo.group('size')) r.datetime = ' '.join((mo.group('dow'), mo.group('mon'), mo.group('day'), mo.group('time'))) r.sender = mo.group('sender') continue mo = reasonre.match(line) if mo: reason = line.strip() continue mo = recre.match(line) if mo: r.recipients.append([mo.group('recip'), reason]); reason = '' continue mo = sepre.match(line) if mo: recs.append(r) mq.close() return recs def usage(): print >> sys.stderr, """ mailq [-vie] [--verbose] [--id-only] [--show-errors] [recip RE] show the queued mail (using mailq) -v/--verbose: show a multiline view of each queued mail (default is one line per queue item) -i/--id-only: show only the queue IDs -e/--show-errors: show SMTP errors for each recipient recip RE: a regular expression to filter by recipient if that recipient is the only one for a queue item Example: mailq -i foo(a)bar.com shows queue IDs for any mails queued for foo(a)bar.com only Example: mailq -v shows all queue items in verbose format""" def main(): lookfor = None try: (optlist, trail) = getopt.getopt(sys.argv[1:], 'vieh', ['verbose', 'id-only', 'show-errors', 'help']) except getopt.GetoptError, v: print v usage() sys.exit(1) verbose = idonly = errors = 0 for (o, a) in optlist: if o in ("-v", "--verbose"): verbose = 1 if o in ("-i", "--id-only"): idonly = 1 if o in ("-e", "--show-errors"): errors = 1 if o in ("-h", "--help"): usage() sys.exit(0) if trail: lookfor = re.compile(trail[0]) recs = parsemailq() for r in recs: if lookfor: if len(r.recipients) == 1 and \ lookfor.search(r.recipients[0][0]): dump(r, idonly, verbose, errors) else: dump(r, idonly, verbose, errors) def dump(r, idonly, verbose, errors): if idonly: print r.id elif verbose: print r.multiline(errors) else: print r.oneline(errors) main()

1 0

Re: Re: [Mailman-Developers] bounce processing in 2.1CVS
by bob＠nleaudio.com Jan. 30, 2002

Jan. 30, 2002

Also, don't forget that you need to ignore days that go by without posts. You need to count "posting" days. Bob > >> 1) are we a member? No, return >> 2) do we have previous bounces? No, register it skip to step 7 >> 3) is the member disabled? yes, quit >> 4) is the bounce info from today? yes, don't increment, skip to step 7 >> 5) is the info stale? yes, reset it and skip to step 7 >> 6) increment score for today's bounce >> 7) is score > threshold? yes, disable 'em. > >Yeah, I like that better; at least it handles "I need to reset >my bounce threshold to get out of a hole" better. > >> DM> And besides, that seems like a useful knob to turn (i.e. "how >> DM> old does the bounce info have to be before I start looking for >> DM> new ones"). >> >> It might be, but I think for now I don't want to add that. > >I suppose I know how to write that much Python. :) > >> I realized you also want to add a step in cron/disabled that performs >> a sweep over the currently-bouncing-but-not-yet-disabled members, >> checking their scores against the threshold. Say this morning the >> threshold was 4.0 and some member had a score of 3.0. Then I go in >> and say screw it, set the threshold to 2.0. I'd like today's >> cron/disabled run to disable all those members with a score of 3.0. > >Yeah, good point. > > >_______________________________________________ >Mailman-Developers mailing list >Mailman-Developers(a)python.org >http://mail.python.org/mailman/listinfo/mailman-developers >

2 1

Importing email 0.96 problem
by Colin Mackinlay Jan. 29, 2002

Jan. 29, 2002

I've had some problems with getting 2.1a4 going on my Mandrake 8.1 installation and have gone back to a "clean" system. The documented way to install email is: python setup.py install This doesn't work for me as 2.1.1 is installed from an RPM and throws errors in relation to configure. What seemed ok was to copy the email folder into the python2.1 site-packages folder. Running python interactively enables me to import email and check the version. I can also import email.Utils and others too. I've created the site wide mailman list and also a test list from the command line. When I go to the /mailman/admin page I get a Bug error message Traceback (most recent call last) File "/usr/local/mailman/scripts/driver", line 80 in run_main pkg = __import__('Mailman.Cgi', globals(), locals(), [scriptname]) File "../mailman/Cgi/admin.py", line 31, in ? from email.Utils import unquote, parseaddr ImportError: No module named Utils Is there some othe part of python 2.1 that I need to hack? Colin -- Colin Mackinlay

3 9

Re: Theoretical way to minimize IO load with MTA supported VERP
by Russell Nelson Jan. 29, 2002

Jan. 29, 2002

tneff(a)bigfoot.com writes: > "Jay R. Ashworth" <jra(a)baylink.com> wrote: > > I don't see that there *is* any theoretical way to *keep* loads down > > with VERP, by it's very nature. > > If one was willing to extend SMTP again, Not necessary. If you know the remote MTA supports VERP, you can just send email like this: MAIL FROM: <mylist-owner-@mydomain.com-@[]> 250 mylist-owner(a)mydomain.com... Sender ok RCPT TO: <johnsmith(a)aol.com> 250 johnsmith(a)aol.com... Recipient ok RCPT TO: <joerandom(a)aol.com> 250 joerandom(a)aol.com... Recipient ok DATA In other words, no extensions to SMTP are needed other than an announcement in the ESMTP banner that the MTA will take responsibility for expanding VERP addresses. My own feeling is that qmail ought to recognize that it's talking to another qmail server, and send a VERP envelope sender, and multiple envelope recipients. There's no reason why mailman couldn't use a bulk mailer which sorts users by recipient hostname, knows how to recognize qmail, and uses VERP when it does. It could issue "EHLO hostname" and look for "VERP" (for future use) or else just two lines: 250-ns.crynwr.com NO UCE 250-PIPELINING 250 8BITMIME If just -PIPELINING and 8BITMIME are present, then it could issue "HELP" and look for the qmail help message: 214 qmail home page: http://pobox.com/~djb/qmail.html As Barry says, VERP tells you precisely which address is bouncing, but it doesn't say why. -- -russ nelson http://russnelson.com | The problem with do-gooders Crynwr sells support for free software | PGPok | and governments is that they 521 Pleasant Valley Rd. | +1 315 268 1925 voice | always think the government Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | will choose as they would.

2 1