MM feature request - spread out the pain

Perhaps this is on the feature request list already, or better yet, Barry borrowed Guido's time machine when I wasn't looking, however...
I help administer a number of python.org mailing lists. MM sends out subscription reminders on the first of each month. This inevitably leads to a huge number of mail bounces many of which MM doesn't understand. The python.org postmaster folk then get to go in and manually remove a ton of email addresses. (Looks like about 100 this month. I'm sure other sites have to handle many more such bounces.) I don't know how to automate this. The best thing I can think of to do is to write an Emacs macro which copies a highlighted email address, pops over to a shell mode window ssh'd into mail.python.org and runs remove_members with the given address. That's still more manual than I would like, but given the variability in bounce formats that seems about as good as I can do without expending a lot of effort.
I realize that getting Mailman to grok more rejection notice formats is a worthwhile goal, however I think it would also be helpful to spread the pain of invalid email addresses more evenly through the month, at least across mailing lists but ideally within mailing lists. Notification day for any given email might be
ord(email[0]) % 28 + 1
This would be close enough for gummint work (the 29th, 30th and 31st would be declared to be postmaster holidays).
-- Skip Montanaro - skip@pobox.com - http://www.smontanaro.net/

On Thu, Apr 01, 2010 at 11:59:05AM -0500, skip@pobox.com wrote:
shove all the postmaster@ mails into one mailbox, and...
awk/grep the email addresses and list name as a (c|t)sv'd line; select a list, and feed the email addresses to remove_members ?
(maybe doing a check on the body content, too)
cron that to happen at $interval, on Mailman-day +n, perhaps?
I don't really have this problem, but very few of my lists (I think it may be ~300 now) send reminders.
-- ``Should not the Society of Indexers be known as Indexers, Society of, The?'' (Keith Waterhouse)

Barry> Or how about we just get rid of them? :)
I wasn't sure what you meant until I saw Mark's reply. I guess I wasn't listening to the silence between the notes. ;-)
At any rate, that solves one problem but creates another, right? In MM3 how will you weed out bad email addresses? Again, this may already be accounted for somehow. If so, I'd be happy to learn how you do it without attempting to send email to an address then process the bounces.
Skip

skip@pobox.com wrote:
If the list is active and the user's delivery is enabled, The list's bounce processing should take care of ultimately removing bouncing members. If it doesn't, the list's bounce processing needs to be adjusted.
The dead addresses of concern, are those for which the user or admin disabled delivery before the address died. We have talked about some kind of periodic mailing from the list to all members. If we have such a thing, a bounce of that mail could identify such addresses.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

At 03:01 PM 4/1/2010, Mark Sapiro wrote:
Some other list software, I believe, has a feature that sends out a canned message once a month, such as list rules. I would like to see this in MM3 as I would like to send people the rules once a month, and not all my lists are that active, but still need to be in place and used occasionally.
Dave

David Andrews writes:
-1. This is mission creep. A cron job will serve perfectly well.
Something that assists in setting up such cron jobs (and/or generalizes to whatever weirdness Windows does instead of cron jobs) would be good as a contrib module, of course.

Stephen> David Andrews writes:
>> Some other list software, I believe, has a feature that sends out a
>> canned message once a month, such as list rules. I would like to see
>> this in MM3
Stephen> -1. This is mission creep. A cron job will serve perfectly
Stephen> well.
Stephen> Something that assists in setting up such cron jobs (and/or
Stephen> generalizes to whatever weirdness Windows does instead of cron
Stephen> jobs) would be good as a contrib module, of course.
I assume the monthly reminder emails have be canned for a reason. They do have the nice property of sending something to every subscribed address, even those who are set NOMAIL. I can do something from the command line (list_members? I use the command line so infrequently that I always have to look), but many list admins don't have shell access to their lists.
I hope I didn't open a can of worms. MM2 doesn't seem to have a through-the-web way to gather all list members from the admin's web interface. Will MM3? That would at least allow admins to assemble a list for sending all-hands or canned messages manually.
Skip

skip@pobox.com writes:
That's a good point. I think a better way to handle this might be an Urgent flag (I guess we can't use Precedence for this, unfortunately, list mail should still be bulk in most cases) accompanied by an appropriate authentication.
This could eventually be generalized so that users could filter (some users might want only Announce or above, others of more catholic tasts would go for Spam or better<wink>). That might (or might not) be a better idea than having PROJECT-discuss subscribed to PROJECT-announce.
MM2 doesn't seem to have a through-the-web way to gather all list members from the admin's web interface. Will MM3?
I hope so[1]; this is a FAQ anyway.
Footnotes: [1] Ie, this should be done by Mailman 3's RESTful API, not an end-around by using an LDAP server or whatnot to hold the address db, and accessing the LDAP server outside of Mailman.

David Andrews writes:
Of course it would be useful to people who are using platforms that were designed to make Mailman use painful. Nevertheless, Mailman's mission is "providing the best possible mailing list manager", not "providing poor emulations of everything that Unix does that Windows and cPanel do not". Supporting these Misfeatured Platforms already takes up a lot of time. It's like sharpening the claw end of a hammer head: the more such features we add, the more people will take up up using hammers to drive screws. It's still better to spend the $1 (or $15, if you go Snap-On) for the screwdriver.
If Mailman can do it *better* than cron, that would be another matter. That point is arguable, but my personal preference would be to provide a generic feature that both cron and other possible schedulers could take advantage of.
Alternatively, if this feature can be provided easily as a side effect of something Mailman already has to do, that would be good, too -- as long as it's not a step onto the slippery slope of mission creep.

Stephen> If Mailman can do it *better* than cron, that would be another
Stephen> matter. That point is arguable, but my personal preference
Stephen> would be to provide a generic feature that both cron and other
Stephen> possible schedulers could take advantage of.
In fact, it is quite easy to argue that with a programmatic interface and access to cron my "spread out the pain" request is better implemented there. The monthly mailing becomes a daily mailing which only sends reminder notices to a fraction of the users.
Skip

skip@pobox.com writes:
"There" being in Mailman itself? Or in cron? (It's not obvious to me whether Mailman could do this better internally or not; of course I'd try to do it in cron before looking for a feature in Mailman, but Jack's mileage would vary, I'm sure. :-)
The monthly mailing becomes a daily mailing which only sends reminder notices to a fraction of the users.
Indeed, or weekly, or even 4x daily, and providing that kind of flexibility of timing is something that cron has a lot of existing support for. OTOH, somebody still has to write the code to manage the partial lists. However, that code might be useful for other purposes (eg, if a user who refers to the member list page often wants it split into 4 equal-sized pages rather than 27 pages of wildly varying size). (The above is what I mean by "generic features", and an example of how they might be applied.)

>> In fact, it is quite easy to argue that with a programmatic interface
>> and access to cron my "spread out the pain" request is better
>> implemented there.
Stephen> "There" being in Mailman itself? Or in cron? (It's not
Stephen> obvious to me whether Mailman could do this better internally
Stephen> or not; of course I'd try to do it in cron before looking for a
Stephen> feature in Mailman, but Jack's mileage would vary, I'm
Stephen> sure. :-)
In cron via a web services api of some sort.
>> The monthly mailing becomes a daily mailing which only sends reminder
>> notices to a fraction of the users.
Stephen> Indeed, or weekly, or even 4x daily, and providing that kind of
Stephen> flexibility of timing is something that cron has a lot of
Stephen> existing support for. OTOH, somebody still has to write the
Stephen> code to manage the partial lists. However, that code might be
Stephen> useful for other purposes (eg, if a user who refers to the
Stephen> member list page often wants it split into 4 equal-sized pages
Stephen> rather than 27 pages of wildly varying size). (The above is
Stephen> what I mean by "generic features", and an example of how they
Stephen> might be applied.)
In my original post I wrote the selector I would use to determine which day of the month an email was to get a notice:
ord(email[0]) % 28 + 1
Assuming you can pull the current list members from an API you select those whose first letter matches the current day of the month. (*)
Now, as to actually performing this experiment, I find starting up Mailman a rather high barrier simply because you have to create and populate a mailing list. How do people generally go about doing that? Do you just create a bunch of dummy aliases which map to a single address?
Skip
(*) In the Apocryphal Bible we find these extra two verses:
Genesis 2
...
4 And on the 29th, 30th and 31st days of the month the Moderator rested from all the labors to which he had been assigned by the Administrator.
5 And the Administrator blessed the 29th, 30th and 31st days of the month and sanctified them by pulling a beer from the cooler.

[ on the topic of the demise of the monthly mailing ]
I haven't followed MM3 discussions closely, but if there is some sort of extension capability or programmatic API it might be useful to provide a demonstration of how to reproduce the defunct monthly mailing using such an interface.
Is MM3 far enough along to be running somewhere that one could experiment?
Skip

On Apr 05, 2010, at 06:15 AM, skip@pobox.com wrote:
Yes to both, though the passwords are no longer stored in plaintext so the monthly reminder can't be a *password* reminder. You could definitely do a monthly subscription reminder slash membership probe. The nice thing is that because folks can register multiple emails under one account, you should be able to send one message w/CCs I think to each person.
Is MM3 far enough along to be running somewhere that one could experiment?
Yes, certainly for this. Let us know how it goes!
-Barry

skip@pobox.com wrote:
Perhaps this is on the feature request list already, or better yet, Barry borrowed Guido's time machine when I wasn't looking, however...
As barry already noted, there will be no reminders in MM 3.
MM does not try to understand them because it can't handle them automatically. See the FAQ at <http://wiki.list.org/x/aICB>
Mailman probably already can recognize almost all of these. The problem is not in the recognition, it's in what to do about them. In addition to the info in the above FAQ, there's lots of detail about this in comments in Mailman/Queue/BounceRunner.py.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On Thu, Apr 01, 2010 at 11:59:05AM -0500, skip@pobox.com wrote:
shove all the postmaster@ mails into one mailbox, and...
awk/grep the email addresses and list name as a (c|t)sv'd line; select a list, and feed the email addresses to remove_members ?
(maybe doing a check on the body content, too)
cron that to happen at $interval, on Mailman-day +n, perhaps?
I don't really have this problem, but very few of my lists (I think it may be ~300 now) send reminders.
-- ``Should not the Society of Indexers be known as Indexers, Society of, The?'' (Keith Waterhouse)

Barry> Or how about we just get rid of them? :)
I wasn't sure what you meant until I saw Mark's reply. I guess I wasn't listening to the silence between the notes. ;-)
At any rate, that solves one problem but creates another, right? In MM3 how will you weed out bad email addresses? Again, this may already be accounted for somehow. If so, I'd be happy to learn how you do it without attempting to send email to an address then process the bounces.
Skip

skip@pobox.com wrote:
If the list is active and the user's delivery is enabled, The list's bounce processing should take care of ultimately removing bouncing members. If it doesn't, the list's bounce processing needs to be adjusted.
The dead addresses of concern, are those for which the user or admin disabled delivery before the address died. We have talked about some kind of periodic mailing from the list to all members. If we have such a thing, a bounce of that mail could identify such addresses.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

At 03:01 PM 4/1/2010, Mark Sapiro wrote:
Some other list software, I believe, has a feature that sends out a canned message once a month, such as list rules. I would like to see this in MM3 as I would like to send people the rules once a month, and not all my lists are that active, but still need to be in place and used occasionally.
Dave

David Andrews writes:
-1. This is mission creep. A cron job will serve perfectly well.
Something that assists in setting up such cron jobs (and/or generalizes to whatever weirdness Windows does instead of cron jobs) would be good as a contrib module, of course.

Stephen> David Andrews writes:
>> Some other list software, I believe, has a feature that sends out a
>> canned message once a month, such as list rules. I would like to see
>> this in MM3
Stephen> -1. This is mission creep. A cron job will serve perfectly
Stephen> well.
Stephen> Something that assists in setting up such cron jobs (and/or
Stephen> generalizes to whatever weirdness Windows does instead of cron
Stephen> jobs) would be good as a contrib module, of course.
I assume the monthly reminder emails have be canned for a reason. They do have the nice property of sending something to every subscribed address, even those who are set NOMAIL. I can do something from the command line (list_members? I use the command line so infrequently that I always have to look), but many list admins don't have shell access to their lists.
I hope I didn't open a can of worms. MM2 doesn't seem to have a through-the-web way to gather all list members from the admin's web interface. Will MM3? That would at least allow admins to assemble a list for sending all-hands or canned messages manually.
Skip

skip@pobox.com writes:
That's a good point. I think a better way to handle this might be an Urgent flag (I guess we can't use Precedence for this, unfortunately, list mail should still be bulk in most cases) accompanied by an appropriate authentication.
This could eventually be generalized so that users could filter (some users might want only Announce or above, others of more catholic tasts would go for Spam or better<wink>). That might (or might not) be a better idea than having PROJECT-discuss subscribed to PROJECT-announce.
MM2 doesn't seem to have a through-the-web way to gather all list members from the admin's web interface. Will MM3?
I hope so[1]; this is a FAQ anyway.
Footnotes: [1] Ie, this should be done by Mailman 3's RESTful API, not an end-around by using an LDAP server or whatnot to hold the address db, and accessing the LDAP server outside of Mailman.

David Andrews writes:
Of course it would be useful to people who are using platforms that were designed to make Mailman use painful. Nevertheless, Mailman's mission is "providing the best possible mailing list manager", not "providing poor emulations of everything that Unix does that Windows and cPanel do not". Supporting these Misfeatured Platforms already takes up a lot of time. It's like sharpening the claw end of a hammer head: the more such features we add, the more people will take up up using hammers to drive screws. It's still better to spend the $1 (or $15, if you go Snap-On) for the screwdriver.
If Mailman can do it *better* than cron, that would be another matter. That point is arguable, but my personal preference would be to provide a generic feature that both cron and other possible schedulers could take advantage of.
Alternatively, if this feature can be provided easily as a side effect of something Mailman already has to do, that would be good, too -- as long as it's not a step onto the slippery slope of mission creep.

Stephen> If Mailman can do it *better* than cron, that would be another
Stephen> matter. That point is arguable, but my personal preference
Stephen> would be to provide a generic feature that both cron and other
Stephen> possible schedulers could take advantage of.
In fact, it is quite easy to argue that with a programmatic interface and access to cron my "spread out the pain" request is better implemented there. The monthly mailing becomes a daily mailing which only sends reminder notices to a fraction of the users.
Skip

skip@pobox.com writes:
"There" being in Mailman itself? Or in cron? (It's not obvious to me whether Mailman could do this better internally or not; of course I'd try to do it in cron before looking for a feature in Mailman, but Jack's mileage would vary, I'm sure. :-)
The monthly mailing becomes a daily mailing which only sends reminder notices to a fraction of the users.
Indeed, or weekly, or even 4x daily, and providing that kind of flexibility of timing is something that cron has a lot of existing support for. OTOH, somebody still has to write the code to manage the partial lists. However, that code might be useful for other purposes (eg, if a user who refers to the member list page often wants it split into 4 equal-sized pages rather than 27 pages of wildly varying size). (The above is what I mean by "generic features", and an example of how they might be applied.)

>> In fact, it is quite easy to argue that with a programmatic interface
>> and access to cron my "spread out the pain" request is better
>> implemented there.
Stephen> "There" being in Mailman itself? Or in cron? (It's not
Stephen> obvious to me whether Mailman could do this better internally
Stephen> or not; of course I'd try to do it in cron before looking for a
Stephen> feature in Mailman, but Jack's mileage would vary, I'm
Stephen> sure. :-)
In cron via a web services api of some sort.
>> The monthly mailing becomes a daily mailing which only sends reminder
>> notices to a fraction of the users.
Stephen> Indeed, or weekly, or even 4x daily, and providing that kind of
Stephen> flexibility of timing is something that cron has a lot of
Stephen> existing support for. OTOH, somebody still has to write the
Stephen> code to manage the partial lists. However, that code might be
Stephen> useful for other purposes (eg, if a user who refers to the
Stephen> member list page often wants it split into 4 equal-sized pages
Stephen> rather than 27 pages of wildly varying size). (The above is
Stephen> what I mean by "generic features", and an example of how they
Stephen> might be applied.)
In my original post I wrote the selector I would use to determine which day of the month an email was to get a notice:
ord(email[0]) % 28 + 1
Assuming you can pull the current list members from an API you select those whose first letter matches the current day of the month. (*)
Now, as to actually performing this experiment, I find starting up Mailman a rather high barrier simply because you have to create and populate a mailing list. How do people generally go about doing that? Do you just create a bunch of dummy aliases which map to a single address?
Skip
(*) In the Apocryphal Bible we find these extra two verses:
Genesis 2
...
4 And on the 29th, 30th and 31st days of the month the Moderator rested from all the labors to which he had been assigned by the Administrator.
5 And the Administrator blessed the 29th, 30th and 31st days of the month and sanctified them by pulling a beer from the cooler.

[ on the topic of the demise of the monthly mailing ]
I haven't followed MM3 discussions closely, but if there is some sort of extension capability or programmatic API it might be useful to provide a demonstration of how to reproduce the defunct monthly mailing using such an interface.
Is MM3 far enough along to be running somewhere that one could experiment?
Skip

On Apr 05, 2010, at 06:15 AM, skip@pobox.com wrote:
Yes to both, though the passwords are no longer stored in plaintext so the monthly reminder can't be a *password* reminder. You could definitely do a monthly subscription reminder slash membership probe. The nice thing is that because folks can register multiple emails under one account, you should be able to send one message w/CCs I think to each person.
Is MM3 far enough along to be running somewhere that one could experiment?
Yes, certainly for this. Let us know how it goes!
-Barry

skip@pobox.com wrote:
Perhaps this is on the feature request list already, or better yet, Barry borrowed Guido's time machine when I wasn't looking, however...
As barry already noted, there will be no reminders in MM 3.
MM does not try to understand them because it can't handle them automatically. See the FAQ at <http://wiki.list.org/x/aICB>
Mailman probably already can recognize almost all of these. The problem is not in the recognition, it's in what to do about them. In addition to the info in the above FAQ, there's lots of detail about this in comments in Mailman/Queue/BounceRunner.py.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (6)
-
Adam McGreggor
-
Barry Warsaw
-
David Andrews
-
Mark Sapiro
-
skip@pobox.com
-
Stephen J. Turnbull