Mailman 2.1.10b4 Released
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I am happy to announce the next beta release of Mailman 2.1.10.
This is a security and bug fix release and it is highly recommended that all sites upgrade to this version. Mailman 2.1.10 also adds support for three new language translations, Galician, Hebrew and Slovak and a few new features.
Mailman is free software for managing email mailing lists and e- newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites.
For more information, including download links, please see:
http://www.list.org http://mailman.sf.net http://www.gnu.org/software/mailman
Special thanks are due to Barry Warsaw and Tokio Kikuchi for much coding and support, Moritz Naumann for help with security issues and Jim Tittsler for a significant patch.
Here's a list of the major changes.
Security
- The 2.1.9 fixes for CVE-2006-3636 were not complete. In particular,
some potential cross-site scripting attacks were not detected in editing templates and updating the list's info attribute via the web admin interface. This has been assigned CVE-2008-0564 and has been
fixed. Thanks again to Moritz Naumann for assistance with this.
New Features
- Changed cmd_who.py to list all members if authorization is with the
list's admin or moderator password and to accept the password if the roster is public. Also changed the web roster to show hidden members when authorization is by site or list's admin or moderator password (1587651).
- Added the ability to put a list name in accept_these_nonmembers
to accept posts from members of that list (1220144).
- Added a new 'sibling list' feature to exclude members of another list
from receiving a post from this list if the other list is in the To: or Cc: of the post or to include members of the other list if that list is not in the To: or Cc: of the post (Patch ID 1347962).
- Added the admin_member_chunksize attribute to the admin General Options
interface (Bug 1072002, Partial RFE 782436).
Internationalization
- Added the Hebrew translation from Dov Zamir. This includes addition of
a direction ('ltr', 'rtl') to the LC_DESCRIPTIONS table. The add_language() function defaults direction to 'ltr' to not break existing mm_cfg.py files.
- Added the Slovak translation from Martin Matuska.
- Added the Galician translation from Frco. Javier Rial Rodríguez.
Changes since 2.1.10b3 include the Galician translation and updates to the French translation (Vietnamese and Danish translations were updated in 2.1.10b3). Other changes since 2.1.10b3 include:
- In 2.1.9, queue runner processing was made
~ more robust by making backups of queue entries when they were dequeued ~ so they could be recovered in the event of a system failure. This ~ opened the possibility that if a message itself caused a runner to ~ crash, a loop could result that would endlessly reprocess the message. ~ This has now been fixed by adding a dequeue count to the entry and ~ moving the entry aside and logging the fact after the third dequeue of ~ the same entry.
- Fixed the command line scripts add_members, sync_members and
~ clone_member to properly handle banned addresses (1904737).
- Fixed bin/newlist to add the list's preferred language to the list's
~ available_languages if it is other than the server's default language ~ (1906368).
- Changed the first URL in the RFC 2369 List-Unsubscribe: header to go
~ to the options login page instead of the listinfo page.
- Changed the options login page to not issue the "No address given"
~ error when coming from the List-Unsubscribe and other direct links. ~ Also changed to remember the user's language selection when ~ redisplaying the page following an error.
/Mark Sapiro
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
--On 13 March 2008 19:26:54 -0700 Mark Sapiro mark@msapiro.net wrote:
- Added a new 'sibling list' feature to exclude members of another list
from receiving a post from this list if the other list is in the To: or Cc: of the post or to include members of the other list if that list is not in the To: or Cc: of the post (Patch ID 1347962).
I don't understand this feature.... Hmm, Tokio Kikuchi asked about the name in 2005. I'm sorry that I didn't comment earlier.
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1347962&group_id=103
Sibling is a *completely* misleading term for this feature, because sibling relationships are necessarily symmetrical: if A is a sibling of B, then B must be a sibling of A. This feature is necessarily anti-symmetrical, more like "child" or "descendent". The term "sibling" will lead people to misconfigure their lists.
http://en.wikipedia.org/wiki/Symmetric_relation http://en.wikipedia.org/wiki/Antisymmetric_relation
Question: is the relationship also transitive? IE, if C is a child of B, and B is a child of A, then will postings to A go to members of C? If so, then the relationship should be called "descendent".
http://en.wikipedia.org/wiki/Transitive_relation
As far as inclusion is concerned, we then have a partially ordered set of mailing lists under this relationship. If the code handles the (presumably erroneous case) where a list is marked as a "sibling" of itself properly (ie, the listing should be ignored).
http://en.wikipedia.org/wiki/Partial_order
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mar 14, 2008, at 7:30 AM, Ian Eiloart wrote:
--On 13 March 2008 19:26:54 -0700 Mark Sapiro mark@msapiro.net
wrote:
- Added a new 'sibling list' feature to exclude members of
another list from receiving a post from this list if the other list is in the
To: or Cc: of the post or to include members of the other list if that
list is not in the To: or Cc: of the post (Patch ID 1347962).I don't understand this feature.... Hmm, Tokio Kikuchi asked about
the name in 2005. I'm sorry that I didn't comment earlier.<https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1347962...
Sibling is a *completely* misleading term for this feature, because
sibling relationships are necessarily symmetrical: if A is a sibling of B,
then B must be a sibling of A. This feature is necessarily anti- symmetrical, more like "child" or "descendent". The term "sibling" will lead people to misconfigure their lists.http://en.wikipedia.org/wiki/Symmetric_relation http://en.wikipedia.org/wiki/Antisymmetric_relation
Question: is the relationship also transitive? IE, if C is a child
of B, and B is a child of A, then will postings to A go to members of C?
If so, then the relationship should be called "descendent".http://en.wikipedia.org/wiki/Transitive_relation
As far as inclusion is concerned, we then have a partially ordered
set of mailing lists under this relationship. If the code handles the
(presumably erroneous case) where a list is marked as a "sibling" of itself
properly (ie, the listing should be ignored).
I didn't see any follow up on this, but perhaps I missed it while I
was at Pycon.
Tokio can correct me, but I do not believe this feature is
transitive. There is a strictly one-level inclusion or exclusion of
regular delivery addresses.
I see what you're saying about the term "sibling" lists being
misleading. But what's a good term? I can't think of anything that
encompasses both the inclusion and exclusion lists. "Related" is
about as close as I could come, but that's pretty uninformative. It
also might be too late to change for 2.1.10, especially if the u/i
strings have already been translated. Ultimately, it's Mark's
decision, and changing it would be predicated on finding a good
alternative.
- -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Barry Warsaw wrote: | On Mar 14, 2008, at 7:30 AM, Ian Eiloart wrote: | |> --On 13 March 2008 19:26:54 -0700 Mark Sapiro mark@msapiro.net |> wrote: | |>> - - Added a new 'sibling list' feature to exclude members of |>> another list |>> from receiving a post from this list if the other list is in the |>> To: or |>> Cc: of the post or to include members of the other list if that |>> list is |>> not in the To: or Cc: of the post (Patch ID 1347962). |> I don't understand this feature.... Hmm, Tokio Kikuchi asked about |> the name |> in 2005. I'm sorry that I didn't comment earlier. | |> <https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1347962...
|> Sibling is a *completely* misleading term for this feature, because |> sibling |> relationships are necessarily symmetrical: if A is a sibling of B, |> then B |> must be a sibling of A. This feature is necessarily anti- |> symmetrical, more |> like "child" or "descendent". The term "sibling" will lead people to |> misconfigure their lists. | |> http://en.wikipedia.org/wiki/Symmetric_relation |> http://en.wikipedia.org/wiki/Antisymmetric_relation | |> Question: is the relationship also transitive? IE, if C is a child |> of B, |> and B is a child of A, then will postings to A go to members of C? |> If so, |> then the relationship should be called "descendent". | |> http://en.wikipedia.org/wiki/Transitive_relation | |> As far as inclusion is concerned, we then have a partially ordered |> set of |> mailing lists under this relationship. If the code handles the |> (presumably |> erroneous case) where a list is marked as a "sibling" of itself |> properly |> (ie, the listing should be ignored). | |> http://en.wikipedia.org/wiki/Partial_order | | I didn't see any follow up on this, but perhaps I missed it while I | was at Pycon.
There wasn't any. At one point, I was going to reply, but I got distracted.
| Tokio can correct me, but I do not believe this feature is | transitive. There is a strictly one-level inclusion or exclusion of | regular delivery addresses.
That is correct.
| I see what you're saying about the term "sibling" lists being | misleading. But what's a good term? I can't think of anything that | encompasses both the inclusion and exclusion lists. "Related" is | about as close as I could come, but that's pretty uninformative.
Related (or Relative) was the alternative idea that occurred to me as well, but as you say, it isn't very informative.
| It | also might be too late to change for 2.1.10, especially if the u/i | strings have already been translated. Ultimately, it's Mark's | decision, and changing it would be predicated on finding a good | alternative.
The actual word "sibling" appears in a few places. It is in the title of the subsection for configuring regular_(in|ex)clude_lists, and is mentioned in the detailed help for these settings in the phrase "site administrator may prohibit cross domain siblings". It is also in the configuration variable ALLOW_CROSS_DOMAIN_SIBLING.
I would be reluctant to change this for 2.1.10 because the existing strings have been translated in a few updates, and I have been actively trying to keep from making any further changes to i18n strings.
As far as whether the 'sibling' relationship is symmetric or antisymmetric, this is a question that can only be answered if the definition of the relationship is precise enough. There are at least two possible definitions that make sense to me.
- List A is a sibling of list B iff list A appears in one of list B's
regular_(in|ex)clude_lists attributes.
- List A is a sibling of List B iff list A appears in one of list B's
regular_(in|ex)clude_lists attributes or list B appears in one of list A's regular_(in|ex)clude_lists attributes.
Under definition 1), the 'sibling' relationship is neither symmetric nor antisymmetric. Under definition 2), the 'sibling' relationship is symmetric and not antisymmetric.
As far as a list being in it's own regular_(in|ex)clude_lists, as a result of Ian's original post, I changed the GUI to not allow it and changed the handler to ignore it, but log it.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mar 27, 2008, at 4:09 PM, Mark Sapiro wrote:
The actual word "sibling" appears in a few places. It is in the
title of the subsection for configuring regular_(in|ex)clude_lists, and is mentioned in the detailed help for these settings in the phrase "site administrator may prohibit cross domain siblings". It is also in the configuration variable ALLOW_CROSS_DOMAIN_SIBLING.I would be reluctant to change this for 2.1.10 because the existing strings have been translated in a few updates, and I have been
actively trying to keep from making any further changes to i18n strings.As far as whether the 'sibling' relationship is symmetric or antisymmetric, this is a question that can only be answered if the definition of the relationship is precise enough. There are at least
two possible definitions that make sense to me.
- List A is a sibling of list B iff list A appears in one of list B's
regular_(in|ex)clude_lists attributes.
- List A is a sibling of List B iff list A appears in one of list B's
regular_(in|ex)clude_lists attributes or list B appears in one of list A's regular_(in|ex)clude_lists attributes.
Under definition 1), the 'sibling' relationship is neither symmetric
nor antisymmetric. Under definition 2), the 'sibling' relationship is symmetric and not antisymmetric.
I agree that it's too late to change this in 2.1.10, which probably
means it's too late to change for 2.1. I think you're follow up
rationale indicates that "sibling" is probably an acceptable term for
the feature given one way of looking at it, and because we have
nothing better. ;)
As far as a list being in it's own regular_(in|ex)clude_lists, as a result of Ian's original post, I changed the GUI to not allow it and changed the handler to ignore it, but log it.
Great, thanks!
- -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
For translators who haven't yet updated their translations for 2.1.10 and for other interested parties, I plan to release a 2.1.10 release candidate on April 14, 2008 and barring any issues with the RC, the 2.1.10 final on April 21, 2008.
Note to translators - the latest mailman.pot for the 2.1 branch is at https://code.launchpad.net/~mailman-coders/mailman/2.1. The SVN repository on sourceforge is no longer maintained and is out of date.
One way to get your updated translation to me is to register on launchpad and make your own private branch off the 2.1 branch, update your branch and publish it and send me a note when it's ready. Then I can just merge your branch back into the 2.1 branch.
Of course, if you prefer, you can just send me your updated message catalog and templates if any or make them available anywhere on the web and tell me where to find them.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 5, 2008, at 1:10 PM, Mark Sapiro wrote:
For translators who haven't yet updated their translations for 2.1.10 and for other interested parties, I plan to release a 2.1.10 release candidate on April 14, 2008 and barring any issues with the RC, the 2.1.10 final on April 21, 2008.
Sounds great, Mark. Thanks!
- -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I am happy to announce the release of Mailman 2.1.10rc1.
This is a security and bug fix release and it is highly recommended that all sites upgrade to this version. Mailman 2.1.10 also adds support for three new language translations, Galician, Hebrew and Slovak and a few new features.
Mailman is free software for managing email mailing lists and e- newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites.
For more information, including download links, please see:
http://www.list.org http://mailman.sf.net http://www.gnu.org/software/mailman
Special thanks are due to Barry Warsaw and Tokio Kikuchi for much coding and support, Moritz Naumann for help with security issues and Jim Tittsler for a significant patch.
Here's a list of the major changes.
Note in particular, the second item under Security as this is new since 2.1.10b4 and requires an mm_cfg.py change to maintain current behavior.
Security
- The 2.1.9 fixes for CVE-2006-3636 were not complete. In particular,
~ some potential cross-site scripting attacks were not detected in ~ editing templates and updating the list's info attribute via the web ~ admin interface. This has been assigned CVE-2008-0564 and has been ~ fixed. Thanks again to Moritz Naumann for assistance with this.
- There is a new mm_cfg.py/Defaults.py variable
~ OWNERS_CAN_CHANGE_MEMBER_PASSWORDS which controls whether the list ~ owner can change a member's password from the member's options page. ~ This defaults to No and should be changed to Yes only if list owners ~ are trusted to not change a member's password, log in as the member ~ and make global membership changes.
New Features
- Changed cmd_who.py to list all members if authorization is with the
~ list's admin or moderator password and to accept the password if the ~ roster is public. Also changed the web roster to show hidden members ~ when authorization is by site or list's admin or moderator password ~ (1587651).
- Added the ability to put a list name in accept_these_nonmembers
~ to accept posts from members of that list (1220144).
- Added a new 'sibling list' feature to exclude members of another list
~ from receiving a post from this list if the other list is in the To: ~ or Cc: of the post or to include members of the other list if that ~ list is not in the To: or Cc: of the post (Patch ID 1347962).
- Added the admin_member_chunksize attribute to the admin General
~ Options interface (Bug 1072002, Partial RFE 782436).
Internationalization
- Added the Hebrew translation from Dov Zamir. This includes addition
~ of a direction ('ltr', 'rtl') to the LC_DESCRIPTIONS table. The ~ add_language() function defaults direction to 'ltr' to not break ~ existing mm_cfg.py files.
- Added the Slovak translation from Martin Matuska.
- Added the Galician translation from Frco. Javier Rial Rodríguez.
Changes since 2.1.10b4 include the OWNERS_CAN_CHANGE_MEMBER_PASSWORDS setting mentioned above plus
- Changed cmd_subscribe.py to properly accept (no)digest without a
~ password and to recognize (no)digest and address= case insensitively.
- An updated mm-handler (mm-handler-2.1.10) that can help reduce
~ backscatter has been added to the contrib directory.
and updates to the Italian and Polish translations.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I am happy to announce the release of Mailman 2.1.10.
This is a security and bug fix release and it is highly recommended that all sites upgrade to this version. Mailman 2.1.10 also adds support for three new language translations, Galician, Hebrew and Slovak and a few new features.
Mailman is free software for managing email mailing lists and e- newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites.
For more information, including download links, please see:
http://www.list.org http://mailman.sf.net http://www.gnu.org/software/mailman
Special thanks are due to Barry Warsaw and Tokio Kikuchi for much coding and support, Moritz Naumann for help with security issues and Jim Tittsler for a significant patch.
Here's a list of the major changes.
Security
- The 2.1.9 fixes for CVE-2006-3636 were not complete. In particular,
some potential cross-site scripting attacks were not detected in editing templates and updating the list's info attribute via the web admin interface. This has been assigned CVE-2008-0564 and has been fixed. Thanks again to Moritz Naumann for assistance with this.
- There is a new mm_cfg.py/Defaults.py variable
OWNERS_CAN_CHANGE_MEMBER_PASSWORDS which controls whether the list owner can change a member's password from the member's options page. This defaults to No and should be changed to Yes only if list owners are trusted to not change a member's password, log in as the member and make global membership changes.
Note: If you are not ready to upgrade, patches for these two issues are available at http://sourceforge.net/project/showfiles.php?group_id=103 in the 2.1.9 file list.
New Features
- Changed cmd_who.py to list all members if authorization is with the
list's admin or moderator password and to accept the password if the roster is public. Also changed the web roster to show hidden members when authorization is by site or list's admin or moderator password (1587651).
- Added the ability to put a list name in accept_these_nonmembers
to accept posts from members of that list (1220144).
- Added a new 'sibling list' feature to exclude members of another list
from receiving a post from this list if the other list is in the To: or Cc: of the post or to include members of the other list if that list is not in the To: or Cc: of the post (Patch ID 1347962).
- Added the admin_member_chunksize attribute to the admin General Options
interface (Bug 1072002, Partial RFE 782436).
Internationalization
- Added the Hebrew translation from Dov Zamir. This includes addition of
a direction ('ltr', 'rtl') to the LC_DESCRIPTIONS table. The add_language() function defaults direction to 'ltr' to not break existing mm_cfg.py files.
- Added the Slovak translation from Martin Matuska.
- Added the Galician translation from Frco. Javier Rial Rodríguez.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I am happy to announce the release of Mailman 2.1.10.
Now running on mail.python.org. Please let us know if there are any problems.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 21, 2008, at 5:28 PM, Brad Knowles wrote:
Mark Sapiro wrote:
I am happy to announce the release of Mailman 2.1.10.
Now running on mail.python.org. Please let us know if there are any
problems.
Wow, awesome Brad. Thanks!
- -Barry
Mark Sapiro wrote:
I am happy to announce the release of Mailman 2.1.10.
This is a security and bug fix release and it is highly recommended that all sites upgrade to this version. Mailman 2.1.10 also adds support for three new language translations, Galician, Hebrew and Slovak and a few new features.
Remind me next time that we *MUST* upgrade to the betas and RCs on python.org once you've made them available. The changes made to the code which supports skipping unparseable messages means that mmdsr has to be changed to suit, otherwise you could wind up with a daily report that is 800MB in size, depending on how many "unparseable messages" you might have.
We'll need to coordinate the updates to mmdsr and get the official version with all code contributions from all parties out there on the SourceForge page.
Brad Knowles wrote:
Remind me next time that we *MUST* upgrade to the betas and RCs on python.org once you've made them available. The changes made to the code which supports skipping unparseable messages means that mmdsr has to be changed to suit, otherwise you could wind up with a daily report that is 800MB in size, depending on how many "unparseable messages" you might have.
We'll need to coordinate the updates to mmdsr and get the official version with all code contributions from all parties out there on the SourceForge page.
OK, I'll try to remember.
FYI, I've been running all the 2.1.10 beta and rc releases and mmdsr and haven't seen an issue, but I think in my case Postgrey/MailScanner must be getting rid of all the unparseable messages before they reach Mailman.
Note that in addition to the logging change for unparseable messages, they will now be saved in the shunt queue, so if you have a lot, you'll have to deal with that too.
Mark Sapiro wrote:
Note that in addition to the logging change for unparseable messages, they will now be saved in the shunt queue, so if you have a lot, you'll have to deal with that too.
These are the *.psv files? Yeah, we've got almost 8000 of them on python.org. Do you have any documentation on what should be done with them?
Brad Knowles wrote:
Mark Sapiro wrote:
Note that in addition to the logging change for unparseable messages, they will now be saved in the shunt queue, so if you have a lot, you'll have to deal with that too.
These are the *.psv files? Yeah, we've got almost 8000 of them on python.org. Do you have any documentation on what should be done with them?
Wow! I never thought there would be anything like this.
Here's the situation. Pre 2.1.9, when Mailman dequeued an incoming message and then the Python email library couldn't parse the MIME structure the message was just lost. There wasn't anything that could be done other than log the fact. This is not a big deal as the message's MIME structure was defective and the message was almost certainly spam.
Beginning in 2.1.9, we implemented the .bak files to back up an in process message so the message could be recovered if something died horribly (e.g. power failure) while it was in process. Nothing was done at that time about the unparseable messages.
I then realized that the unparseable message had an intact .bak file in the queue, so for 2.1.10, I decided to preserve this file (thus the .psv in the shunt queue) in case some human wanted to look at it with bin/dumpdb or whatever. Obviously, if you get 8000+ in a few hours, no one is going to look at them all or even much of a sample.
You could set up a cron to run every hour or some other interval to efectively do
rm $var_prefix/qfiles/shunt/*.psv
The problem with that is there can occasionally be queue entries preserved for other conditions which are hopefully much rarer, but you might actually want to look at those.
I think the best solution is to turn off the preservation of unparseable messages, and add an mm_cfg.py setting to turn it on. I can work up a patch.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Mark Sapiro wrote: | | I think the best solution is to turn off the preservation of | unparseable messages, and add an mm_cfg.py setting to turn it on. I | can work up a patch. |
A patch is attached. It doesn't turn off preservation by default because it piggybacks on an existing Defaults.py/mm_cfg.py setting which was lately used only by bin/update. The Defaults.py setting is
QRUNNER_SAVE_BAD_MESSAGES = Yes
Setting
QRUNNER_SAVE_BAD_MESSAGES = No
in mm_cfg.py will stop preserving the unparseable messages, but certain other problem qfiles will still be preserved. Additionally, preserved files will be stored in qfiles/bad and not qfiles/shunt. Finally, if QRUNNER_SAVE_BAD_MESSAGES = No, the log message will revert to the old "Ignoring unparseable message:".
Brad,
If you could apply this patch (you can apply it directly to the installation directory, by e.g.
cd /usr/local/mailman patch -p0 < path/to/2.1.10.patch.txt
and set
QRUNNER_SAVE_BAD_MESSAGES = No
in mm_cfg.py and restart Mailman, things should be back more or less the way they were in 2.1.9.
I have applied the patch to my installation and I'm sure it's good, but I haven't seen any unparseable messages.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
=== modified file 'Mailman/Queue/Runner.py' --- Mailman/Queue/Runner.py 2007-05-08 03:16:04 +0000 +++ Mailman/Queue/Runner.py 2008-04-22 02:02:38 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 1998-2007 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2008 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -103,12 +103,18 @@ # but other problems can occur in message parsing, e.g. # ValueError, and exceptions can occur in unpickling too. # We don't want the runner to die, so we just log and skip
# this entry, but preserve it for analysis.
# this entry, but maybe preserve it for analysis. self._log(e)
syslog('error','Skipping and preserving unparseable message: %s',filebase)self._switchboard.finish(filebase, preserve=True)
if mm_cfg.QRUNNER_SAVE_BAD_MESSAGES:syslog('error','Skipping and preserving unparseable message: %s',filebase)preserve=Trueelse:syslog('error','Ignoring unparseable message: %s', filebase)preserve=Falseself._switchboard.finish(filebase, preserve=preserve) continue try: self._onefile(msg, msgdata)
=== modified file 'Mailman/Queue/Switchboard.py' --- Mailman/Queue/Switchboard.py 2008-02-14 16:53:52 +0000 +++ Mailman/Queue/Switchboard.py 2008-04-22 02:11:19 +0000 @@ -169,13 +169,13 @@ bakfile = os.path.join(self.__whichq, filebase + '.bak') try: if preserve:
psvfile = os.path.join(mm_cfg.SHUNTQUEUE_DIR, filebase + '.psv')
psvfile = os.path.join(mm_cfg.BADQUEUE_DIR, filebase + '.psv') # Create the directory if it doesn't yet exist. # Copied from __init__. omask = os.umask(0) # rwxrws--- try: try:
os.mkdir(mm_cfg.SHUNTQUEUE_DIR, 0770)
os.mkdir(mm_cfg.BADQUEUE_DIR, 0770) except OSError, e: if e.errno <> errno.EEXIST: raise finally:
On 4/21/08, Mark Sapiro wrote:
If you could apply this patch (you can apply it directly to the installation directory, by e.g.
cd /usr/local/mailman patch -p0 < path/to/2.1.10.patch.txt
Patch applied fine, no complaints.
and set
QRUNNER_SAVE_BAD_MESSAGES = No
Done.
in mm_cfg.py and restart Mailman, things should be back more or less the way they were in 2.1.9.
I have applied the patch to my installation and I'm sure it's good, but I haven't seen any unparseable messages.
I haven't seen any more unparseable messages in the last few minutes, but let's see how things go.
On 4/21/08, Brad Knowles wrote:
I have applied the patch to my installation and I'm sure it's good, but I haven't seen any unparseable messages.
I haven't seen any more unparseable messages in the last few minutes, but let's see how things go.
We've now had a couple of unparseable messages since applying the patch, and it seems to be working as expected.
Going back to Jun 12 23:48:51 2007, it looks like we've had a total of about 378,989 unparseable messages, although we only have just over 8000 messages in the mailman/qfiles/shunt and mailman/qfiles/shunt.old directories (7759 .psv files, and 312 .pck files).
That works out to about 37,900 unparseable messages per month, or something like about 125 unparseable messages per day.
Brad Knowles wrote:
On 4/21/08, Brad Knowles wrote:
I have applied the patch to my installation and I'm sure it's good, but I haven't seen any unparseable messages.
I haven't seen any more unparseable messages in the last few minutes, but let's see how things go.
We've now had a couple of unparseable messages since applying the patch, and it seems to be working as expected.
Going back to Jun 12 23:48:51 2007, it looks like we've had a total of about 378,989 unparseable messages, although we only have just over 8000 messages in the mailman/qfiles/shunt and mailman/qfiles/shunt.old directories (7759 .psv files, and 312 .pck files).
That works out to about 37,900 unparseable messages per month, or something like about 125 unparseable messages per day.
The light dawns! (thanks to the post at http://mail.python.org/pipermail/mailman-users/2008-April/061365.html).
There was an error in 2.1.9 that didn't delete (or move aside) the .bak file in qfiles/in from an unparseable message. Thus, there were 7500+ unparseable message qfiles/in/*.bak files when you restarted mailman. These were all 'recovered' and reprocessed, reproducing the unparseable message error, and this time moved to qfiles/shunt/*.psv (so it wouldn't happen again).
This explains why there were over 7500 of these at once, when the normal rate is /only/ 125 per day.
Brad Knowles wrote:
Going back to Jun 12 23:48:51 2007, it looks like we've had a total of about 378,989 unparseable messages, although we only have just over 8000 messages in the mailman/qfiles/shunt and mailman/qfiles/shunt.old directories (7759 .psv files, and 312 .pck files).
There are two reasons why you've had 378,989 unparseable messages, but only have 7759 .psv files.
First, some of the unparseable .bak files that 2.1.9 left behind in qfiles/in may have been removed or moved aside.
Second, whenever Mailman was restarted, any qfiles/in/*.bak files would have been reprocessed resulting in a flood of unparseable message errors and leaving just one new .bak file per message. Thus a single message and .bak file could have been responsible for several logged unparseable message errors if Mailman had been restarted several times.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 21, 2008, at 9:30 PM, Mark Sapiro wrote:
You could set up a cron to run every hour or some other interval to efectively do
rm $var_prefix/qfiles/shunt/*.psv
The problem with that is there can occasionally be queue entries preserved for other conditions which are hopefully much rarer, but you might actually want to look at those.
I think the best solution is to turn off the preservation of unparseable messages, and add an mm_cfg.py setting to turn it on. I can work up a patch.
We should probably have some kind of shunt queue culler cron script in
place, either that archives and deletes those files, or just expires
them after a certain amount of time. What to people generally do with
their shunt files?
- -Barry
On 4/21/08, Barry Warsaw wrote:
We should probably have some kind of shunt queue culler cron script in place, either that archives and deletes those files, or just expires them after a certain amount of time.
That's easy enough to do with cron and find. You tell me what you want, and I'll be glad to set that up.
What to people generally do withtheir shunt files?
Leave them untouched for months or years? ;-)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 21, 2008, at 3:50 PM, Mark Sapiro wrote:
I am happy to announce the release of Mailman 2.1.10.
Congratulations Mark! Long live Mailman 2.2. :)
I will update the web sites.
- -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Mark Sapiro wrote: | I am happy to announce the release of Mailman 2.1.10.
I have discovered a few problems with the release. None is a major show stopper, but the most significant so far is that I broke cmd_subscribe so that email subscribe to the -subscribe or -join address or the
- -request address with a bare 'subscribe' command results in the message
being shunted. A patch for this is attached, but I plan to make a patch release probably next week.
Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
=== modified file 'Mailman/Commands/cmd_subscribe.py' --- Mailman/Commands/cmd_subscribe.py 2008-03-20 03:07:51 +0000 +++ Mailman/Commands/cmd_subscribe.py 2008-04-23 14:32:48 +0000 @@ -71,7 +71,8 @@ return STOP argnum += 1 # Fix the password/digest issue
- if digest is None and password.lower() in ('digest', 'nodigest'):
- if (digest is None
and password and password.lower() in ('digest', 'nodigest')): if password.lower() == 'digest': digest = 1 else:
participants (4)
-
Barry Warsaw -
Brad Knowles -
Ian Eiloart -
Mark Sapiro