Multi-slice qrunners to be fixed for 2.1.6? (Re: SF Patch 1008983)
Just wondering if this patch (http://sourceforge.net/tracker/index.php?func=detail&aid=1008983&group_id=103&atid=300103) has been noticed. It should have a fairly high priority, as it will result in any qrunner class with multiple slices failing completely in a short amount of time (hours to a few days, depending on the amount of traffic and number of slices).
The fix is trivial, merely clarifying an if statement, so it should have minimal impact on other areas.
Brian. ObOtherPatchPlug: Patch 1045656 reduces logfile chatter resulting from a lockfile race that does not impact operation.
Brian Greenberg grnbrg@gmail.com
Brian Greenberg wrote:
Just wondering if this patch (http://sourceforge.net/tracker/index.php?func=detail&aid=1008983&group_id=103&atid=300103) has been noticed.
Yes, I know it is there.
It should have a fairly high priority, as it will result in any qrunner class with multiple slices failing completely in a short amount of time (hours to a few days, depending on the amount of traffic and number of slices).
I really don't understand why the multiple feature is needed for mail delivery as multi-threading may be handled by MTA. It was left there because I am not in a position to test or qualify your patch immediately. If there is anyone who can backup this patch, I will be ready to merge in CVS.
The fix is trivial, merely clarifying an if statement, so it should have minimal impact on other areas.
I would like to ask patience here again as there were many FAQ related bug fixes in the past month. I always browse through patch/bug trackers whenever I have time.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
On Oct 21, 2004, at 4:37 PM, Tokio Kikuchi wrote:
Brian Greenberg wrote:
It should have a fairly high priority, as it will result in any qrunner class with multiple slices failing completely in a short amount of time (hours to a few days, depending on the amount of traffic and number of slices).
I really don't understand why the multiple feature is needed for mail delivery as multi-threading may be handled by MTA. It was left there because I am not in a position to test or qualify your patch immediately. If there is anyone who can backup this patch, I will be ready to merge in CVS.
The MTA cannot perform multithreading between multiple lists. For instance, if there is one list with 10,000 subscribers no other deliveries can be made while the emails to those 10,000 people are being transferred to the MTA. In some cases that may delay all other list delivery by several minutes and if that happens many times a day it can become problematic to end users. Having multiple outgoing qrunners allows other list mail to be delivered simultaneously.
I have also seen the bug this patch addresses happen. I saw one of the outgoing qrunner processes die within about 15 minutes of restarting mailman to have 4 of them. They continued to die (and be restarted by the master) for the next several hours. Once I applied the patch, the errors stopped.
I have not done any sort of extensive testing, but it has been working as expected for the past 3 days with the 3000 or so lists we have running on the machine.
DallasSo, you back up. That's I wanted to hear.
Dallas Bethune wrote:
On Oct 21, 2004, at 4:37 PM, Tokio Kikuchi wrote:
Brian Greenberg wrote:
It should have a fairly high priority, as it will result in any qrunner class with multiple slices failing completely in a short amount of time (hours to a few days, depending on the amount of traffic and number of slices).
I really don't understand why the multiple feature is needed for mail delivery as multi-threading may be handled by MTA. It was left there because I am not in a position to test or qualify your patch immediately. If there is anyone who can backup this patch, I will be ready to merge in CVS.
The MTA cannot perform multithreading between multiple lists. For instance, if there is one list with 10,000 subscribers no other deliveries can be made while the emails to those 10,000 people are being transferred to the MTA. In some cases that may delay all other list delivery by several minutes and if that happens many times a day it can become problematic to end users. Having multiple outgoing qrunners allows other list mail to be delivered simultaneously.
I have also seen the bug this patch addresses happen. I saw one of the outgoing qrunner processes die within about 15 minutes of restarting mailman to have 4 of them. They continued to die (and be restarted by the master) for the next several hours. Once I applied the patch, the errors stopped.
I have not done any sort of extensive testing, but it has been working as expected for the past 3 days with the 3000 or so lists we have running on the machine.
Dallas
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/tkikuchi%40is.koch...
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
At 4:54 PM -0700 2004-10-21, Dallas Bethune wrote:
The MTA cannot perform multithreading between multiple lists.
More accurately, the MTA is limited in its ability to apply multi-threaded solutions to problems when the input bottleneck from the qrunner is single-threaded.
Forinstance, if there is one list with 10,000 subscribers no other deliveries can be made while the emails to those 10,000 people are being transferred to the MTA. In some cases that may delay all other list delivery by several minutes and if that happens many times a day it can become problematic to end users. Having multiple outgoing qrunners allows other list mail to be delivered simultaneously.
Some of these problems can be addressed through improvements to MTA configuration, but it certainly sounds like the ability to have multiple qrunners going at the same time would allow the MTA a better chance to maximize the potential overall mailing list throughput that the server is capable of.
-- Brad Knowles, brad@stop.mail-abuse.org
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See http://www.sage.org/ for more info.
On Thu, 2004-10-21 at 13:59, Brian Greenberg wrote:
The fix is trivial, merely clarifying an if statement, so it should have minimal impact on other areas.
As I mentioned in the tracker comment, I think a better patch is to change the "if not lower" to "if lower is None".
-Barry
On Thu, 2004-10-21 at 19:54, Dallas Bethune wrote:
The MTA cannot perform multithreading between multiple lists. For instance, if there is one list with 10,000 subscribers no other deliveries can be made while the emails to those 10,000 people are being transferred to the MTA. In some cases that may delay all other list delivery by several minutes and if that happens many times a day it can become problematic to end users. Having multiple outgoing qrunners allows other list mail to be delivered simultaneously.
Note that the OutgoingRunner was designed to never (or almost never) need the list lock, so it should not block other access to the MailList while it's doing its job.
-Barry
- On 2004.10.21, in
- "Brad Knowles" brad@stop.mail-abuse.org wrote:
MTA configuration, but it certainly sounds like the ability to have multiple qrunners going at the same time would allow the MTA a better chance to maximize the potential overall mailing list throughput that the server is capable of.
Yes, yes, yes. I'll back up the problem here: we see this kind of situation regularly, even moreso before fixing a couple of other bugs in qrunner. We're still a touch behind in our Mailman version, and I'd been hoping an upgrade would cure qrunner -- so this is great, in my view.
-- -D. dgc@uchicago.edu NSIT::ENSS
participants (6)
-
Barry Warsaw -
Brad Knowles -
Brian Greenberg -
Dallas Bethune -
David Champion -
Tokio Kikuchi