Can anyone offer any suggestions on resolving a high cpu load with Python.
We're using Mailman 2.1.4, Python 2.3.3, Sendmail & Amavis to support an announcement only list of about 40k members on an IBM Netfinity server (1 Gb RAM).
During list activity CPU load will often exceed 90% with the load being spread fairly evenly between active qrunners (e.g. command, outgoing & virgin runners).
This I can live with but what I can't live with is the 98% load generated from the web admin interface (driver admin script) which effectively makes the interface unusable. In fact mailmanctl must be shut down in order to do any remote administration at all!
I've scoured the archives and also tried most options within 'mm_cfg.py' all without success.
It's probably worth pointing out that the same issues existed with previous versions of Python and Mailman.
Suggestions anyone?
--
Jason Shouler DATAFLOW Tel/Fax : +44 (0)1202-722599 Mobile : +44 (0)7946-615718 www.data-flow.org.uk Linux integration specialists
At 10:24 PM +0100 2004/05/30, Jason Shouler wrote:
Can anyone offer any suggestions on resolving a high cpu load with Python.
We're using Mailman 2.1.4, Python 2.3.3, Sendmail & Amavis to support an announcement only list of about 40k members on an IBM Netfinity server (1 Gb RAM).
During list activity CPU load will often exceed 90% with the load being spread fairly evenly between active qrunners (e.g. command, outgoing & virgin runners).
Have you seen the Mailman FAQ entry at <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.041.htp>?
This I can live with but what I can't live with is the 98% load generated from the web admin interface (driver admin script) which effectively makes the interface unusable. In fact mailmanctl must be shut down in order to do any remote administration at all!
I've seen reports of high load with qrunners, and created the above FAQ entry to consolidate the information that I had seen to that point in time. I do not recall seeing reports of high load with the web interface, however.
I've scoured the archives and also tried most options within 'mm_cfg.py' all without success.
It's probably worth pointing out that the same issues existed with previous versions of Python and Mailman.
I'd suggest upgrading to versions 2.1.5, and trying the other suggestions recommended by the FAQ. If one or the other or both of those don't solve the problem, please let us know.
-- Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania.
SAGE member since 1995. See <http://www.sage.org/> for more info.
Thanks for the suggestions
I'd suggest upgrading to versions 2.1.5, and trying the other suggestions recommended by the FAQ. If one or the other or both of those don't solve the problem, please let us know.
Sorry, this was a typo - we're already up to 2.1.5
The Netfinity (PII 450 single cpu - Twin RAID 5) is running SuSE 8.0
I'll get back to you after further tests.
--
Jason Shouler DATAFLOW Tel/Fax : +44 (0)1202-722599 Mobile : +44 (0)7946-615718 www.data-flow.org.uk Linux integration specialists
At 11:57 PM +0100 2004/05/30, Jason Shouler wrote:
Sorry, this was a typo - we're already up to 2.1.5
Ahh.The Netfinity (PII 450 single cpu - Twin RAID 5) is running SuSE 8.0
Hmm. RAID-5 can cause problems with latency in synchronous meta-data update performance.
Just to be sure, you should also take a look at <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.011.htp>, <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.012.htp>, <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.003.htp>, <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.006.htp>, and <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.008.htp>.
I'll get back to you after further tests.
Please do. We may need to update the FAQ entries.-- Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania.
SAGE member since 1995. See <http://www.sage.org/> for more info.
On 30 May 2004, at 22:24, Jason Shouler wrote:
Can anyone offer any suggestions on resolving a high cpu load with Python.
We're using Mailman 2.1.4, Python 2.3.3, Sendmail & Amavis to support an announcement only list of about 40k members
It might be worth considering splitting a list with a fairly big set of subscribers into a series of sub lists (say alphabetical by subscriber name) and making those sublists the subscribers of the main list. Mailman uses pickled Python structures for the per-list "databases" which with large numbers of subscribers is stretching things a bit.
on an IBM Netfinity server (1 Gb RAM).
"IBM Netfinity server" does tell us much about what hardware and OS & version is supporting things.
During list activity
What do you mean by list activity? Handling a single outgoing post or what.
CPU load will often exceed 90% with the load being spread fairly evenly between active qrunners (e.g. command, outgoing & virgin runners).
How many posts per day are you handling via this lists. How much subscribe/unsubscribe and list admin activity are you talking about?
Does anything of significance regarding problems show up in the Mailman or MTA logs?
This I can live with but what I can't live with is the 98% load generated from the web admin interface (driver admin script) which effectively makes the interface unusable.
If this is happening when you do web gui list admin stuff on this very large list it tends to reinforce my view that breaking that list into a top-level list with a series of sublists, each with fewer subscribers, might be worthwhile; smaller pickles to load and save.
In fact mailmanctl must be shut down in order to do any remote administration at all!
Also have you checked for and cleared out any lurking lock files.
I've scoured the archives and also tried most options within 'mm_cfg.py' all without success.
Is outgoing mail via the MTA on the Mailman server? What sort of values are are using for SMTP_MAX_RCPTS?
It's probably worth pointing out that the same issues existed with previous versions of Python and Mailman.
Same list, same problem, maybe ...
Suggestions anyone?
============================= Jason Shouler DATAFLOW Tel/Fax : +44 (0)1202-722599 Mobile : +44 (0)7946-615718 www.data-flow.org.uk Linux integration specialists
Richard Barrett wrote:
What do you mean by list activity? Handling a single outgoing post or what.
Activity should be pretty minimal - one outgoing post a week and that's about it. What appears to be creating the heaviest load is the bounce runner which with something over 3000 messages in it's queue looks a likely culprit (or rather I guess the messages are).
I'm not sure how the bounce queue fits in with the bounce mechanism?
Are these actually queued messages waiting for delivery or are they
already dead! I've looked at a few and they would appear to belong to
unsubscribed members??
Is outgoing mail via the MTA on the Mailman server? What sort of values are are using for SMTP_MAX_RCPTS?
All services are running on the one server. Sendmail is fairly well optimized and takes submissions from Mailman via the MSA port which is configured to avoid DNS. SMTP_MAX_RCPTS = 10 and VERP is enabled for everything other than normal postings and of course the SMTPPORT is changed.
Basically Mailman works very well aside from the heavy load issue.
As is often the case in these situations, after almost giving up, I've now found the web interface is usable again.
All the queue runners are clear apart from the bounce runner which is going up very gradually at the moment. When I see it going down I'll be a lot happier.
Regards,
Jason.
--
Jason Shouler DATAFLOW Tel/Fax : +44 (0)1202-722599 Mobile : +44 (0)7946-615718 www.data-flow.org.uk Linux integration specialists
On 31 May 2004, at 20:59, Jason Shouler wrote:
Richard Barrett wrote:
What do you mean by list activity? Handling a single outgoing post or what.
Activity should be pretty minimal - one outgoing post a week and that's about it. What appears to be creating the heaviest load is the bounce runner which with something over 3000 messages in it's queue looks a likely culprit (or rather I guess the messages are). I'm not sure how the bounce queue fits in with the bounce mechanism?
Are these actually queued messages waiting for delivery or are they already dead! I've looked at a few and they would appear to belong to unsubscribed members??Is outgoing mail via the MTA on the Mailman server? What sort of values are are using for SMTP_MAX_RCPTS?
All services are running on the one server. Sendmail is fairly well optimized and takes submissions from Mailman via the MSA port which is configured to avoid DNS. SMTP_MAX_RCPTS = 10 and VERP is enabled for everything other than normal postings and of course the SMTPPORT is changed.
Basically Mailman works very well aside from the heavy load issue.
As is often the case in these situations, after almost giving up, I've now found the web interface is usable again.
All the queue runners are clear apart from the bounce runner which is going up very gradually at the moment. When I see it going down I'll be a lot happier.
Take a look at the NEWS file in the Mailman source distribution; grep for 'bounce' and read the items that are identified. There were significant bounce handling changes in MM 2.1.3 and MM 2.1.5 the both of which will influence your system's operation as regards things you are noting above. Also, take a look at the Mailman bounce log.
Maybe changing REGISTER_BOUNCES_EVERY config variable's value in mm_cfg.py and doing mailmanctl restart might be worth considering.
Be useful if you could let me/us have your summary/conclusions/actions when (if) you manage to get a satisfactory resolution.
Regards,
Jason.
--
Jason Shouler DATAFLOW Tel/Fax : +44 (0)1202-722599 Mobile : +44 (0)7946-615718 www.data-flow.org.uk Linux integration specialists
participants (3)
-
Brad Knowles -
Jason Shouler -
Richard Barrett