I've been doing some (limited) performance testing lately, and I wanted to share some numbers and get some feedback. I've also been having fun re-reading some old mm-dev threads related to performance. :)
I'm specifically looking for places to improve Mailman's raw throughput. I understand that MTA tuning can have a huge impact on the system, but I think that subject's been hashed out quite well in the past. On the table are anything from low-hanging fruit hacks to mm3-level redesigns. What I actually can implement all depends. :)
I've been testing the following set up:
Postfix 2.0.9 configured with a special test transport such that all email @example.com gets dd'd to /dev/null
Postfix running on the same machine as Mailman 2.1.2+ and a second test with Postfix (similarly configured) on a separate, very unloaded, but less beefy machine sitting next to me on a 100Mb ethernet.
RH9 2.4.20-9 kernel, 863Mhz Dell PIII, 512MB (1723 bogomips), ext3, a WDC IDE drive of some 2 y.o. vintage.
Python 2.2.2 built from source
My list consists of 8000 members like abcdefg@example.com where the localpart varies randomly. I've tried deliveries of 10KB, 50KB, 220KB, 1MB of text/plain and a 220KB multipart/related snapshot of a web page [1]. I have VERP and personalization both turned on. I started looking at memory usage, but I'm not so concerned about that now. It may be something to address later but I think it's "reasonable".
First the (approximate) numbers. All deliveries are to 8000 members, each with their own personalized copy. SMTP_MAX_RCPTS is 500 unless otherwise specified (minimal impact seemingly).
msgsz type time msg/hour
10k plain/text 6min 80k/hr 50k p/t 9.5min 50k/hr (SMTP_MAX_RCPTS=5) 220k p/t 24m 20k/hr 1MB p/t 105m 4500/hr 220k m/related 44m 10k/hr 220k m/related 41m 11k/hr (SMTP_MAX_RCPTS=5) 220k m/related 46m 10k/hr (remote MTA)
A few high-level bits:
Disk i/o probably isn't much of an issue. Once the message hits the out qrunner, it's only two files and all the personalization weaving happens in memory just before the message goes out on the socket. Since using a remote MTA was actually slightly slower, I'm guessing that MTA overhead in the /dev/null pipe is actually minimal (the remote machine is a 500MHz, 128 KB, 999 bogomips, mostly idle).
email.Generator.Generator (and email.Parser.Parser) are good candidates for optimization. You can see that with the 220KB messages, the fact that one has structure and the other doesn't, affects performance significantly. That doesn't surprise me. ;)
Even so, a factor of 100 in message size has a 20x hit on performance. Part of that may be the way the personalization weaving gets done. Right now, we make a copy.deepcopy() of the original message object model, then poke in the personalization parts in the headers and such, then do all the complex stuff in Decorate.py (footers, headers, etc.), then generate the flat text. Maybe we can speed things up by converting the message to flat text as early as possible and just doing string substitution at the point of weaving.
What kind of a hit does the memberdb-in-a-pickle take? Would things go faster if we stored the member data in a Berkeley, MySQL, or other real database? I'd like to do some testing with my BDB member code and I'm wondering if the folks working on other member adapters have any performance feedback.
XVERP might be interesting, but it seems useless for personalization.
Do we win or lose with the process model, as compared to say, a threading model? I've been wondering if our fears of the Python GIL are unfounded. We could certainly reduce memory overhead by multi-threading, and we might be able to leverage something like Twisted, which is still in the back of my mind as a very cool way to get multi-protocol support into Mailman.
Does our "NFS-safe" locks impose too much complexity and overhead to be worth it? Does anybody actually /use/ Mailman over NFS? Don't we sorta suspect the LockFile implementation anyway? Would we be better off using kernel file locks, or thread locks if we go to a MT model?
Okay, now I'm rambling. What is the lowest hanging fruit that we might be able to attack? I'm up for any other ideas people have.
-Barry
[1] wget -E -H -k -p -nH -nd -Pdownload <url> followed by a little Python script to multipart/related it
Barry Warsaw wrote:
- What kind of a hit does the memberdb-in-a-pickle take? Would things go faster if we stored the member data in a Berkeley, MySQL, or other real database? I'd like to do some testing with my BDB member code and I'm wondering if the folks working on other member adapters have any performance feedback.
Apart from the fact that you'll have tough times when loading 100k members into memory, the use of a real database is a tradeoff:
Mailman has lots of code which is built in a way that assumes membership data access is fast. With a few 1000 members, things can still fit nicely into memory, so that's a valid assumption. However, with a few 100k members you really want to be more careful and only load data in chunks into memory.
Mailman needs to be redesigned in a couple of places for that to work. One example is the membership admin interface, another is personalization.
So, no, you don't gain performance by putting small lists into a database and, yes, if your list size grows beyond certain limits, there's simply no alternative. Once your machine starts swapping, the database + chunking approach is faster.
- Do we win or lose with the process model, as compared to say, a threading model? I've been wondering if our fears of the Python GIL are unfounded. We could certainly reduce memory overhead by multi-threading, and we might be able to leverage something like Twisted, which is still in the back of my mind as a very cool way to get multi-protocol support into Mailman.
I don't think that multi-threading would gain any performance. It would make more sense to have Mailman use multiple SMTP backends for delivery (MTA clustering).
BTW, is Mailman thread-safe ?
- Does our "NFS-safe" locks impose too much complexity and overhead to be worth it? Does anybody actually /use/ Mailman over NFS? Don't we sorta suspect the LockFile implementation anyway? Would we be better off using kernel file locks, or thread locks if we go to a MT model?
You may want to have a look at mx.Misc.FileLock (in egenix-mx-base). That's a portable file locking mechanism which is not the fastest, but fast enough for most cases.
Okay, now I'm rambling. What is the lowest hanging fruit that we might be able to attack? I'm up for any other ideas people have.
MTA clustering support. Basically just do round-robin delivery to a list of SMTP hosts.
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, May 14 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
EuroPython 2003, Charleroi, Belgium: 41 days left
On Wed, 2003-05-14 at 12:41, M.-A. Lemburg wrote:
Apart from the fact that you'll have tough times when loading 100k members into memory, the use of a real database is a tradeoff:
Mailman has lots of code which is built in a way that assumes membership data access is fast. With a few 1000 members, things can still fit nicely into memory, so that's a valid assumption. However, with a few 100k members you really want to be more careful and only load data in chunks into memory.
Mailman needs to be redesigned in a couple of places for that to work. One example is the membership admin interface, another is personalization.
I totally agree. I've been fleshing out some api's for mm3 that attempt to address these exact issues. I'm not ready to show them to the world yet, but I'm thinking exactly along these lines.
I don't think that multi-threading would gain any performance. It would make more sense to have Mailman use multiple SMTP backends for delivery (MTA clustering).
Here's one place where it would help: with multiple threads, we won't need to keep multiple copies of the databases in memory. Maybe chunking and caching can help with this instead of moving to a threading model.
BTW, is Mailman thread-safe ?
Depends. If we use one thread per qrunner, then I think so.
- Does our "NFS-safe" locks impose too much complexity and overhead to be worth it? Does anybody actually /use/ Mailman over NFS? Don't we sorta suspect the LockFile implementation anyway? Would we be better off using kernel file locks, or thread locks if we go to a MT model?
You may want to have a look at mx.Misc.FileLock (in egenix-mx-base). That's a portable file locking mechanism which is not the fastest, but fast enough for most cases.
I'll look: in the meantime, how does it do locking?
Okay, now I'm rambling. What is the lowest hanging fruit that we might be able to attack? I'm up for any other ideas people have.
MTA clustering support. Basically just do round-robin delivery to a list of SMTP hosts.
I'm pretty sure there are folks on this list that are doing this. We may want to make it easier to configure though.
-Barry
At 17:20 14/05/2003, Barry Warsaw wrote:
big snip, because I am still thinking about the bulk of your post
- Does our "NFS-safe" locks impose too much complexity and overhead to be worth it? Does anybody actually /use/ Mailman over NFS? Don't we sorta suspect the LockFile implementation anyway? Would we be better off using kernel file locks, or thread locks if we go to a MT model?
Mailman runs entirely out of NFS file space on my major client's site. It is central to the backup and resilience (backup server) strategy for that site. I would want to retain NFS locking as an option.
Okay, now I'm rambling. What is the lowest hanging fruit that we might be able to attack? I'm up for any other ideas people have.
-Barry
[1] wget -E -H -k -p -nH -nd -Pdownload <url> followed by a little Python script to multipart/related it
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Richard Barrett http://www.openinfo.co.uk
On Wed, 2003-05-14 at 12:47, Richard Barrett wrote:
Mailman runs entirely out of NFS file space on my major client's site. It is central to the backup and resilience (backup server) strategy for that site. I would want to retain NFS locking as an option.
Thanks for the data point. -Barry
participants (3)
-
Barry Warsaw -
M.-A. Lemburg -
Richard Barrett