
Folks,
I'd like to do a lot more testing of huge mailing lists. I think the largest I've seen people try (with not much success) is about 250k recipients. My goal would be for stock Mailman using a fairly simple installation on relatively common hardware to easily handle 1M recipients.
However, I'm at a slight disadvantage right now with my development environment. What I think would help would be for some of you to `donate' large blocks of fake addresses, which I could subscribe to email lists of varying sizes. I'd like to create lists of 1k, 10k, 100k, 250k, 500k, and 1M recipients. Do you think between us we can gather 1M fake recipients for a test list?
Best would be to have a mix of addresses on a number of sites, and I think for now, those addrs can just discard messages they receive. At some point it might be nice to collate receptions and send back a summary (e.g. "97k of 100k addresses received the message") -- but lets not worry about that for now.
What do you think? How many addresses do you think you could donate?
-Barry

On Tue, 23 May 2000, Barry A. Warsaw wrote:
What do you think? How many addresses do you think you could donate?
If you are not thinking in sending huge messages with fancy attachments, which I'm sure you are not, I think I could arrange 10-20 addresses connected to the bit bucket in 5-10 different machines.
-- Victoriano Giralt Systems Programmer Central Computing Facility University of Málaga SPAIN

"VG" == Victoriano Giralt <vic@vgg.sci.uma.es> writes:
VG> If you are not thinking in sending huge messages with fancy
VG> attachments, which I'm sure you are not, I think I could
VG> arrange 10-20 addresses connected to the bit bucket in 5-10
VG> different machines.
That would be great. Right now, I'm not concerned with message sizes, but list membership sizes. If my testing focus were to change, I'd ask beforehand.
Thanks for the offer, and I'll let you know when I'm ready.
-Barry

At 10:23 PM -0400 5/23/2000, bwarsaw@python.org wrote:
Message size isn't that significant unless you're doing full VERP. And even with VERP, it looks like message size is less important than the delays from DNS in resovling slow domains, especially international ones.
(this doesn't mean I'd run a 100,000 address list with full VERP on a 56K line, but assuming you have reasonable network bandwidth for the list size, you're more likely to run into DNS delays than anything else).
Also, remember that delivery issues are really the MTA's responsibility (and problem), not the MLM's -- the important thing for Mailman is to pass along the messages in a reasonable matter, and let the MTA (sendmail, postfix, etc) worry about optimizing delivery -- unless you want to integrate direct delivery into the list manager, and it's unclear that really wins you anything.
The important thing is to work on optimizing dropping mail into the MTA, and then letting the MTA do its job (and tune it!). In my work, the biggest problems is the way MLMs drop mail off for delivery -- usually by doing nothing more than a single-threaded drop sorted by a reversed address. That puts domain names together (but really ought to be a sort by MX instead, it's quite sub-optimal, especially with big domain-hosting sites like iname.com), but still limits delivery to how fast the MTA will accept addresses, and that's limited by DNS, since sendmail resolves addresses as it accepts mail...
So for something like mailman, being able to parallelize delivery into the MTA with multiple threads is a lot more important than anything else. segregrating slow-resolving domains from faster ones would be second, but easier said than done (but you can go a long way simply by delivering .com first, .net and .edu second, and then everything else...)
I'd guess you can get the first 90% simply by setting up mailman to deliver using four threads: .com, .net, .edu/.us/.ca and everything else, and allowing people to configure extra threads if the capacity allows (and I'd do that by splitting each feed in half), and then coming up with guides on how to tune the MTA for fast delivery (I'm just starting to figure out sendmail 8.10, but it looks like a nice improvement over earlier releases).
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> The important thing is to work on optimizing dropping mail
CVR> into the MTA, and then letting the MTA do its job (and tune
CVR> it!). In my work, the biggest problems is the way MLMs drop
CVR> mail off for delivery -- usually by doing nothing more than a
CVR> single-threaded drop sorted by a reversed address. That puts
CVR> domain names together (but really ought to be a sort by MX
CVR> instead, it's quite sub-optimal, especially with big
CVR> domain-hosting sites like iname.com), but still limits
CVR> delivery to how fast the MTA will accept addresses, and
CVR> that's limited by DNS, since sendmail resolves addresses as
CVR> it accepts mail...
This is very interesting. I've just added some code to SMTPDirect.py to support multiple MTA drop-off threads. I actually see worse performance with this code, which either means I screwed up the implementation or Something Else Is Going On.
i set up a 25k+ list and timed the drop-off. With sequential, single threaded delivery, I could deliver a small message to Postfix in about 41 seconds. Chunking the recips to 500 gave me about 50 chunks, and if I give each chunk it's own thread, I'm seeing no better than 58 seconds for delivery.
I have a suspicion about what else might be happening, but I'm not sure. Currently, Python has a fairly limited threading model. There is a global interpreter lock which only allows a single thread to be running Python code at any time. This works well if you're doing a lot of I/O, but not so well for other cpu intensive calculations. Eventually, Python will probably support "free threading" which will allow multiple threads running Python code.
Now to my eyes, the drop-off part is mostly I/O, shoving data across the socket, so I dunno. And maybe based on what Chuq says above, the threading approach would work better for Sendmail than for Postfix. So I think I will keep the code, but disable it by default and let you guys play with it. Please note that not all Python interpreters are built with threading enabled. It looks like the latest RH rpms are built with threading, but if you've built Python 1.5.2 from source, you'll have to explicitly configure --with-threads (I hope to change that for Python 1.6).
An alternative would be to fork off separate processes, but that seems too heavyweight, and makes collating the failures from the subprocesses more difficult.
CVR> I'd guess you can get the first 90% simply by setting up
CVR> mailman to deliver using four threads: .com, .net,
CVR> .edu/.us/.ca and everything else, and allowing people to
CVR> configure extra threads if the capacity allows (and I'd do
CVR> that by splitting each feed in half), and then coming up with
CVR> guides on how to tune the MTA for fast delivery (I'm just
CVR> starting to figure out sendmail 8.10, but it looks like a
CVR> nice improvement over earlier releases).
So with the next check-in, the chunking algorithm is to create 4 buckets: .com, .net/.org, .edu/.us/.ca, everything-else. Chunks in these buckets are no bigger than SMTP_MAX_RCPTS and buckets are not back-filled.
Eventually we'll need to separate out the entire hand-off process from the main process, but that's not currently feasible. I think this is the best I can do for 2.0.
-Barry

At 4:38 PM -0400 6/2/2000, bwarsaw@python.org wrote:
Are you talking about performance loading into the MTA? Or delivery?
I'll bet one thing in the Something Else category is disk I/O. That's what usually nukes MTA's first (and foremost).
the socket, so I dunno. And maybe based on what Chuq says above, the threading approach would work better for Sendmail than for Postfix.
I really, really need to sit down and beat postfix into a pulp. But I've got four sites to ship before I can (two of them based on mailman). I'm typing as fast as I can... but I really want to get a handle on postfix....
So I think I will keep the code, but disable it by default and let you guys play with it.
Very fair. It's nice the hooks are there, and IMHO, figuring out how to use it properly, tune is and all of that stuff sounds like a perfect reason for a 2.1 release. I sure wouldn't hold up other parts of 2.0 for this...
I think this is a great start, and I agree -- there's enough going on with 2.0 that we don't want to wait for this to release it. It'll give us time to sit down and research the beast in different environments more, too.
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
>> This is very interesting. I've just added some code to
>> SMTPDirect.py to support multiple MTA drop-off threads. I
>> actually see worse performance with this code, which either
>> means I screwed up the implementation or Something Else Is
>> Going On.
CVR> Are you talking about performance loading into the MTA? Or
CVR> delivery?
Loading into the MTA.
CVR> I'll bet one thing in the Something Else category is disk
CVR> I/O. That's what usually nukes MTA's first (and foremost).
Yes. I didn't even mention the times I was getting before shutting off syslogd for mail.* -- it was much worse. But the times I quoted are (AFAICT) just the I/O that Postfix does. Mailman isn't doing any different I/O for threaded delivery than for sequential delivery.
CVR> I really, really need to sit down and beat postfix into a
CVR> pulp.
Ooo, can I watch? :)
CVR> But I've got four sites to ship before I can (two of them
CVR> based on mailman). I'm typing as fast as I can... but I
CVR> really want to get a handle on postfix....
Cool.
>> So I think I will keep the code, but disable it by default and
>> let you guys play with it.
CVR> Very fair. It's nice the hooks are there, and IMHO, figuring
CVR> out how to use it properly, tune is and all of that stuff
CVR> sounds like a perfect reason for a 2.1 release. I sure
CVR> wouldn't hold up other parts of 2.0 for this...
>> Eventually we'll need to separate out the entire hand-off
>> process from the main process, but that's not currently
>> feasible. I think this is the best I can do for 2.0.
CVR> I think this is a great start, and I agree -- there's enough
CVR> going on with 2.0 that we don't want to wait for this to
CVR> release it. It'll give us time to sit down and research the
CVR> beast in different environments more, too.
That's a plan then. I have one more thing I want to play with for SMTPDirect.py and then that should be it. Guido's getting married this weekend so I'll be mostly off-line. Figure 2.0b3 early next week and then fast track toward 2.0 final. I'm ready to move on.
Cheers, -Barry

On Sat, 3 Jun 2000 00:31:26 -0400 (EDT) bwarsaw <bwarsaw@python.org> wrote:
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
Yes. I didn't even mention the times I was getting before shutting off syslogd for mail.* -- it was much worse.
Just setting syslog to not sync on every message can boost performance enourmously.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

On Tue, May 23, 2000 at 12:47:12PM -0400, Barry A. Warsaw wrote:
I guess you'd better define "relatively common hardware". What are we talking about here? Some benchmarking I did of real-world recpieints has shown that a dual PII 450 with a GB of RAM and the mail spool set up on a 500MB RAM disc, and upstream connectivity of dual DS3s, was able to handle around 100k deliveries an hour. This was using qmail with a remote concurrency of 250 and a bunch of little tricks.
So, unless I missed something pretty serious in the configuration of this environment, "relatively common hardware" should include at least a dozen boxes.
Again, this was a real-world list of around 400k addresses of users who had signed up to receive a particular announcement. They have since moved to a cluster of around 10 machines running some custom software and are getting over 1M per hour with that setup.
I could probably donate 5% to 10% of that amount across a few upstream lines.
Sean
What exactly would a "free, no-risk trial" of an on-line casino be? -- Evelyn Mitchell, 1998 Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

At 11:38 AM -0600 5/23/2000, Sean Reifschneider wrote:
If you're going to do this, you should generate a list that includes a range of addresses, including hard and soft bounces and successful deliveries. Which would imply addresses that are pre-set up to go to domains that voluntarily eat and/or bounce them.
you can also use any number of the spam holes out there (pretty much any domain one letter off from hotmail.com, for instance hotmal.com) that are preying on user typos to suck folks into web sites -- almost all of those reject mail or have disconnected SMTP completely, and since they're slime, I have no problem using them to test mail bouncing addresses.. (grin)
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> If you're going to do this, you should generate a list that
CVR> includes a range of addresses, including hard and soft
CVR> bounces and successful deliveries. Which would imply
CVR> addresses that are pre-set up to go to domains that
CVR> voluntarily eat and/or bounce them.
Very good idea.
CVR> you can also use any number of the spam holes out there
CVR> (pretty much any domain one letter off from hotmail.com, for
CVR> instance hotmal.com) that are preying on user typos to suck
CVR> folks into web sites -- almost all of those reject mail or
CVR> have disconnected SMTP completely, and since they're slime, I
CVR> have no problem using them to test mail bouncing
CVR> addresses.. (grin)
Heh. I'm not sure I'm prepared to use sites that don't volunteer to help, no matter how slimy they are :)
-Barry

"SR" == Sean Reifschneider <jafo-mailman-developers@tummy.com> writes:
SR> I guess you'd better define "relatively common hardware".
SR> What are we talking about here? Some benchmarking I did of
SR> real-world recpieints has shown that a dual PII 450 with a GB
SR> of RAM and the mail spool set up on a 500MB RAM disc, and
SR> upstream connectivity of dual DS3s, was able to handle around
SR> 100k deliveries an hour. This was using qmail with a remote
SR> concurrency of 250 and a bunch of little tricks.
Very interesting, and good numbers to know. I'm here with a single PIII 650, 256MB and no RAM disk. My currently limiting factor is my IP connection, but that should soon improve considerably.
The 100k deliveries/hr that your seeing -- what size messages are you seeing? Is that relatively constant as a product of the number of msgs/hr times the number of recipients/msg? Do you think Mailman is the limiting factor to your throughput and have you done any benchmarking of the MM software?
Right now, I'm primarily interested in seeing how MM handles very large recipient lists. Later I'd like to do some testing to see how much we can boost throughput, but I'd check with addr-volunteers before doing that kind of testing.
SR> So, unless I missed something pretty serious in the
SR> configuration of this environment, "relatively common
SR> hardware" should include at least a dozen boxes.
Which I definitely don't have access to.
SR> Again, this was a real-world list of around 400k addresses of
SR> users who had signed up to receive a particular announcement.
SR> They have since moved to a cluster of around 10 machines
SR> running some custom software and are getting over 1M per hour
SR> with that setup.
>> email lists of varying sizes. I'd like to create lists of 1k,
>> 10k, 100k, 250k, 500k, and 1M recipients. Do you think between
>> us we can gather 1M fake recipients for a test list?
SR> I could probably donate 5% to 10% of that amount across a few
SR> upstream lines.
That would be great. I'm not quite ready to start this kind of testing, but soon.
Thanks, -Barry

On Tue, May 23, 2000 at 10:29:21PM -0400, bwarsaw@python.org wrote:
Well, if you want to eliminate the IP and latency issues and just test Mailman, configure Mailman to go to a null SMTP or /usr/sbin/sendmail program. Eh? Or at the least use recipients at a local SMTP sink.
The 100k deliveries/hr that your seeing -- what size messages are you seeing? Is that relatively constant as a product of the number of
Hmm, those were 3 to 6k messages. I actually didn't see much in the way of change until the size went WAY up -- 50 to 100K. Most of the cost was dealing with file-system meta data -- writing 1K instead of 5K was fairly down in the noise when compared to the meta-data stuff, though to a lesser extent on the RAM disc.
This was not using Mailman at all. Based on the experience with the lists I run, I'd expect mailman to really pound on the machine in addition to this. These messages were generated using a custom program that had a number of processes feeding the queue, and would watch for the queue to fill up and then throttle. Especially important with stock QMail configurations where it's fairly easy to swamp the input queue. The big-todo patch helps this a lot (at around 25k messages with the stock, performance really drops off on ufs-like file-systems).
Which I definitely don't have access to.
We don't at the moment either, but check back later, we may have some spare boxes to run some tests. I also have a stripped down balls-to-the-wall SMTP server that you could use as a sink. This SMTP server on my old laptop (P133 with 40MB RAM and IDE hard drive) was able to handle over distinct incoming messages per second. (or around 360K per hour). I haven't really tested it on a more modern machine.
Sean
A ship in port is safe, but that is not what ships are for. -- Rear Admiral Grace Murray Hopper Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

On Tue, 23 May 2000 12:47:12 -0400 (EDT) Barry A Warsaw <bwarsaw@python.org> wrote:
I'll provisionally offer, ohh, two blocks, one of 5K addresses and another of 1K. Setup for the addresses will be as aliases piping directly to /dev/null.
For obvious reasons, traffic levels would have to be low.
-- J C Lawrence Internet: claw@kanga.nu ----------(*) Internet: coder@kanga.nu ...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...

J C Lawrence (claw@cp.net) wrote:
@some.dom.ain /dev/null
any number of addresses, almost instantly, postfix and sendmail have this functionality.
of course, you have to be careful to balance the bits in your mail or /dev/null could become charged with too many 1's.
P
-- Remember The 5 K's. The Justified Agents of Munya-munya-muuuu ...

"JCL" == J C Lawrence <claw@cp.net> writes:
JCL> I'll provisionally offer, ohh, two blocks, one of 5K
JCL> addresses and another of 1K. Setup for the addresses will be
JCL> as aliases piping directly to /dev/null.
JCL> For obvious reasons, traffic levels would have to be low.
Definitely. And thanks. I'll let you know when I'm ready for them.
-Barry

On Tue, May 23, 2000 at 12:47:12PM -0400, Barry A. Warsaw wrote:
Well, I'm not sure what number we could easily accomodate, it kind of depends on howmuch mail you want to send there, but I think we can manage something around 100, 200k, at least. Or perhaps I can do even better, setup a dedicated box for this testing and let you swamp away as much as you want -- at worst you'll get timeout warnings, but I suspect you'd want to test those too, anyway :-) The uplink is not the problem, it's solely the load of the machine that might be. We use three different incoming mailservers, for all our customers, so I'm a bit worried those mailman tests would take them out of commision if let run rampant ;-)
I can find an unused box and set it up for this sometime next week... (I'm currently at the SANE2000 conference, sharing a single 64kbit line with 30-some other system administrators... 45 second pingtimes do not create ideal working circumstances, let me tell you :-)
-- Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Tue, 23 May 2000, Barry A. Warsaw wrote:
What do you think? How many addresses do you think you could donate?
If you are not thinking in sending huge messages with fancy attachments, which I'm sure you are not, I think I could arrange 10-20 addresses connected to the bit bucket in 5-10 different machines.
-- Victoriano Giralt Systems Programmer Central Computing Facility University of Málaga SPAIN

"VG" == Victoriano Giralt <vic@vgg.sci.uma.es> writes:
VG> If you are not thinking in sending huge messages with fancy
VG> attachments, which I'm sure you are not, I think I could
VG> arrange 10-20 addresses connected to the bit bucket in 5-10
VG> different machines.
That would be great. Right now, I'm not concerned with message sizes, but list membership sizes. If my testing focus were to change, I'd ask beforehand.
Thanks for the offer, and I'll let you know when I'm ready.
-Barry

At 10:23 PM -0400 5/23/2000, bwarsaw@python.org wrote:
Message size isn't that significant unless you're doing full VERP. And even with VERP, it looks like message size is less important than the delays from DNS in resovling slow domains, especially international ones.
(this doesn't mean I'd run a 100,000 address list with full VERP on a 56K line, but assuming you have reasonable network bandwidth for the list size, you're more likely to run into DNS delays than anything else).
Also, remember that delivery issues are really the MTA's responsibility (and problem), not the MLM's -- the important thing for Mailman is to pass along the messages in a reasonable matter, and let the MTA (sendmail, postfix, etc) worry about optimizing delivery -- unless you want to integrate direct delivery into the list manager, and it's unclear that really wins you anything.
The important thing is to work on optimizing dropping mail into the MTA, and then letting the MTA do its job (and tune it!). In my work, the biggest problems is the way MLMs drop mail off for delivery -- usually by doing nothing more than a single-threaded drop sorted by a reversed address. That puts domain names together (but really ought to be a sort by MX instead, it's quite sub-optimal, especially with big domain-hosting sites like iname.com), but still limits delivery to how fast the MTA will accept addresses, and that's limited by DNS, since sendmail resolves addresses as it accepts mail...
So for something like mailman, being able to parallelize delivery into the MTA with multiple threads is a lot more important than anything else. segregrating slow-resolving domains from faster ones would be second, but easier said than done (but you can go a long way simply by delivering .com first, .net and .edu second, and then everything else...)
I'd guess you can get the first 90% simply by setting up mailman to deliver using four threads: .com, .net, .edu/.us/.ca and everything else, and allowing people to configure extra threads if the capacity allows (and I'd do that by splitting each feed in half), and then coming up with guides on how to tune the MTA for fast delivery (I'm just starting to figure out sendmail 8.10, but it looks like a nice improvement over earlier releases).
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> The important thing is to work on optimizing dropping mail
CVR> into the MTA, and then letting the MTA do its job (and tune
CVR> it!). In my work, the biggest problems is the way MLMs drop
CVR> mail off for delivery -- usually by doing nothing more than a
CVR> single-threaded drop sorted by a reversed address. That puts
CVR> domain names together (but really ought to be a sort by MX
CVR> instead, it's quite sub-optimal, especially with big
CVR> domain-hosting sites like iname.com), but still limits
CVR> delivery to how fast the MTA will accept addresses, and
CVR> that's limited by DNS, since sendmail resolves addresses as
CVR> it accepts mail...
This is very interesting. I've just added some code to SMTPDirect.py to support multiple MTA drop-off threads. I actually see worse performance with this code, which either means I screwed up the implementation or Something Else Is Going On.
i set up a 25k+ list and timed the drop-off. With sequential, single threaded delivery, I could deliver a small message to Postfix in about 41 seconds. Chunking the recips to 500 gave me about 50 chunks, and if I give each chunk it's own thread, I'm seeing no better than 58 seconds for delivery.
I have a suspicion about what else might be happening, but I'm not sure. Currently, Python has a fairly limited threading model. There is a global interpreter lock which only allows a single thread to be running Python code at any time. This works well if you're doing a lot of I/O, but not so well for other cpu intensive calculations. Eventually, Python will probably support "free threading" which will allow multiple threads running Python code.
Now to my eyes, the drop-off part is mostly I/O, shoving data across the socket, so I dunno. And maybe based on what Chuq says above, the threading approach would work better for Sendmail than for Postfix. So I think I will keep the code, but disable it by default and let you guys play with it. Please note that not all Python interpreters are built with threading enabled. It looks like the latest RH rpms are built with threading, but if you've built Python 1.5.2 from source, you'll have to explicitly configure --with-threads (I hope to change that for Python 1.6).
An alternative would be to fork off separate processes, but that seems too heavyweight, and makes collating the failures from the subprocesses more difficult.
CVR> I'd guess you can get the first 90% simply by setting up
CVR> mailman to deliver using four threads: .com, .net,
CVR> .edu/.us/.ca and everything else, and allowing people to
CVR> configure extra threads if the capacity allows (and I'd do
CVR> that by splitting each feed in half), and then coming up with
CVR> guides on how to tune the MTA for fast delivery (I'm just
CVR> starting to figure out sendmail 8.10, but it looks like a
CVR> nice improvement over earlier releases).
So with the next check-in, the chunking algorithm is to create 4 buckets: .com, .net/.org, .edu/.us/.ca, everything-else. Chunks in these buckets are no bigger than SMTP_MAX_RCPTS and buckets are not back-filled.
Eventually we'll need to separate out the entire hand-off process from the main process, but that's not currently feasible. I think this is the best I can do for 2.0.
-Barry

At 4:38 PM -0400 6/2/2000, bwarsaw@python.org wrote:
Are you talking about performance loading into the MTA? Or delivery?
I'll bet one thing in the Something Else category is disk I/O. That's what usually nukes MTA's first (and foremost).
the socket, so I dunno. And maybe based on what Chuq says above, the threading approach would work better for Sendmail than for Postfix.
I really, really need to sit down and beat postfix into a pulp. But I've got four sites to ship before I can (two of them based on mailman). I'm typing as fast as I can... but I really want to get a handle on postfix....
So I think I will keep the code, but disable it by default and let you guys play with it.
Very fair. It's nice the hooks are there, and IMHO, figuring out how to use it properly, tune is and all of that stuff sounds like a perfect reason for a 2.1 release. I sure wouldn't hold up other parts of 2.0 for this...
I think this is a great start, and I agree -- there's enough going on with 2.0 that we don't want to wait for this to release it. It'll give us time to sit down and research the beast in different environments more, too.
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
>> This is very interesting. I've just added some code to
>> SMTPDirect.py to support multiple MTA drop-off threads. I
>> actually see worse performance with this code, which either
>> means I screwed up the implementation or Something Else Is
>> Going On.
CVR> Are you talking about performance loading into the MTA? Or
CVR> delivery?
Loading into the MTA.
CVR> I'll bet one thing in the Something Else category is disk
CVR> I/O. That's what usually nukes MTA's first (and foremost).
Yes. I didn't even mention the times I was getting before shutting off syslogd for mail.* -- it was much worse. But the times I quoted are (AFAICT) just the I/O that Postfix does. Mailman isn't doing any different I/O for threaded delivery than for sequential delivery.
CVR> I really, really need to sit down and beat postfix into a
CVR> pulp.
Ooo, can I watch? :)
CVR> But I've got four sites to ship before I can (two of them
CVR> based on mailman). I'm typing as fast as I can... but I
CVR> really want to get a handle on postfix....
Cool.
>> So I think I will keep the code, but disable it by default and
>> let you guys play with it.
CVR> Very fair. It's nice the hooks are there, and IMHO, figuring
CVR> out how to use it properly, tune is and all of that stuff
CVR> sounds like a perfect reason for a 2.1 release. I sure
CVR> wouldn't hold up other parts of 2.0 for this...
>> Eventually we'll need to separate out the entire hand-off
>> process from the main process, but that's not currently
>> feasible. I think this is the best I can do for 2.0.
CVR> I think this is a great start, and I agree -- there's enough
CVR> going on with 2.0 that we don't want to wait for this to
CVR> release it. It'll give us time to sit down and research the
CVR> beast in different environments more, too.
That's a plan then. I have one more thing I want to play with for SMTPDirect.py and then that should be it. Guido's getting married this weekend so I'll be mostly off-line. Figure 2.0b3 early next week and then fast track toward 2.0 final. I'm ready to move on.
Cheers, -Barry

On Sat, 3 Jun 2000 00:31:26 -0400 (EDT) bwarsaw <bwarsaw@python.org> wrote:
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
Yes. I didn't even mention the times I was getting before shutting off syslogd for mail.* -- it was much worse.
Just setting syslog to not sync on every message can boost performance enourmously.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

On Tue, May 23, 2000 at 12:47:12PM -0400, Barry A. Warsaw wrote:
I guess you'd better define "relatively common hardware". What are we talking about here? Some benchmarking I did of real-world recpieints has shown that a dual PII 450 with a GB of RAM and the mail spool set up on a 500MB RAM disc, and upstream connectivity of dual DS3s, was able to handle around 100k deliveries an hour. This was using qmail with a remote concurrency of 250 and a bunch of little tricks.
So, unless I missed something pretty serious in the configuration of this environment, "relatively common hardware" should include at least a dozen boxes.
Again, this was a real-world list of around 400k addresses of users who had signed up to receive a particular announcement. They have since moved to a cluster of around 10 machines running some custom software and are getting over 1M per hour with that setup.
I could probably donate 5% to 10% of that amount across a few upstream lines.
Sean
What exactly would a "free, no-risk trial" of an on-line casino be? -- Evelyn Mitchell, 1998 Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

At 11:38 AM -0600 5/23/2000, Sean Reifschneider wrote:
If you're going to do this, you should generate a list that includes a range of addresses, including hard and soft bounces and successful deliveries. Which would imply addresses that are pre-set up to go to domains that voluntarily eat and/or bounce them.
you can also use any number of the spam holes out there (pretty much any domain one letter off from hotmail.com, for instance hotmal.com) that are preying on user typos to suck folks into web sites -- almost all of those reject mail or have disconnected SMTP completely, and since they're slime, I have no problem using them to test mail bouncing addresses.. (grin)
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"

"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> If you're going to do this, you should generate a list that
CVR> includes a range of addresses, including hard and soft
CVR> bounces and successful deliveries. Which would imply
CVR> addresses that are pre-set up to go to domains that
CVR> voluntarily eat and/or bounce them.
Very good idea.
CVR> you can also use any number of the spam holes out there
CVR> (pretty much any domain one letter off from hotmail.com, for
CVR> instance hotmal.com) that are preying on user typos to suck
CVR> folks into web sites -- almost all of those reject mail or
CVR> have disconnected SMTP completely, and since they're slime, I
CVR> have no problem using them to test mail bouncing
CVR> addresses.. (grin)
Heh. I'm not sure I'm prepared to use sites that don't volunteer to help, no matter how slimy they are :)
-Barry

"SR" == Sean Reifschneider <jafo-mailman-developers@tummy.com> writes:
SR> I guess you'd better define "relatively common hardware".
SR> What are we talking about here? Some benchmarking I did of
SR> real-world recpieints has shown that a dual PII 450 with a GB
SR> of RAM and the mail spool set up on a 500MB RAM disc, and
SR> upstream connectivity of dual DS3s, was able to handle around
SR> 100k deliveries an hour. This was using qmail with a remote
SR> concurrency of 250 and a bunch of little tricks.
Very interesting, and good numbers to know. I'm here with a single PIII 650, 256MB and no RAM disk. My currently limiting factor is my IP connection, but that should soon improve considerably.
The 100k deliveries/hr that your seeing -- what size messages are you seeing? Is that relatively constant as a product of the number of msgs/hr times the number of recipients/msg? Do you think Mailman is the limiting factor to your throughput and have you done any benchmarking of the MM software?
Right now, I'm primarily interested in seeing how MM handles very large recipient lists. Later I'd like to do some testing to see how much we can boost throughput, but I'd check with addr-volunteers before doing that kind of testing.
SR> So, unless I missed something pretty serious in the
SR> configuration of this environment, "relatively common
SR> hardware" should include at least a dozen boxes.
Which I definitely don't have access to.
SR> Again, this was a real-world list of around 400k addresses of
SR> users who had signed up to receive a particular announcement.
SR> They have since moved to a cluster of around 10 machines
SR> running some custom software and are getting over 1M per hour
SR> with that setup.
>> email lists of varying sizes. I'd like to create lists of 1k,
>> 10k, 100k, 250k, 500k, and 1M recipients. Do you think between
>> us we can gather 1M fake recipients for a test list?
SR> I could probably donate 5% to 10% of that amount across a few
SR> upstream lines.
That would be great. I'm not quite ready to start this kind of testing, but soon.
Thanks, -Barry

On Tue, May 23, 2000 at 10:29:21PM -0400, bwarsaw@python.org wrote:
Well, if you want to eliminate the IP and latency issues and just test Mailman, configure Mailman to go to a null SMTP or /usr/sbin/sendmail program. Eh? Or at the least use recipients at a local SMTP sink.
The 100k deliveries/hr that your seeing -- what size messages are you seeing? Is that relatively constant as a product of the number of
Hmm, those were 3 to 6k messages. I actually didn't see much in the way of change until the size went WAY up -- 50 to 100K. Most of the cost was dealing with file-system meta data -- writing 1K instead of 5K was fairly down in the noise when compared to the meta-data stuff, though to a lesser extent on the RAM disc.
This was not using Mailman at all. Based on the experience with the lists I run, I'd expect mailman to really pound on the machine in addition to this. These messages were generated using a custom program that had a number of processes feeding the queue, and would watch for the queue to fill up and then throttle. Especially important with stock QMail configurations where it's fairly easy to swamp the input queue. The big-todo patch helps this a lot (at around 25k messages with the stock, performance really drops off on ufs-like file-systems).
Which I definitely don't have access to.
We don't at the moment either, but check back later, we may have some spare boxes to run some tests. I also have a stripped down balls-to-the-wall SMTP server that you could use as a sink. This SMTP server on my old laptop (P133 with 40MB RAM and IDE hard drive) was able to handle over distinct incoming messages per second. (or around 360K per hour). I haven't really tested it on a more modern machine.
Sean
A ship in port is safe, but that is not what ships are for. -- Rear Admiral Grace Murray Hopper Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

On Tue, 23 May 2000 12:47:12 -0400 (EDT) Barry A Warsaw <bwarsaw@python.org> wrote:
I'll provisionally offer, ohh, two blocks, one of 5K addresses and another of 1K. Setup for the addresses will be as aliases piping directly to /dev/null.
For obvious reasons, traffic levels would have to be low.
-- J C Lawrence Internet: claw@kanga.nu ----------(*) Internet: coder@kanga.nu ...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...

J C Lawrence (claw@cp.net) wrote:
@some.dom.ain /dev/null
any number of addresses, almost instantly, postfix and sendmail have this functionality.
of course, you have to be careful to balance the bits in your mail or /dev/null could become charged with too many 1's.
P
-- Remember The 5 K's. The Justified Agents of Munya-munya-muuuu ...

"JCL" == J C Lawrence <claw@cp.net> writes:
JCL> I'll provisionally offer, ohh, two blocks, one of 5K
JCL> addresses and another of 1K. Setup for the addresses will be
JCL> as aliases piping directly to /dev/null.
JCL> For obvious reasons, traffic levels would have to be low.
Definitely. And thanks. I'll let you know when I'm ready for them.
-Barry

On Tue, May 23, 2000 at 12:47:12PM -0400, Barry A. Warsaw wrote:
Well, I'm not sure what number we could easily accomodate, it kind of depends on howmuch mail you want to send there, but I think we can manage something around 100, 200k, at least. Or perhaps I can do even better, setup a dedicated box for this testing and let you swamp away as much as you want -- at worst you'll get timeout warnings, but I suspect you'd want to test those too, anyway :-) The uplink is not the problem, it's solely the load of the machine that might be. We use three different incoming mailservers, for all our customers, so I'm a bit worried those mailman tests would take them out of commision if let run rampant ;-)
I can find an unused box and set it up for this sometime next week... (I'm currently at the SANE2000 conference, sharing a single 64kbit line with 30-some other system administrators... 45 second pingtimes do not create ideal working circumstances, let me tell you :-)
-- Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
participants (8)
-
bwarsaw@python.org
-
Chuq Von Rospach
-
J C Lawrence
-
J C Lawrence
-
Peter Evans
-
Sean Reifschneider
-
Thomas Wouters
-
Victoriano Giralt