[Mailman-Developers] Huge lists

[Sean Reifschneider]

On Tue, May 23, 2000 at 10:29:21PM -0400, bwarsaw@python.org wrote:
>Very interesting, and good numbers to know.  I'm here with a single
>PIII 650, 256MB and no RAM disk.  My currently limiting factor is my
>IP connection, but that should soon improve considerably.

Well, if you want to eliminate the IP and latency issues and just
test Mailman, configure Mailman to go to a null SMTP or /usr/sbin/sendmail
program.  Eh?  Or at the least use recipients at a local SMTP sink.

>The 100k deliveries/hr that your seeing -- what size messages are you
>seeing?  Is that relatively constant as a product of the number of

Hmm, those were 3 to 6k messages.  I actually didn't see much in the
way of change until the size went WAY up -- 50 to 100K.  Most of
the cost was dealing with file-system meta data -- writing
1K instead of 5K was fairly down in the noise when compared to
the meta-data stuff, though to a lesser extent on the RAM disc.

>msgs/hr times the number of recipients/msg?  Do you think Mailman is
>the limiting factor to your throughput and have you done any
>benchmarking of the MM software?

This was not using Mailman at all.  Based on the experience with the
lists I run, I'd expect mailman to really pound on the machine
in addition to this.  These messages were generated using a
custom program that had a number of processes feeding the queue,
and would watch for the queue to fill up and then throttle.
Especially important with stock QMail configurations where it's
fairly easy to swamp the input queue.  The big-todo patch helps
this a lot (at around 25k messages with the stock, performance
really drops off on ufs-like file-systems).

>Which I definitely don't have access to.

We don't at the moment either, but check back later, we may have some
spare boxes to run some tests.  I also have a stripped down
balls-to-the-wall SMTP server that you could use as a sink.  This
SMTP server on my old laptop (P133 with 40MB RAM and IDE hard drive)
was able to handle over distinct incoming messages per second.
(or around 360K per hour).  I haven't really tested it on a
more modern machine.

Sean
-- 
 A ship in port is safe, but that is not what ships are for.
                 -- Rear Admiral Grace Murray Hopper
Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com>
URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.