[Mailman-Developers] Huge lists

[bwarsaw@python.org]

>>>>> "SR" == Sean Reifschneider <jafo-mailman-developers@tummy.com> writes:

    SR> I guess you'd better define "relatively common hardware".
    SR> What are we talking about here?  Some benchmarking I did of
    SR> real-world recpieints has shown that a dual PII 450 with a GB
    SR> of RAM and the mail spool set up on a 500MB RAM disc, and
    SR> upstream connectivity of dual DS3s, was able to handle around
    SR> 100k deliveries an hour.  This was using qmail with a remote
    SR> concurrency of 250 and a bunch of little tricks.

Very interesting, and good numbers to know.  I'm here with a single
PIII 650, 256MB and no RAM disk.  My currently limiting factor is my
IP connection, but that should soon improve considerably.

The 100k deliveries/hr that your seeing -- what size messages are you
seeing?  Is that relatively constant as a product of the number of
msgs/hr times the number of recipients/msg?  Do you think Mailman is
the limiting factor to your throughput and have you done any
benchmarking of the MM software?

Right now, I'm primarily interested in seeing how MM handles very
large recipient lists.  Later I'd like to do some testing to see how
much we can boost throughput, but I'd check with addr-volunteers
before doing that kind of testing.

    SR> So, unless I missed something pretty serious in the
    SR> configuration of this environment, "relatively common
    SR> hardware" should include at least a dozen boxes.

Which I definitely don't have access to.

    SR> Again, this was a real-world list of around 400k addresses of
    SR> users who had signed up to receive a particular announcement.
    SR> They have since moved to a cluster of around 10 machines
    SR> running some custom software and are getting over 1M per hour
    SR> with that setup.

    >> email lists of varying sizes.  I'd like to create lists of 1k,
    >> 10k, 100k, 250k, 500k, and 1M recipients.  Do you think between
    >> us we can gather 1M fake recipients for a test list?

    SR> I could probably donate 5% to 10% of that amount across a few
    SR> upstream lines.

That would be great.  I'm not quite ready to start this kind of
testing, but soon.

Thanks,
-Barry