[Mailman-Users] Mailman + giant lists + the infinite weight of the cosmos
brad at stop.mail-abuse.org
Tue Feb 8 21:12:42 CET 2005
At 10:40 AM -0800 2005-02-08, Cabel Sasser wrote:
> I'll get right to the point: one of our Mailman lists currently has
> 121,047 members. It uses full personalization and auto bounce processing.
> I'm sure you know where this is headed... :)
Cool! That may be the largest Mailman-hosted currently operating
single mailing list, at least the largest we've heard of so far. I
would imagine that lists.apple.com probably does more traffic on a
daily basis, but so far as I know, it doesn't have any individual
lists that are anywhere near that large.
And from the company that created Audion, Unison, CandyBar,
Stattoo, the amazingly funny "USA vs. Japan Food" page, and so much
more. Very, very cool.
> My problem is not with the mailing, but seemingly rather with the user
Once you get to this kind of size, the web administration tools
start to fail. To make things work, I imagine you're going to have
to go to command-line administration techniques. Most of what you
might want to do should be able to be done directly from the
command-line using existing scripts and programs, but you may need a
few additional tools to help round out the mix.
I recall hearing not too long ago that the folks who run the
Mailman-hosted mailing lists for FreeBSD.org have gotten to the point
where they don't even bother with the web interface any more.
However, while they have a very large mailing list server (by Mailman
standards), their largest list is probably around 15,000 subscribers.
They just have a much higher traffic load, sometimes a hundred
messages or more on lists with tens of thousands of subscribers.
Your situation is likely to share some problems with theirs, but
you might also have some unique issues.
> 1. Should I split the list into a series of smaller lists? What's
> the best way to do that easily?
This is the first that I have personally heard of a single list
being this large. I would imagine that splitting things out into
multiple sub-lists with a parent umbrella list, would be a pretty big
help. It shouldn't be too hard to use the command-line tool
list_members to get a list of your subscribers to the list, and then
to split that up unto multiple smaller chunks.
It will take a bit more work to copy out all their preferences
settings so that you can replicate those on the new sub-list.
> 2. Is there any way to "optimize" this database, other than throwing
> more memory at the machine, etc.?
Optimizing Python pickles? No, not that I know of.
> 3. Any general advice for handling ridiculously large lists?
You've already done a lot of homework, but see also FAQ 1.24.
It's not going to add that much to your knowledge base, but you might
as well round out the list.
Disk and memory are going to be your biggest constraints. If you
can set up a RAID 1+0 filesystem (preferably using an XServe RAID
array), plus throw as much memory at the machine as you can, that
will get you most of the performance gain you're likely to see in
terms of hardware tuning.
> I've seen in the FAQ (1.15) that The Guardian ran a list with
> 147,000 subscribers, and I'm wondering how it was done! I've read
> performance tuning (4.11) but that seems to focus on the mail
> _delivery_, which is generally working just fine, not the user
> management -- think bursting Python pickles.
Most of the "large list" issues that we have dealt with so far
have been with relation to servers with more lists, each of which is
moderately large, but the machine has a high level of incoming
traffic to those lists. Do the math, and you find that they are
doing very high delivery volumes, relatively speaking.
The Web administration issues tend to be less common, and
therefore they've gotten less attention.
Brad Knowles, <brad at stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.
More information about the Mailman-Users