[Mailman-Users] Mailman + giant lists + the infinite weight of the cosmos

[Brad Knowles]

At 10:40 AM -0800 2005-02-08, Cabel Sasser wrote:

>  I'll get right to the point: one of our Mailman lists currently has
>  121,047 members. It uses full personalization and auto bounce processing.
>  I'm sure you know where this is headed... :)

	Cool!  That may be the largest Mailman-hosted currently operating 
single mailing list, at least the largest we've heard of so far.  I 
would imagine that lists.apple.com probably does more traffic on a 
daily basis, but so far as I know, it doesn't have any individual 
lists that are anywhere near that large.

	And from the company that created Audion, Unison, CandyBar, 
Stattoo, the amazingly funny "USA vs. Japan Food" page, and so much 
more.  Very, very cool.

>  My problem is not with the mailing, but seemingly rather with the user
>  management.

	Once you get to this kind of size, the web administration tools 
start to fail.  To make things work, I imagine you're going to have 
to go to command-line administration techniques.  Most of what you 
might want to do should be able to be done directly from the 
command-line using existing scripts and programs, but you may need a 
few additional tools to help round out the mix.

	I recall hearing not too long ago that the folks who run the 
Mailman-hosted mailing lists for FreeBSD.org have gotten to the point 
where they don't even bother with the web interface any more. 
However, while they have a very large mailing list server (by Mailman 
standards), their largest list is probably around 15,000 subscribers. 
They just have a much higher traffic load, sometimes a hundred 
messages or more on lists with tens of thousands of subscribers.

	Your situation is likely to share some problems with theirs, but 
you might also have some unique issues.

>  	1. Should I split the list into a series of smaller lists? What's
>  the best way to do that easily?

	This is the first that I have personally heard of a single list 
being this large.  I would imagine that splitting things out into 
multiple sub-lists with a parent umbrella list, would be a pretty big 
help.  It shouldn't be too hard to use the command-line tool 
list_members to get a list of your subscribers to the list, and then 
to split that up unto multiple smaller chunks.

	It will take a bit more work to copy out all their preferences 
settings so that you can replicate those on the new sub-list.

>  	2. Is there any way to "optimize" this database, other than throwing
>  more memory at the machine, etc.?

	Optimizing Python pickles?  No, not that I know of.

>  	3. Any general advice for handling ridiculously large lists?

	You've already done a lot of homework, but see also FAQ 1.24. 
It's not going to add that much to your knowledge base, but you might 
as well round out the list.

	Disk and memory are going to be your biggest constraints.  If you 
can set up a RAID 1+0 filesystem (preferably using an XServe RAID 
array), plus throw as much memory at the machine as you can, that 
will get you most of the performance gain you're likely to see in 
terms of hardware tuning.

>  I've seen in the FAQ (1.15) that The Guardian ran a list with
>  147,000 subscribers, and I'm wondering how it was done! I've read
>  performance tuning (4.11) but that seems to focus on the mail
>  _delivery_, which is generally working just fine, not the user
>  management -- think bursting Python pickles.

	Most of the "large list" issues that we have dealt with so far 
have been with relation to servers with more lists, each of which is 
moderately large, but the machine has a high level of incoming 
traffic to those lists.  Do the math, and you find that they are 
doing very high delivery volumes, relatively speaking.

	The Web administration issues tend to be less common, and 
therefore they've gotten less attention.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See <http://www.sage.org/> for more info.