[Mailman-Users] Mailman + giant lists + the infinite weight of the cosmos

[Brad Knowles]

At 11:52 AM -0800 2005-02-08, Carl Zwanzig wrote:

>>  	3. Any general advice for handling ridiculously large lists?
>
>  You might want to look at http://www.python.org/doc/faq/library.html#id35,
>  otherwise, I think you're hacking MM to default to the marshal method. The
>  code is there, but as the backup method. (Or really hacking it to use
>  mysql...)

	Using Berkeley db is pretty blindingly fast, and if done right, 
should result in an extremely reliable database.  MySQL uses 
BerkeleyDB as their way of achieving full ACID compliance in their 
high-end MaxSQL database product.  Rock-damn-bloody-solid stuff, if 
done right.  But also easier to screw up.

	If you can replace the marshal method with one based on Berkeley 
DB, I imagine that will probably solve most of the performance 
problems right there.  Assuming you choose the right type of file 
format, that is.  I imagine that a btree is likely to perform better 
in this application than a hash table.

	Alternatively, a Berkeley DB-based member adaptor should not be 
too excessively difficult to code.  However, if such a beast doesn't 
already exist, you might want to first try the LDAP member adaptor.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See <http://www.sage.org/> for more info.