[Mailman-Users] Trouble with LARGE mailing list

[Jon Carnes]

Wow, what a cool problem...  I think that v 2.1 (once it is out of beta) is
for you.  You'll be able to move that database into a PostGresSQL database
or any database of your choice.

I do hope that you are running a SCSI disk subsystem (or better).  That will
help immensely with resource problems.

Here is an obscure thought - what if you actually set aside some memory for
a ram drive - say about 64 Mb and then backed up the directory
~mailman/lists/<listname>/.. to the RAM drive, then moved the physical
mapping of that directory and mapped the RAM drive there.

Your database and its backups/rebuilds would all take place in RAM.  They
would be very fast so you would have less simultaneous accesses to the
database.
Every hour copy the database from RAM back to your HD somewhere.

Alternately, you could make your main list an umbrella list and then pull
folks out of the main list and into a sublist.  Mailing addresses starting
with an A would go to one list, the B's to another, the Z's to another list
and any numeric emails to yet another list.  (you would have to add each
sub-list to the main list, just like a user).

Folks would still subscribe to the main/umbrella list.  They would stay
there till that night, when your cron ran a script to pull them out and
re-subscribe them to their proper sublist.

That way all your subscribes are taking place on a nearly empty list... and
you control when all the heavy lifting occurs (the movement of folks into
the larger sub-lists).

You will have to generate your own listinfo page (not really all that hard -
just write a simple html page that does what you want and stick that in
place of the current Listinfo page - using the web admin).  Your list info
page could ask folks to enter their email address and then you would use
some back-end logic to isolate the first letter and transfer them to the
proper Listinfo page.  From there, they would be okay!

Good Luck.  Keep us in the loop and let us know what your do!

Jon

----- Original Message -----
From: "Juergen Geinitz" <geinitz at novalis-media.de>
To: <mailman-users at python.org>
Sent: Thursday, April 18, 2002 8:18 AM
Subject: [Mailman-Users] Trouble with LARGE mailing list


> Hello,
> I have a mailing list here running for one of our customers
> which is getting extremely large and therefore slow.
> The list is a newsletter list with one post a week - that doesn't
> harm me. But I hav about 500 new subscribes a day which makes up to now
> about 150000 members - and thats my problem:
> the config.db is
> -rw-rw----    1 www-data www-data 15587680 Apr 18 14:13 config.db
>
> and therefore is blowing my main memory. Lokk at this output of "top"
> 31084 www-data  18   0  81M           R    48.7 10.8   0:07
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31262 www-data  20   0  42M           R    48.1  5.5   0:03
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31260 geinitz   13   0 2456           R     0.9  0.3   0:00 top
> 30983 www-data   9   0  73M do_select S     0.3  9.7   0:11
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31102 www-data   9   0  42M do_select S     0.3  5.6   0:04
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31120 www-data  10   0  73M do_select S     0.3  9.7   0:11
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31219 www-data   9   0  42M do_select S     0.3  5.6   0:04
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31240 www-data   9   0  42M do_select S     0.3  5.6   0:04
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
>    11 root       9   0    0 mosix_mem SW    0.1  0.0  10:07 memsorter
> 30682 www-data   9   0  984 do_select S     0.1  0.1   0:05
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31220 www-data   9   0  42M do_select S     0.1  5.6   0:03
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31222 www-data   9   0  42M do_select S     0.1  5.6   0:03
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
> 31242 www-data   9   0  42M do_select S     0.1  5.6   0:03
> /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe
>
> so every subscribe is between 40 an 80 M in core. At the moment the
> system is running an own apache
> for this with a connect limit of 15 which gives 768M main memory used
> plus 200M swap. When I remove
> the apache limit, the system goes into swapping - more subscribe
> requests arrive - less memory
> an about 5 minutes later - BANG
>
> Any Idea?
> Any hint?
>
> Regards
> Juergen
> --
> novalis media
>   Ein Bereich der Vereinigte Verlagsanstalten GmbH -
>     http://www.vva.de
>     http://www.novalis-media.de
> Geschaeftsfuehrer Stefan Meutsch HRB 658 Duesseldorf
>
>
> ------------------------------------------------------
> Mailman-Users mailing list
> Mailman-Users at python.org
> http://mail.python.org/mailman/listinfo/mailman-users
> Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py