Mailman-Developers December 2008

mailman-developers@python.org

3 participants
4 discussions

"Orignal" MySQL Member Adaptor - 1.71
by Kyrian (list) 08 Jan '09

08 Jan '09

All, I finally managed to find some vestige of free time after midnight (hardly the best time to be coding...) recently, and accordingly put out a new version of my MySQL MemberAdaptor. I have no time, and little motivation (no customers wanting to use it at the present time) to test it, so I haven't, but you're welcome to try it out. It should according to the docs I found, provided I've applied them correctly, fix the non-ASCII character encoding problems (I've relied on the assertions about list members to cover that angle, and just checked encoding on the 'user supplied' parameters to various function calls). Now would be a good time for someone to send me a Python language reference book ;-) However since I've changed development environment since the last version, it's possible that various things have gone wrong (certainly the version numbering has gone mad between RCS and CVS/Eclipse), so don't do it in a production environment. I've put it here, as version 1.71, whatever that might actually mean: http://www.orenet.co.uk/opensource/MailmanMysql/ Incidentally, since Fil seems to be taking all the credit for what I started I think he should actually put a proper credit in there for me, linking to my business website at http://www.orenet.co.uk/ rather than the vague passing mentions that are in the docs/code of his version of the system at the present time. With luck I will be able to coax one of my customers into 'sponsoring' further development of the MySQL Adaptor with proper integration into Mailman in a sane way that is agreed by everyone instead of (as seems to be the case from what little of this list I have had time to read) just going off on its own sweet way and reducing the likelyhood of acceptance into the main code base of Mailman, but I am not sure how that will go. If anyone else wants to sponsor my time to code this thing better and further, then just email me. Best of luck with it, whatever happens. More updates in whichever year I manage to find spare/free time. Yours Sincerely, Kev Green.

7 19

Memory pinned in ram, with huge lists
by Jesus Cea 14 Dec '08

14 Dec '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I experienced huge mailman queue processes, with ram usage of 700MB and more for each one of the six queue workers. I have several big mailing lists. One of about 180.000 subscriber, and other of about 70.000. Debugging the issue I found: 1. Each queue worker touching a message will load the entire mailing list database in RAM. So, the RAM used by each worker is the sum of all mailing lists in the system (if all of them have traffic). This is a big issue if you have huge mailing lists. 2. The list data is keep in memory using a cache managed via weak references. But the cache is never evicted, so there is a hard reference out there, somewhere. 3. I found a memory reference cycle between a Mailing list and its OldStyleMemberships component, linked via "self._memberadaptor". This cycle keeps the mailing list alive and, so, the cache never evicted the data. I changed the OldStyleMemberships constructor to: """ class OldStyleMemberships(MemberAdaptor.MemberAdaptor): def __init__(self, mlist): import weakref self.__mlist = weakref.proxy(mlist) """ to keep only a weak reference to the mailing list, breaking the cycle. Now, when a worker is done with a mailing list, the cache is correctly evicted. Since python doesn't give back memory to system, the consequence of this change is: 1. Now, memory used by each worker is proportional to the size of the biggest mailing list, instead of the sum of all mailing list sizes. Not perfect, but a huge improvement is you have some big lists. 2. Now, since cache in evicted frequently, mailing list data must be reloaded every time. This is a performance hit, but my mailing list are huge but with little traffic (maybe a couple of mails per week), so this is a non issue for me. I would suggest to separate the subscriber info from the rest of the mailing metadata, since most workers doesn't need the subscriber data in RAM to do its work. So, instead of 6 processes eating RAM, only of them (the outgoing worker) will use significant memory. In fact, mailing list subscribers could be splitted in several files, to avoid to load the entire membership at once. Let say, use 256 files and putting each subscriber in a file according to the last significant byte of its MD5 hash, for instance. Studying the code, it seems easy to migrate membership to a separate persistence system (let say, ZODB, Durus) or use a backend like sqlite. Any plan for that?. Any interest in patches?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSUH+RJlgi5GaxT1NAQJq7AQAm5tbsJQL2zqLFJlHLvha9RUnguzEYKRW tS2LkHkZbmcFFXrYLswfl9Qn20x9FPA9iWN/j9hwh8YK3j7o0sdwS2Yll/44A8NX 4OtfYeOto4aIbYd8VWYa5RPe7ebSYwypkEvbH/FJRt8nDIEvLkr0t9iB7tQ42MsN z+ssg6D6DF4= =yOKL -----END PGP SIGNATURE-----

3 3

Duplicate Prevention
by Dan Mahoney, System Admin 06 Dec '08

06 Dec '08

Hello all, We've recently taken on mailman to handle many large, popular lists at my day job, and one strangely-missing feature is the inability to avoid duplicates when someone cc's a list's old alias and new alias (we also moved the lists to a subdomain, out from under our primary). It's suggested in the FAQs that this could be done using procmail -- but I'd think it (could/should) be done in mailman itself. The list-id header is ideal for this, and it SEEMS to be not more than a dozen lines of code to do the same sort of per-list "lockfile" that the procmail recipe does (plus add a variable to make it tunable in either the config file, or the web-ui). Presumably lockfiles could expire after 7 days. Locks could also be an entry in a .pck database. The question is (and the reason I ask this on a developer list rather than -users) -- is there anyone here who could provide a patch to add this functionality, or who is interested in taking on such a task for some sort of compensation/donation? Please contact me off (or on) list if anyone's available. Another great feature would be to have mailman "strip" multiple cc recipients. I.e. if a message is sent: to: list cc: list-alias(*), another-list-alias(*) -or- to: person cc: list, list-alias(*) To have these (*) stripped (and prevent the need for this). But that's more work, and right now the duplicates are a major regression from what we had before. -Dan -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---------------------------

2 1

Re: [Mailman-Developers] [Mailman-checkins] [Branch ~mailman-administrivia/mailman-administrivia/admin] Rev 38: Replaced an obsolete SF link to NEWS with the current LP link.
by Barry Warsaw 05 Dec '08

05 Dec '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 4, 2008, at 5:44 PM, noreply(a)launchpad.net wrote: > ------------------------------------------------------------ > revno: 38 > committer: Mark Sapiro <mark(a)msapiro.net> > branch nick: admin > timestamp: Thu 2008-12-04 14:41:29 -0800 > message: > Replaced an obsolete SF link to NEWS with the current LP link. > modified: > www/features.ht > www/features.html > www/newsite/features/_index.html Pushed. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkk4aiEACgkQ2YZpQepbvXFNkgCdFmQ13oI2X4TcYZ16BMdtfdYg Z5cAoJCR5gMsE3u+72qab+HExZUUHtGd =7rlw -----END PGP SIGNATURE-----

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

Mailman-Developers December 2008