Re: [Mailman-Developers] [Mailman-Users] Subscribers suddenly "disappear"

I tried the "sleep" approach as well. But then I thought that it was not viable. I am setting a relatively high number of mailing list that will receive asynchronous "subscribe" requests via web and/or shell API.
It would be simply not possible to prevent bug this from happening as the chance of two "subscribe" being processsed almost simultaneously are not that small...
I also vote for some kind of timing/lock issue.
Max
Barry Finkel wrote:

Max Lanfranconi wrote:
Based on my running of the test script (see <http://mail.python.org/pipermail/mailman-users/2008-August/062885.html>, with
LIST_LOCK_DEBUGGING = True
I think I see the problem. It is related to qrunner list caching and the fact the there is insufficient precision in the list instance's __timestamp
The scenario is the following
add_member saves the list with the first member.
VirginRunner gets there first, instantiates and caches the list. It then locks the list, processes the welcome and saves and unlocks the list.
add_member gets the lock, adds the second member and saves the list.
Virgin runner gets the second welcome. The list is cached, so it uses the cached instance. It then locks the list which ultimately calls MailList.__load() to refresh the list data, but __load() does
mtime = os.path.getmtime(dbfile) if mtime <= self.__timestamp: # File is not newer return None, None
The problem is os.path.getmtime() returns a time in seconds so if we are still in the same second as step 2), we don't refresh the list.
Thank you very much Max for providing a script to reproduce the problem.
I'm not sure of the best solution. I'm leaning towards dropping the __timestamp test or perhaps replacing it with a file size test, but that too may be unreliable.
Other thoughts?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark,
I am more than willing to try any workaround as this bug is currently a showstopper for my project. I am not very versed in mailman internals, so I need some guidance here.
Do you recommend I try getting rid of the __timestamp test and give it a try ?
If that is you suggestion, may I ask what was the original purpose of that test ?
Thanks again for your help in solving this.
Regards,
Max
Mark Sapiro wrote:
--
<http://jcp.org> * Max Lanfranconi * JCP Program Management Office Marketing Manager
Phone +1 408 404 6893 Mobile +1 408 505 1020 Email max@jcp.org

Massimo Lanfranconi wrote:
Yes.
If that is you suggestion, may I ask what was the original purpose of that test ?
Lots of Mailman processes will instantiate a list unlocked for some preliminary checks and then later, lock the list to do some update. The purpose of the timestamp test was to avoid rereading the list data from the config.pck file if the already loaded data is still current.
What I suggest for the moment is to just delete lines 594-603 of Mailman/MailList.py which includes the following 4 lines and the preceding comment.
This will result in some additional I/O in some cases, but it will avoid this problem of sometimes not loading the list data when it has actually changed.
Please do that and restart Mailman and I'm sure the bug will be fixed (but don't take my word for it).
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Sapiro wrote: | What I suggest for the moment is to just delete lines 594-603 of | Mailman/MailList.py which includes the following 4 lines and the | preceding comment. | |>> mtime = os.path.getmtime(dbfile) |>> if mtime <= self.__timestamp: |>> # File is not newer |>> return None, None Ooops!. That won't quite do it. you can't just remove the ~ mtime = os.path.getmtime(dbfile) line as mtime is referenced later and will be undefined. A patch is attached which has been applied and tested. I ran the test script twice with this patch installed with no lost subscribers. It isn't the final patch which should remove __timestamp completely and won't remove all the comment, but it will do as a workaround. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFImSKDVVuXXpU7hpMRArrKAJ9K2fA8MzI7Iarp0WfnNClwtwVbdACfaxCO Xsx7ru3fK2/v5ZRw0YHYU9s= =BdZw -----END PGP SIGNATURE----- --- Mailman/MailList.py 2008-06-30 09:29:46.000000000 -0700 +++ Mailman/MailList.py 2008-08-05 20:43:19.000000000 -0700 @@ -591,16 +591,7 @@ else: assert 0, 'Bad database file name' try: - # Check the mod time of the file first. If it matches our - # timestamp, then the state hasn't change since the last time we - # loaded it. Otherwise open the file for loading, below. If the - # file doesn't exist, we'll get an EnvironmentError with errno set - # to ENOENT (EnvironmentError is the base class of IOError and - # OSError). mtime = os.path.getmtime(dbfile) - if mtime <= self.__timestamp: - # File is not newer - return None, None fp = open(dbfile) except EnvironmentError, e: if e.errno <> errno.ENOENT: raise

Mark Sapiro writes:
Shouldn't "mtime < self.__timestamp" do the right thing (much more often)? You're still vulnerable to "date -s", adjtime, and friends, though, and of course you'll have some undesirable cache misses at times when it would be nice if you didn't.
A better way would be to add a serial number.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 9:08 PM, Stephen J. Turnbull wrote:
Probably so, but is the optimization still worth it?
A better way would be to add a serial number.
You'd probably want to store the serial number in the file name so you
wouldn't have to load the pickle just to get the current serial. It's
similar to what happens with the queue files. But again, I'm not sure
it's worth it.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkiY+ygACgkQ2YZpQepbvXHdBwCfZRO73w4KiA0FMg6eU3yDU8YN Y7AAoLE5760wZWw536oMj1zyHMNi8h4a =w1PV -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Stephen J. Turnbull wrote:
I didn't reply sooner because when I first saw this, I didn't read it carefully, and I didn't "get" what Stephen was saying. Then I had to think about it.
I have concluded that barring resetting of clocks backward, "mtime < self.__timestamp" is equivalent to "False". Whenever a config.pck is written or read in a process, __timestamp is set to the current mod time of the config.pck. Thus, the value of self.__timestamp for a cached list object is always <= mtime.
Always, I think.
A better way would be to add a serial number.
As has already been mentioned, it does no good to put a serial number in the list object since you'd have to read the config.pck to get the serial number. It would have to be encoded in the file name or stored in a separate database. Storing it separately adds the complication of an additional file plus making sure the additional file is sync'd with config.pck. Encoding it in the file name seems complex as well.
I lean towards the simpler approach of just reading the config.pck every time.
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32)
iD8DBQFIrgfnVVuXXpU7hpMRApeVAKDAv5aw4Mc8x6P9KevoWIBYIjaqFgCcDlcq FpC83W+m1C8Vjlfql27QrtA= =/joM -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Sapiro wrote:
I have thought about this some more and have come up with the attached patch which I have tested with Max's script and propose for Mailman 2.2. The patch changes the '<=' test to '<' as Stephen suggests, but then so it won't effectively disable ever not reloading the config.pck, it changes the setting of __timestamp following a successful read from the mod time of the config.pck to the current time truncated to an int. Thus, once we've read a config.pck that's more than a second old, a subsequent load of the same object will skip rereading the config.pck if it wasn't updated in the interim. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFIryLmVVuXXpU7hpMRApoCAJ9hPl/yxi8n9ZAT04s2+TCG6/rvbwCg1NJ6 HlgyY/PB3s6l7B4diGO0zb8= =5V+H -----END PGP SIGNATURE----- === modified file 'Mailman/MailList.py' --- Mailman/MailList.py 2008-08-21 21:35:20 +0000 +++ Mailman/MailList.py 2008-08-22 18:18:38 +0000 @@ -597,8 +597,11 @@ # file doesn't exist, we'll get an EnvironmentError with errno set # to ENOENT (EnvironmentError is the base class of IOError and # OSError). + # We test strictly less than here because the resolution is whole + # seconds and we have seen cases of the file being updated by + # another process in the same second. mtime = os.path.getmtime(dbfile) - if mtime <= self.__timestamp: + if mtime < self.__timestamp: # File is not newer return None, None fp = open(dbfile) @@ -616,8 +619,9 @@ return None, e finally: fp.close() - # Update timestamp - self.__timestamp = mtime + # Update the timestamp. We use current time here rather than mtime + # so the test above might succeed the next time. + self.__timestamp = int(time.time()) return dict, None def Load(self, check_version=True):

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 21, 2008, at 8:27 PM, Mark Sapiro wrote:
+1
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkizw64ACgkQ2YZpQepbvXFMuwCfRKzhFhEnTIaQLl1YgqeuTp7w O3QAn3fRWR8OytLtI3F4JXjLnmhmgLsY =XpWr -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 7:34 PM, Mark Sapiro wrote:
I wonder if the list cache is still worth it? I've run into trouble
with it in the recent past and I suspect that whatever benefits we got
from it in ancient times, may not be so relevant today. My first
inclination would be to ditch the cache, but that may not completely
solve this issue (ah, for the backing of a real database!).
Ultimately the timestamp check probably has to go. If we try to load
the list, we should just load it and not try to be so fancy to avoid
reading from disk. Yes, it means more I/O but reliability is more
important, IMO. I'm not convinced a size check is worth it.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkiY+qMACgkQ2YZpQepbvXHUbgCfeWIwkAjv/9wYpJa9vQ16dwCP jYIAn0wVNl3JXm+9R2LJ5AUa46FwhNAN =P5Yo -----END PGP SIGNATURE-----

On Tue, Aug 05, 2008 at 09:13:07PM -0400, Barry Warsaw wrote:
I expect cPickle or even pickle are pretty fast, and the config.pck is a fairly straightforward data structure, isn't it? Not deeply recursive or a complicated graph. One experiment would be to create a list with, say, 100,000 random foo@example.com addresses and benchmark how much time it takes to unpickle it. I'll try to do that tomorrow on a real computer (not this laptop).
--amk

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 9:50 PM, A.M. Kuchling wrote:
Hi Andrew, any results?
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkics18ACgkQ2YZpQepbvXFX2gCdHOghfr9eb6xRZ1/delxPnw+2 t3wAnjgqAQhtiUUGYQSqQF0VT+ATyt+V =tyGZ -----END PGP SIGNATURE-----

On Fri, Aug 08, 2008 at 04:58:07PM -0400, Barry Warsaw wrote:
The simple test program I used is below. For a list with 89531 addresses, the config.pck file is 9248317 bytes = 8.9M. Doing ten loads and then ten saves in a row, the average load time is 1.36sec and the average save time is 4.5sec.
This is on a development server here at Matrix, which has two 1.1GHz Intel CPUs and 2Gb of RAM; a respectable machine, but not what you'd currently use for a server. So I think pickle really is pretty fast. Of course, if you had your Mailman installation on a busy mail or database server, all that I/O might kill you, but I think giving up on the mtime caching is not completely unreasonable.
--amk
import time
from Mailman import MailList
L = [] for i in range(10): s = time.time() ml = MailList.MailList('amk-speed-test', lock=1) e = time.time() ml.Unlock() L.append(e-s) print e-s
print 'Average loading time=', sum(L)/len(L)
L = [] ml.Lock() for i in range(10): s = time.time() ml.Save() e = time.time() L.append(e-s) print e-s
ml.Unlock() #print L print 'Average save time=', sum(L)/len(L)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 8, 2008, at 5:41 PM, A.M. Kuchling wrote:
Thanks for the feedback Andrew.
I don't know if it's worth changing for 2.1; I think it's a rare
problem and the workarounds are now all in the archive. It's probably
worth changing for 2.2 (and is of course moot for 3.0), but it's still
probably not worth making it configurable. For 2.2, let's do the
right thing and if we can make it fast in the meantime, great!
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iQCVAwUBSJy+inEjvBPtnXfVAQJiYgP+LPZABA78c604XzB39lBpkF/mfu/EEPE8 7aojEnqdaiorAA4PLQppAPNuF8oz9rQU87q/WVSJLdUo87HieTqddOy/CFwPJfYS ixdJmc/k2TdnoLSFykcHhriJUQtfB1TNCodPG2BbFer7b4tEjyIRK11W6bC4gIM5 HfHYl1GLuck= =ARJn -----END PGP SIGNATURE-----

Max Lanfranconi wrote:
Based on my running of the test script (see <http://mail.python.org/pipermail/mailman-users/2008-August/062885.html>, with
LIST_LOCK_DEBUGGING = True
I think I see the problem. It is related to qrunner list caching and the fact the there is insufficient precision in the list instance's __timestamp
The scenario is the following
add_member saves the list with the first member.
VirginRunner gets there first, instantiates and caches the list. It then locks the list, processes the welcome and saves and unlocks the list.
add_member gets the lock, adds the second member and saves the list.
Virgin runner gets the second welcome. The list is cached, so it uses the cached instance. It then locks the list which ultimately calls MailList.__load() to refresh the list data, but __load() does
mtime = os.path.getmtime(dbfile) if mtime <= self.__timestamp: # File is not newer return None, None
The problem is os.path.getmtime() returns a time in seconds so if we are still in the same second as step 2), we don't refresh the list.
Thank you very much Max for providing a script to reproduce the problem.
I'm not sure of the best solution. I'm leaning towards dropping the __timestamp test or perhaps replacing it with a file size test, but that too may be unreliable.
Other thoughts?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark,
I am more than willing to try any workaround as this bug is currently a showstopper for my project. I am not very versed in mailman internals, so I need some guidance here.
Do you recommend I try getting rid of the __timestamp test and give it a try ?
If that is you suggestion, may I ask what was the original purpose of that test ?
Thanks again for your help in solving this.
Regards,
Max
Mark Sapiro wrote:
--
<http://jcp.org> * Max Lanfranconi * JCP Program Management Office Marketing Manager
Phone +1 408 404 6893 Mobile +1 408 505 1020 Email max@jcp.org

Massimo Lanfranconi wrote:
Yes.
If that is you suggestion, may I ask what was the original purpose of that test ?
Lots of Mailman processes will instantiate a list unlocked for some preliminary checks and then later, lock the list to do some update. The purpose of the timestamp test was to avoid rereading the list data from the config.pck file if the already loaded data is still current.
What I suggest for the moment is to just delete lines 594-603 of Mailman/MailList.py which includes the following 4 lines and the preceding comment.
This will result in some additional I/O in some cases, but it will avoid this problem of sometimes not loading the list data when it has actually changed.
Please do that and restart Mailman and I'm sure the bug will be fixed (but don't take my word for it).
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Sapiro wrote: | What I suggest for the moment is to just delete lines 594-603 of | Mailman/MailList.py which includes the following 4 lines and the | preceding comment. | |>> mtime = os.path.getmtime(dbfile) |>> if mtime <= self.__timestamp: |>> # File is not newer |>> return None, None Ooops!. That won't quite do it. you can't just remove the ~ mtime = os.path.getmtime(dbfile) line as mtime is referenced later and will be undefined. A patch is attached which has been applied and tested. I ran the test script twice with this patch installed with no lost subscribers. It isn't the final patch which should remove __timestamp completely and won't remove all the comment, but it will do as a workaround. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFImSKDVVuXXpU7hpMRArrKAJ9K2fA8MzI7Iarp0WfnNClwtwVbdACfaxCO Xsx7ru3fK2/v5ZRw0YHYU9s= =BdZw -----END PGP SIGNATURE----- --- Mailman/MailList.py 2008-06-30 09:29:46.000000000 -0700 +++ Mailman/MailList.py 2008-08-05 20:43:19.000000000 -0700 @@ -591,16 +591,7 @@ else: assert 0, 'Bad database file name' try: - # Check the mod time of the file first. If it matches our - # timestamp, then the state hasn't change since the last time we - # loaded it. Otherwise open the file for loading, below. If the - # file doesn't exist, we'll get an EnvironmentError with errno set - # to ENOENT (EnvironmentError is the base class of IOError and - # OSError). mtime = os.path.getmtime(dbfile) - if mtime <= self.__timestamp: - # File is not newer - return None, None fp = open(dbfile) except EnvironmentError, e: if e.errno <> errno.ENOENT: raise

Mark Sapiro writes:
Shouldn't "mtime < self.__timestamp" do the right thing (much more often)? You're still vulnerable to "date -s", adjtime, and friends, though, and of course you'll have some undesirable cache misses at times when it would be nice if you didn't.
A better way would be to add a serial number.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 9:08 PM, Stephen J. Turnbull wrote:
Probably so, but is the optimization still worth it?
A better way would be to add a serial number.
You'd probably want to store the serial number in the file name so you
wouldn't have to load the pickle just to get the current serial. It's
similar to what happens with the queue files. But again, I'm not sure
it's worth it.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkiY+ygACgkQ2YZpQepbvXHdBwCfZRO73w4KiA0FMg6eU3yDU8YN Y7AAoLE5760wZWw536oMj1zyHMNi8h4a =w1PV -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Stephen J. Turnbull wrote:
I didn't reply sooner because when I first saw this, I didn't read it carefully, and I didn't "get" what Stephen was saying. Then I had to think about it.
I have concluded that barring resetting of clocks backward, "mtime < self.__timestamp" is equivalent to "False". Whenever a config.pck is written or read in a process, __timestamp is set to the current mod time of the config.pck. Thus, the value of self.__timestamp for a cached list object is always <= mtime.
Always, I think.
A better way would be to add a serial number.
As has already been mentioned, it does no good to put a serial number in the list object since you'd have to read the config.pck to get the serial number. It would have to be encoded in the file name or stored in a separate database. Storing it separately adds the complication of an additional file plus making sure the additional file is sync'd with config.pck. Encoding it in the file name seems complex as well.
I lean towards the simpler approach of just reading the config.pck every time.
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32)
iD8DBQFIrgfnVVuXXpU7hpMRApeVAKDAv5aw4Mc8x6P9KevoWIBYIjaqFgCcDlcq FpC83W+m1C8Vjlfql27QrtA= =/joM -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Sapiro wrote:
I have thought about this some more and have come up with the attached patch which I have tested with Max's script and propose for Mailman 2.2. The patch changes the '<=' test to '<' as Stephen suggests, but then so it won't effectively disable ever not reloading the config.pck, it changes the setting of __timestamp following a successful read from the mod time of the config.pck to the current time truncated to an int. Thus, once we've read a config.pck that's more than a second old, a subsequent load of the same object will skip rereading the config.pck if it wasn't updated in the interim. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFIryLmVVuXXpU7hpMRApoCAJ9hPl/yxi8n9ZAT04s2+TCG6/rvbwCg1NJ6 HlgyY/PB3s6l7B4diGO0zb8= =5V+H -----END PGP SIGNATURE----- === modified file 'Mailman/MailList.py' --- Mailman/MailList.py 2008-08-21 21:35:20 +0000 +++ Mailman/MailList.py 2008-08-22 18:18:38 +0000 @@ -597,8 +597,11 @@ # file doesn't exist, we'll get an EnvironmentError with errno set # to ENOENT (EnvironmentError is the base class of IOError and # OSError). + # We test strictly less than here because the resolution is whole + # seconds and we have seen cases of the file being updated by + # another process in the same second. mtime = os.path.getmtime(dbfile) - if mtime <= self.__timestamp: + if mtime < self.__timestamp: # File is not newer return None, None fp = open(dbfile) @@ -616,8 +619,9 @@ return None, e finally: fp.close() - # Update timestamp - self.__timestamp = mtime + # Update the timestamp. We use current time here rather than mtime + # so the test above might succeed the next time. + self.__timestamp = int(time.time()) return dict, None def Load(self, check_version=True):

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 21, 2008, at 8:27 PM, Mark Sapiro wrote:
+1
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkizw64ACgkQ2YZpQepbvXFMuwCfRKzhFhEnTIaQLl1YgqeuTp7w O3QAn3fRWR8OytLtI3F4JXjLnmhmgLsY =XpWr -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 7:34 PM, Mark Sapiro wrote:
I wonder if the list cache is still worth it? I've run into trouble
with it in the recent past and I suspect that whatever benefits we got
from it in ancient times, may not be so relevant today. My first
inclination would be to ditch the cache, but that may not completely
solve this issue (ah, for the backing of a real database!).
Ultimately the timestamp check probably has to go. If we try to load
the list, we should just load it and not try to be so fancy to avoid
reading from disk. Yes, it means more I/O but reliability is more
important, IMO. I'm not convinced a size check is worth it.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkiY+qMACgkQ2YZpQepbvXHUbgCfeWIwkAjv/9wYpJa9vQ16dwCP jYIAn0wVNl3JXm+9R2LJ5AUa46FwhNAN =P5Yo -----END PGP SIGNATURE-----

On Tue, Aug 05, 2008 at 09:13:07PM -0400, Barry Warsaw wrote:
I expect cPickle or even pickle are pretty fast, and the config.pck is a fairly straightforward data structure, isn't it? Not deeply recursive or a complicated graph. One experiment would be to create a list with, say, 100,000 random foo@example.com addresses and benchmark how much time it takes to unpickle it. I'll try to do that tomorrow on a real computer (not this laptop).
--amk

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 5, 2008, at 9:50 PM, A.M. Kuchling wrote:
Hi Andrew, any results?
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkics18ACgkQ2YZpQepbvXFX2gCdHOghfr9eb6xRZ1/delxPnw+2 t3wAnjgqAQhtiUUGYQSqQF0VT+ATyt+V =tyGZ -----END PGP SIGNATURE-----

On Fri, Aug 08, 2008 at 04:58:07PM -0400, Barry Warsaw wrote:
The simple test program I used is below. For a list with 89531 addresses, the config.pck file is 9248317 bytes = 8.9M. Doing ten loads and then ten saves in a row, the average load time is 1.36sec and the average save time is 4.5sec.
This is on a development server here at Matrix, which has two 1.1GHz Intel CPUs and 2Gb of RAM; a respectable machine, but not what you'd currently use for a server. So I think pickle really is pretty fast. Of course, if you had your Mailman installation on a busy mail or database server, all that I/O might kill you, but I think giving up on the mtime caching is not completely unreasonable.
--amk
import time
from Mailman import MailList
L = [] for i in range(10): s = time.time() ml = MailList.MailList('amk-speed-test', lock=1) e = time.time() ml.Unlock() L.append(e-s) print e-s
print 'Average loading time=', sum(L)/len(L)
L = [] ml.Lock() for i in range(10): s = time.time() ml.Save() e = time.time() L.append(e-s) print e-s
ml.Unlock() #print L print 'Average save time=', sum(L)/len(L)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Aug 8, 2008, at 5:41 PM, A.M. Kuchling wrote:
Thanks for the feedback Andrew.
I don't know if it's worth changing for 2.1; I think it's a rare
problem and the workarounds are now all in the archive. It's probably
worth changing for 2.2 (and is of course moot for 3.0), but it's still
probably not worth making it configurable. For 2.2, let's do the
right thing and if we can make it fast in the meantime, great!
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)
iQCVAwUBSJy+inEjvBPtnXfVAQJiYgP+LPZABA78c604XzB39lBpkF/mfu/EEPE8 7aojEnqdaiorAA4PLQppAPNuF8oz9rQU87q/WVSJLdUo87HieTqddOy/CFwPJfYS ixdJmc/k2TdnoLSFykcHhriJUQtfB1TNCodPG2BbFer7b4tEjyIRK11W6bC4gIM5 HfHYl1GLuck= =ARJn -----END PGP SIGNATURE-----
participants (7)
-
A.M. Kuchling
-
Barry Warsaw
-
Barry Warsaw
-
Mark Sapiro
-
Massimo Lanfranconi
-
Max Lanfranconi
-
Stephen J. Turnbull