On Fri, Aug 08, 2008 at 04:58:07PM -0400, Barry Warsaw wrote:
recursive or a complicated graph. One experiment would be to create a list with, say, 100,000 random foo@example.com addresses and benchmark how much time it takes to unpickle it. I'll try to do that tomorrow on a real computer (not this laptop).
Hi Andrew, any results?
The simple test program I used is below. For a list with 89531 addresses, the config.pck file is 9248317 bytes = 8.9M. Doing ten loads and then ten saves in a row, the average load time is 1.36sec and the average save time is 4.5sec.
This is on a development server here at Matrix, which has two 1.1GHz Intel CPUs and 2Gb of RAM; a respectable machine, but not what you'd currently use for a server. So I think pickle really is pretty fast. Of course, if you had your Mailman installation on a busy mail or database server, all that I/O might kill you, but I think giving up on the mtime caching is not completely unreasonable.
--amk
import time
from Mailman import MailList
L = [] for i in range(10): s = time.time() ml = MailList.MailList('amk-speed-test', lock=1) e = time.time() ml.Unlock() L.append(e-s) print e-s
print 'Average loading time=', sum(L)/len(L)
L = [] ml.Lock() for i in range(10): s = time.time() ml.Save() e = time.time() L.append(e-s) print e-s
ml.Unlock() #print L print 'Average save time=', sum(L)/len(L)