[Spambayes] some preliminary timings

[Neil Schemenauer]

Skip Montanaro wrote:
> Such a huge difference between hammiefilter and a raw filter loop suggests I
> may have done something wrong.  Still, perhaps opening the db file for each
> message and all the imports hammiefilter has to do simply kills the
> performance.

Yes.  Try "strace python hammiefilter.py".  I count 848 open() system
calls.  702 of them return ENOENT.  A (relatively) small sample:

stat64("spambayes/shelve", 0xbfffcd9c)  = -1 ENOENT (No such file or directory)
open("spambayes/shelve.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("spambayes/shelvemodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("spambayes/shelve.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("spambayes/shelve.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("shelve", 0xbfffcd9c)            = -1 ENOENT (No such file or directory)
open("shelve.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("shelvemodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("shelve.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("shelve.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/python23.zip/shelve", 0xbfffcd9c) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python23.zip/shelve.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python23.zip/shelvemodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python23.zip/shelve.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python23.zip/shelve.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/python2.3/shelve", 0xbfffcd9c) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.3/shelve.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.3/shelvemodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.3/shelve.py", O_RDONLY|O_LARGEFILE) = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=4739, ...}) = 0
open("/usr/local/lib/python2.3/shelve.pyc", O_RDONLY|O_LARGEFILE) = 6
fstat64(6, {st_mode=S_IFREG|0664, st_size=8662, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000
read(6, ";\362\r\nN\231\35>c\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0s\355"..., 4096) = 4096

Ouch.  Having a longer sys.path makes things worse.

  Neil