[Python-3000] Nonlinearity in dbm.ndbm?

skip at pobox.com skip at pobox.com
Sun Sep 7 01:25:48 CEST 2008


    >> Now to dig into the abysmal sqlite performance.

    Josiah> The version I just posted to the tracker reads/writes about 30k
    Josiah> entries/second.  You may want to look at the differences (looks
    Josiah> to be due to your lack of a primary key/index).

Thanks.  The real speedup was to avoid using cursors.  Here's the
progression:

* My original (no indexes, keys and values are text, using cursors w/
  commit and explicit close, delete+insert to assign key):

    10 loops, best of 3: 51.4 msec per loop
    10 loops, best of 3: 505 msec per loop

* As above, but with a primary key:

    10 loops, best of 3: 52.5 msec per loop
    10 loops, best of 3: 507 msec per loop

* As above, but keys and values are blobs:

    10 loops, best of 3: 50.4 msec per loop
    10 loops, best of 3: 529 msec per loop

* As above, but get rid of del self[key] in __setitem__ and use the replace
  statement instead of insert:

    10 loops, best of 3: 25.4 msec per loop
    10 loops, best of 3: 263 msec per loop

* Remove try/finally with explicit close() calls (Gerhard says he never
  closes cursors.):

    10 loops, best of 3: 23.2 msec per loop
    10 loops, best of 3: 270 msec per loop

* Get rid of cursors, calling the connection's execute method instead:

    1000 loops, best of 3: 198 usec per loop
    100 loops, best of 3: 2.26 msec per loop

Hmmm...  Should cursors be used?  What benefit are they?  Without them is
the sqlite code thread-safe?

Skip


More information about the Python-3000 mailing list