anydbm is slow

Amir Michail amichail at cse.unsw.edu.au
Tue Jan 1 19:19:24 EST 2002


On Wed, 2 Jan 2002 11:10, Skip Montanaro wrote:
>     Amir> I'm trying to write out a large database file using anydbm.
>     Amir> However, it's quite slow.  Writing out a 200 meg file can take
>     Amir> hours.
>
>     Amir> In fact, I do most of the work using an internal map and write
> out Amir> that map at the end since I thought this would be faster.  (I
> have Amir> lots of memory.) However, it's still very slow.
>
>     Amir> Any hints?
>
> What underlying db module does anydbm use?
>
>     >>> import anydbm
>     >>> anydbm._defaultmod
>
>     <module 'dbhash' from '/usr/local/lib/python2.2/dbhash.pyc'>


Hi,

It returns dbhash from '/usr/lib/python2.1/dbhash.pyc'.  

Perhaps I should use a different db package?  The data consists
of an enormous number of keys and relatively short data strings.

Amir

>
> If it returns the dumbdbm module, you'll have a good idea where the problem
> lies.  On the other hand, writing out a 200meg file is going to take
> awhile. The underlying db package probably has to do quite a bit of
> shuffling.
>
> If you're accessing a relatively small number of entries in the db file,
> you'd probably be better off letting the underlying library cache things
> for you.




More information about the Python-list mailing list