anydbm is slow
Amir Michail
amichail at cse.unsw.edu.au
Tue Jan 1 19:19:24 EST 2002
On Wed, 2 Jan 2002 11:10, Skip Montanaro wrote:
> Amir> I'm trying to write out a large database file using anydbm.
> Amir> However, it's quite slow. Writing out a 200 meg file can take
> Amir> hours.
>
> Amir> In fact, I do most of the work using an internal map and write
> out Amir> that map at the end since I thought this would be faster. (I
> have Amir> lots of memory.) However, it's still very slow.
>
> Amir> Any hints?
>
> What underlying db module does anydbm use?
>
> >>> import anydbm
> >>> anydbm._defaultmod
>
> <module 'dbhash' from '/usr/local/lib/python2.2/dbhash.pyc'>
Hi,
It returns dbhash from '/usr/lib/python2.1/dbhash.pyc'.
Perhaps I should use a different db package? The data consists
of an enormous number of keys and relatively short data strings.
Amir
>
> If it returns the dumbdbm module, you'll have a good idea where the problem
> lies. On the other hand, writing out a 200meg file is going to take
> awhile. The underlying db package probably has to do quite a bit of
> shuffling.
>
> If you're accessing a relatively small number of entries in the db file,
> you'd probably be better off letting the underlying library cache things
> for you.
More information about the Python-list
mailing list