[ python-Bugs-849662 ] reading shelves is really slow

SourceForge.net noreply at sourceforge.net
Thu Nov 27 04:17:05 EST 2003


Bugs item #849662, was opened at 2003-11-26 09:06
Message generated for change (Comment added) made by rhettinger
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=849662&group_id=5470

Category: Extension Modules
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Gottfried Ganßauge (ganssauge)
Assigned to: Raymond Hettinger (rhettinger)
Summary: reading shelves is really slow

Initial Comment:
My application uses a shelve-file which is created by 
another process using the same python version.
Before python2.3 using this shelve with the exact same 
application was almost twice as fast as a binary pickle 
containing the same data.
Now with python2.3 the same application is suddenly 
about 150 times slower than using the binary pickle.

The usage is as follows:
   idx_dict = shelve.open (idx_dict_name, "r")
   ...
   while not infile.eof:
      index = get_index_from_somewhere_else()
      if not idx_dict.has_key (index):
          do_something(index)
      else:
          do_something_else(index)

   idx.dict.close()
   
Profiling revealed that most of the time is spent within 
userdict.

----------------------------------------------------------------------

>Comment By: Raymond Hettinger (rhettinger)
Date: 2003-11-27 04:17

Message:
Logged In: YES 
user_id=80475

I can reproduce a four-fold slowdown that persists even
after the UserDict.DictMixin lines are commented out of
shelve.py and bsddb.__init__.py.  For me, the only thing
that has changed is the underlying bsddb implementation.

Let's see if you system is going somewhere else to get its
shelving done.  After the first line, add:  idx_dict.has_key
([])
Then post the traceback here.

Do that for both Py2.2 and for Py2.3.  Thank you.

Also, post what a typical record in the index and tell me
how many entries are typically in idx_dict.  That way, I can
try to reproduce your timings with greater fidelity.

Which os are you using and what the minor bugfix verion
numbers of the Py2.2 and PY2.3 you are using.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=849662&group_id=5470



More information about the Python-bugs-list mailing list