[ python-Bugs-881522 ] Shelve slow after 7/8000 key

[SourceForge.net]

Bugs item #881522, was opened at 2004-01-21 11:09
Message generated for change (Comment added) made by montanaro
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881522&group_id=5470

Category: Windows
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Marco Beri (marcoberi)
Assigned to: Gregory P. Smith (greg)
Summary: Shelve slow after 7/8000 key

Initial Comment:
After about 8.000 insertion shelve became really, really 
slow.
This happens only with 2.3.3 #51 on Windows, not with 
2.2 and with 2.3 on Linux.
I try with writeback True or False: same problem.
Help! :-))


----------------------------------------------------------------------

>Comment By: Skip Montanaro (montanaro)
Date: 2004-01-22 12:02

Message:
Logged In: YES 
user_id=44345

Try test3skip.py.  You run it like this:

    python test3skip.py hashopen
    python test3skip.py btopen

I ran it on win2k under cygwin so I could use the time command 
(but ran the Windows version of Python).  Using btopen was much 
faster.  I got rid of shelve to eliminate it and pickle as possible 
sources of problems.

$ time /cygdrive/c/Python23/python test3skip.py hashopen

real    0m6.801s
user    0m0.015s
sys     0m0.000s

Administrator at CYCLOPS ~/tmp
$ time /cygdrive/c/Python23/python test3skip.py btopen

real    0m0.345s
user    0m0.015s
sys     0m0.015s

I don't know if the relationship between real, user and sys time 
means anything on cygwin, but the reported real times are very 
repeatable and match my subjective feel of the elapsed time.  This 
suggests there's something fishy with either the underlying library 
or with __setitem__ when using hash files.

I'm assigning to Greg so he can take a peek.  As the bsddb/
pybsddb guy he might have some better insight (certainly better 
than me).

----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2004-01-22 12:01

Message:
Logged In: YES 
user_id=44345

Try test3skip.py.  You run it like this:

    python test3skip.py hashopen
    python test3skip.py btopen

I ran it on win2k under cygwin so I could use the time command 
(but ran the Windows version of Python).  Using btopen was much 
faster.  I got rid of shelve to eliminate it and pickle as possible 
sources of problems.

$ time /cygdrive/c/Python23/python test3skip.py hashopen

real    0m6.801s
user    0m0.015s
sys     0m0.000s

Administrator at CYCLOPS ~/tmp
$ time /cygdrive/c/Python23/python test3skip.py btopen

real    0m0.345s
user    0m0.015s
sys     0m0.015s

I don't know if the relationship between real, user and sys time 
means anything on cygwin, but the reported real times are very 
repeatable and match my subjective feel of the elapsed time.  This 
suggests there's something fishy with either the underlying library 
or with __setitem__ when using hash files.

I'm assigning to Greg so he can take a peek.  As the bsddb/
pybsddb guy he might have some better insight (certainly better 
than me).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-01-22 11:29

Message:
Logged In: YES 
user_id=31435

FYI, on a Win98SE box, test1skip.py took about 30 seconds 
under 2.3.3, and about 1 second under both 2.2.3 and 2.1.3.  
Under 2.3.3, no significant time is taken by a.close(), so it's 
all in the loop.  It prints "dbhash" under all versions.

----------------------------------------------------------------------

Comment By: Marco Beri (marcoberi)
Date: 2004-01-22 01:30

Message:
Logged In: YES 
user_id=588604

I tried your version: 31.36 seconds vs 0.65.
Just to be sure I tried on three different computers with 
Windows 2000: same gap.

[c:\tmp]timer & \Python23\python test1skip.py & timer
Timer 1 on:  8.21.58
dbhash
Timer 1 off:  8.22.29  Elapsed: 0.00.31,36

[c:\tmp]timer & \Python22\python test1skip.py & timer
Timer 1 on:  8.22.40
dbhash
Timer 1 off:  8.22.41  Elapsed: 0.00.00,65


----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2004-01-21 18:28

Message:
Logged In: YES 
user_id=44345

Can't reproduce on Mac OS X.  I tried with 2.2, 2.3 and CVS using
attached test1skip.py (no writeback - 2.2 doesn't support it, no
import pickle - not used, no key prints - just muddies the water,
print whichdb's result).

The times are close enough to not worry me:

montanaro:tmp% time python2.3 test1.py
dbhash

real    0m1.927s
user    0m1.720s
sys     0m0.080s
montanaro:tmp% time python2.2 test1.py
dbhash

real    0m1.250s
user    0m0.850s
sys     0m0.360s
montanaro:tmp% time python test1.py
dbhash

real    0m2.179s
user    0m1.950s
sys     0m0.120s

Please try this modified version just to make sure we are both
looking at the same thing.



----------------------------------------------------------------------

Comment By: Marco Beri (marcoberi)
Date: 2004-01-21 17:57

Message:
Logged In: YES 
user_id=588604

Skip Montanaro discovered that whichdb repors bsddb185 
with python 2.2 and dbhash with 2.3.3.
So why is it so slow after few thousand keys?

----------------------------------------------------------------------

Comment By: Thomas Heller (theller)
Date: 2004-01-21 12:24

Message:
Logged In: YES 
user_id=11105

Hm, are windows bugs automatically assigned to me ;-)??

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881522&group_id=5470