Bug in BSDDB?

Robin Dunn robin at alldunn.com
Sat Dec 16 08:37:20 CET 2000

"Bryan Mongeau" <bryan at eevolved.com> wrote in message
news:I9z_5.9426$0U2.369913 at news20.bellglobal.com...
> Greetings Python gurus,
> I was doing some benchmarks of the dbhash module when I came across some
> unsettling behavior in Windows 98:
> 1 - When shelve creates a new database with the "c" flag, it can only add
> 85652 keys to the dictionary before raising an error in bsddb.
> 2- If that error is caught and the write operation tried a second time,
> key is written just fine.
> 3- A dbhash file with 100000 simple records is 140 MB in Windows, compared
> to 21 MB in Unix. If you attempt to add more than 100000 keys in Windows,
> the size of the file quickly balloons to > 1 GIG. Keep in mind I was using
> unicode strings and cPickle.
> 4- An attempt to write more than 100000 keys will result in errors
> described in point 1, with ever-increasing frequency until eventually
> python itself will explode (illegal operation).

I don't know what's going on with bsddb, but I tweaked your samples to run
with my new bsddb3, and to use the dbshelve module included in that package.
(You can get it at http://pybsddb.sourceforge.net/)  The first sample uses
the default DB_HASH access method, and the second uses a DB_BTREE.  This was
run on a Win2K system so there may be some behavior differences for win98,
but I'm not sure to what extent.

As you can see from the outputs below the tests performed without crashing
or getting exceptions all the way up to 250000 and 500000 records.  The only
anomalous behavior is that the time for the hash test of 250000 records was
an order of magnitude larger than expected given the time progression for
the previous tests.  It may be that there were a lot of hash collisions...
Eventhough the keys were being added in sequential order the BTree tests
performed very well.

Other advantages to using bsddb3 are things like database cursors for
iterating over all the records in an efficient manner, the ability for
multiple processes and or threads to access the DB at once with automatic
single writer, multi reader locking semantics (you just have to set a flag),
or full transactional capabilities with logging, commit and rollback.

Here's my test run:

[c:\Temp\Others] python dbshelvetest1.py 10000
writing 10000 records
write took  2.31299996376  seconds
reading 10000 records
read took  1.34200000763  seconds

[c:\Temp\Others] python dbshelvetest1.py 50000
writing 50000 records
write took  14.2899999619  seconds
reading 50000 records
read took  7.41100001335  seconds

[c:\Temp\Others] python dbshelvetest1.py 100000
writing 100000 records
write took  38.7259999514  seconds
reading 100000 records
read took  15.0110000372  seconds

[c:\Temp\Others]  python dbshelvetest1.py 250000
writing 250000 records
write took  311.909000039  seconds
reading 250000 records
read took  33.1779999733  seconds

[c:\Temp\Others] dir testdb.*

 Volume in drive C is unlabeled      Serial number is A06A:886B
 Directory of  C:\Temp\Others\testdb.*

12/15/00  22:54       1,368,064  testdb.10000
12/15/00  22:55       5,193,728  testdb.50000
12/15/00  22:56      10,469,376  testdb.100000
12/15/00  23:04      41,361,408  testdb.250000
     58,392,576 bytes in 4 files and 0 dirs    58,392,576 bytes allocated
  4,029,243,392 bytes free


[c:\Temp\Others] python dbshelvetest2.py 10000
writing 10000 records
write took  1.08200001717  seconds
reading 10000 records
read took  0.570999979973  seconds

[c:\Temp\Others] python dbshelvetest2.py 50000
writing 50000 records
write took  4.43599998951  seconds
reading 50000 records
read took  2.8939999342  seconds

[c:\Temp\Others] python dbshelvetest2.py 100000
writing 100000 records
write took  8.70200002193  seconds
reading 100000 records
read took  5.84899997711  seconds

[c:\Temp\Others] python dbshelvetest2.py 250000
writing 250000 records
write took  21.1699999571  seconds
reading 250000 records
read took  14.8210000992  seconds

[c:\Temp\Others] python dbshelvetest2.py 500000
writing 500000 records
write took  40.699000001  seconds
reading 500000 records
read took  29.7120000124  seconds

[c:\Temp\Others] dir testdbbt.*

 Volume in drive C is unlabeled      Serial number is A06A:886B
 Directory of  C:\Temp\Others\testdbbt.*

12/15/00  23:09       1,466,368  testdbbt.10000
12/15/00  23:10       7,299,072  testdbbt.50000
12/15/00  23:10      14,598,144  testdbbt.100000
12/15/00  23:11      37,601,280  testdbbt.250000
12/15/00  23:12      75,956,224  testdbbt.500000
    136,921,088 bytes in 5 files and 0 dirs    136,921,088 bytes allocated
  3,892,310,016 bytes free


Robin Dunn
Software Craftsman
robin at AllDunn.com
http://wxPython.org     Java give you jitters?
http://wxPROs.com        Relax with wxPython!

begin 666 dbshelvetest1.py
M;G0@/2!I;G0H<WES+F%R9W9;,5TI#0H-"F1B(#T at 9&)S:&5L=F4N;W!E;B at B
M=&5S=&1B+B5D(B E(')E8V-O=6YT+")C(BD-"G1S(#T@=&EM92YT:6UE*"D-
M"@T*<')I;G0@(G=R:71I;F<@)60@<F5C;W)D<R(@)2!R96-C;W5N= T*#0IF
M>3UX#0H@('=H:6QE(#$Z#0H@(" @=')Y. at T*(" @(" @9&);<W1R*&ME>2E=
M(#T at 9&%T80T*(" @(" @8G)E86L-"B @("!E>&-E<'0Z#0H@(" @("!P<FEN
M=" B97)R;W(B+"!S='(H;G5M17)R;W)S*2P@(F%T(')E8V]R9"(L('-T<BAX
M*0T*(" @(" @;G5M17)R;W)S*STQ#0H-"F1B+G-Y;F,H*0T*#0IT9B ]('1I
M;64N=&EM92 at I#0IP<FEN=" B=W)I=&4@=&]O:R B+"!S='(H=&8M=',I+" B
M=6YT*3H-"B @9&%T82 ](&1B6W-T<BAX*5T-"@T*=&8@/2!T:6UE+G1I;64H
M*0T*<')I;G0@(G)E860@=&]O:R B+"!S='(H=&8M=',I+" B('-E8V]N9',B

begin 666 dbshelvetest2.py
M8G-D9&(S(&EM<&]R="!D8G-H96QV92P at 9&(-"@T*;G5M17)R;W)S/3$-"G)E
M8V-O=6YT(#T@:6YT*'-Y<RYA<F=V6S%=*0T*#0ID8B ](&1B<VAE;'9E+F]P
M96XH(G1E<W1D8F)T+B5D(B E(')E8V-O=6YT+" B8R(L(&9I;&5T>7!E/61B
M+D1"7T)44D5%*0T*=',@/2!T:6UE+G1I;64H*0T*#0IP<FEN=" B=W)I=&EN
M9R E9"!R96-O<F1S(B E(')E8V-O=6YT#0H-"F9O<B!X(&EN(')A;F=E*')E
M8V-O=6YT*3H-"B @9&%T83U;=2)S;VUE('1E<W0 at 9&%T82(L('4B<V]M92!M
M;W)E('1E<W0 at 9&%T82(L('-T<BAX*5T-"B @:V5Y/7 at -"B @=VAI;&4@,3H-
M"B @("!T<GDZ#0H@(" @("!D8EMS='(H:V5Y*5T@/2!D871A#0H@(" @("!B
M<F5A:PT*(" @(&5X8V5P=#H-"B @(" @('!R:6YT(")E<G)O<B(L('-T<BAN
M=6U%<G)O<G,I+" B870@<F5C;W)D(BP@<W1R*'@I#0H@(" @("!N=6U%<G)O
M<G,K/3$-"@T*9&(N<WEN8R at I#0H-"G1F(#T@=&EM92YT:6UE*"D-"G!R:6YT
M;G0@(G)E861I;F<@)60@<F5C;W)D<R(@)2!R96-C;W5N= T*=',@/2!T:6UE
M+G1I;64H*0T*9F]R('@@:6X@<F%N9V4H<F5C8V]U;G0I. at T*("!D871A(#T@
M9&);<W1R*'@I70T*#0IT9B ]('1I;64N=&EM92 at I#0IP<FEN=" B<F5A9"!T

More information about the Python-list mailing list