[Python-bugs-list] [ python-Bugs-827760 ] Bug in dbm - long strings in keys and values

SourceForge.net noreply at sourceforge.net
Sun Oct 26 10:06:53 EST 2003


Bugs item #827760, was opened at 2003-10-21 22:05
Message generated for change (Comment added) made by tinolange
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=827760&group_id=5470

Category: Python Library
Group: Python 2.3
Status: Open
Resolution: None
Priority: 7
Submitted By: John Draper (jdcrunchman)
Assigned to: Nobody/Anonymous (nobody)
Summary: Bug in dbm - long strings in keys and values

Initial Comment:
#!/usr/local/bin/python
#2003-10-19. Feedback to crunch at shopip.com
import dbm

print """
Python dbm bugs summary:
  1. Long strings cause weirdness.
  2. Long keys fail without returning error.

This demonstrates serious bugs in the Python dbm 
module.
Present in OpenBSD versions 2.2, 2.3, and 2.3.2c1.

len(key+string)>61231 results in the item being 'lost', 
without warning.
If the key or string is one character shorter, it is fine.
Writing multiple long strings causes unpredictable results
(none, some, or all of the items are lost without warning).

Curiously, keys of length 57148 return an error, but
longer keys fail without warning
(sounds like an = instead of a > somewhere).
"""

mdb=dbm.open("mdb","n")
print "Writing 1 item to database, but upon reading,"
k='k'
v='X'*61230 #Long string
mdb[k]=v
mdb.close()

md=dbm.open("mdb","r")
print "database contains %i items"%len(md.keys())
md.close()

----------------------------------------------------------------------

Comment By: Tino Lange (tinolange)
Date: 2003-10-26 16:06

Message:
Logged In: YES 
user_id=212920

Hi!

This is no bug - at least no python bug.
It seems to be a limitation in the underlying dbm engine.

I tested it with different dbm implementations under Linux
today. You are right - when using dbm as ndbm.h (which
belongs to libc6, i.e. db1/ndbm.h) this error/limitation occurs.
Please test: At least for me the dbm engine still finds the
data if you specify it, a print md[k] shows the long v
string. Only the keys() and len() methods which iterate over
the database fail.

I also tested without Python, i. e. just using the C-API and
it's just the same (linking against libdb1.so). It is the
underlying dbm_firstkey() which fails when using that HUGE
keys. It just returns NULL.

Context:
--------

drec=dbm_firstkey(dbm);
while(drec.dptr) {
  size++;
  drec = dbm_nextkey(dbm);
}


If you happen to have a system that uses gdbm/ndbm.h as
dbm.h (for example SuSE) or if you change the links in your
include and lib directories accordingly and recompile Python
then this bug disappears and your program is working as
expected. The GNU gdbm engine (I tested 1.73 and 1.8.3)
seems to be much better in handling huge keys as the old
libc dbm.

If possible change to gdbm or assure that dbm is linked
against gdbm (-compat). setup.py tries to be smart about
that but sometimes chooses the old dbm even if it could use
gdbm instead.

ldd /usr/local/lib/python2.3/lib-dynload/dbm.so

shows you what lib you're using.

On my Debian system an
ln -sf /usr/include/gdbm/ndbm.h /usr/include/ndbm.h
ln -sf /usr/lib/libgdbm.so /usr/lib/libndbm.so
fixed it.

Ciao

Tino

----------------------------------------------------------------------

Comment By: John Draper (jdcrunchman)
Date: 2003-10-25 00:47

Message:
Logged In: YES 
user_id=891874

I upped the priority since this bug causes dbm database 
corruption, which could be disastrous for users of dbm or 
shelve.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=827760&group_id=5470



More information about the Python-bugs-list mailing list