[Python-bugs-list] [ python-Bugs-775414 ] bsddb3 hash craps out with threads

[SourceForge.net]

Bugs item #775414, was opened at 2003-07-21 21:29
Message generated for change (Comment added) made by montanaro
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=775414&group_id=5470

Category: Extension Modules
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Tim Peters (tim_one)
Assigned to: Gregory P. Smith (greg)
Summary: bsddb3 hash craps out with threads

Initial Comment:
Richie Hindle presented something like the attached 

(hammer.py) on the spambayes-dev mailing list.  On 

Win98SE and Win2K w/ Python 2.3c1 I usually see this 

death pretty quickly:



Traceback (most recent call last):

  File "hammer.py", line 36, in ?

    main()

  File "hammer.py", line 33, in main

    hammer(db)

  File "hammer.py", line 15, in hammer

    x = db[str(int(random.random() * 100000))]

  File "C:\CODE\PYTHON\lib\bsddb\__init__.py", line 86, 

in __getitem__

    return self.db[key]

bsddb._db.DBRunRecoveryError: (-30982,

     'DB_RUNRECOVERY: Fatal error, run database 

recovery -- fatal region error detected; run recovery')



Richie also reported "illegal operation" crashes on 

Win98SE.



It's not clear whether a bsddb3 hash *can* be used 

with threads like this.  If it can't, there's a doc bug.  If it 

should be able to, there's a more serious problem.  Note 

that it looks like hashopen() always merges DB_THREAD 

into the flags, so the absence of specifying DB_THREAD 

probably isn't the problem.

----------------------------------------------------------------------

>Comment By: Skip Montanaro (montanaro)
Date: 2003-09-12 13:23

Message:
Logged In: YES 
user_id=44345

>From what I got back from Sleepycat on this, I'm pretty sure the 

old bsddb interface is not going to be thread safe.  Attached are 

two messages from Sleepycat.



Is there some way for the old interface to create a default

environment shared by all the bsddb.*open() calls and then set

the DB_RECOVER flag in the low-level open() call?



----------------------------------------------------------------------

Comment By: Jeremy Hylton (jhylton)
Date: 2003-09-12 12:14

Message:
Logged In: YES 
user_id=31392

How does the bsddb wrapper achieve thread safety?



I know very little about the wrapper or the underlying bsddb

libraries.  I found the following comment in the C API docs:



http://www.sleepycat.com/docs/ref/program/mt.html#2

> When using the non-cursor Berkeley DB calls to retrieve 

> key/data items (for example, DB->get), the memory to which

the 

> pointer stored into the Dbt refers is valid only until the

next call 

> using the DB handle returned by DB->open. This includes any 

> use of the returned DB handle, including by another thread 

> within the process.



This suggests that a call to a self->db->get() must process

its results (copy them into Python-owned memory) before any

other operation on the same db object can proceed.  Is that

right?



The bsddb wrapper releases the GIL before calling the

low-level DB API functions and the acquires it after the

call returns.  Is there some other lock that prevents

multiple simultaneous calls from stomping on each other?



----------------------------------------------------------------------

Comment By: Jeremy Hylton (jhylton)
Date: 2003-09-12 11:46

Message:
Logged In: YES 
user_id=31392

I'm running this test with CVS Python (built on 9/11/03) on

RH Linux 9 with bsddb 4.1.25.  I see the same error although

it takes a relatively long time to provoke -- a minute or two.



----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-09-12 11:08

Message:
Logged In: YES 
user_id=31435

Greg, any luck?  We're starting to see the same error ("fatal 

region error detected") in some ZODB tests using bsddb3, and 

that's an infinitely more complicated setup than this little 

program.  Jeremy Hylton also sees "fatal region" errors on 

Linux, in the ZODB context.

----------------------------------------------------------------------

Comment By: Gregory P. Smith (greg)
Date: 2003-08-13 18:26

Message:
Logged In: YES 
user_id=413

i'll try and reproduce this.

----------------------------------------------------------------------

Comment By: Richie Hindle (richiehindle)
Date: 2003-07-22 03:50

Message:
Logged In: YES 
user_id=85414

Minor correction: I'm on Plain Old Win98, not SE.



For what it's worth, the script seems more often than not

to provoke an application error when there's background

load, and a DBRunRecoveryError when there isn't.



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=775414&group_id=5470