[Python-bugs-list] [ python-Bugs-834461 ] simple bsddb interface potential for deadlock with threads

[SourceForge.net]

Bugs item #834461, was opened at 2003-11-02 01:30
Message generated for change (Comment added) made by greg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=834461&group_id=5470

Category: Extension Modules
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Gregory P. Smith (greg)
Assigned to: Gregory P. Smith (greg)
Summary: simple bsddb interface potential for deadlock with threads

Initial Comment:
>From Lib/bsddb/__init__.py

        # FIXME-20031101-greg: I believe there is still the potential
        # for deadlocks in a multithreaded environment if someone
        # attempts to use the any of the cursor interfaces in one
        # thread while doing a put or delete in another thread.  The
        # reason is that _checkCursor and _closeCursors are not atomic
        # operations.  Doing our own locking around self.dbc,
        # self.saved_dbc_key and self._iter_cursors could prevent this.
        # TODO: A test case demonstrating the problem needs to be written.


----------------------------------------------------------------------

>Comment By: Gregory P. Smith (greg)
Date: 2003-11-02 19:29

Message:
Logged In: YES 
user_id=413

A fix for my first comment about the memory leak has been committed that uses weak references and callbacks so that DBCursor objects are always destroyed and their references removed from _DBWithCursor's pool when the iterator that created them is destroyed.

The potential threading deadlock issues still stand.  Lockng seems like it could have a serious performance impact by surrounding way too many database operations (all cursor operations, iterator use and __delitem__, __setitem__ and close) with locks.

what should be done:

_DBWithCursor __delitem__, __setitem__ and close methods need to prevent any cursors from being created until they have completed.  But for performance reasons multiple database read operations should be allowed at the same time (get, cursor methods or iterators).

----------------------------------------------------------------------

Comment By: Gregory P. Smith (greg)
Date: 2003-11-02 01:47

Message:
Logged In: YES 
user_id=413

While I'm at it, I think there might be a memory leak in my
__iter__ and iteritems() implementation when the resulting
generator objects are destroyed without completing their
iteration (as will happen in UserDict.DictMixIn.popitem
among other things).

They add their DBCursor to _DBWithCursor._iter_cursors but
only ever delete it from that hash before a return rather
than a yield.  The solution to this should be simple: have
_closeCursors() empty the _iter_cursors hash after calling
close() on all of the cursors.  __iter__ and iteritems()
already ignore a KeyError when trying to remove their cursor
from the map when returning.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=834461&group_id=5470