[Python-Dev] Re: test_bsddb blocks testing popitem - reason

[Gregory P. Smith]

On Tue, Oct 28, 2003 at 11:12:21AM +0100, Alex Martelli wrote:
> On Monday 27 October 2003 10:56 pm, Gregory P. Smith wrote:
> > What about the behaviour of multiple iterators for the same dict being
> > used at once (either interleaved or by multiple threads; it shouldn't
> > matter)?  I expect that works fine in python.
> 
> If the dict is not being modified, or if the only modifications on it are
> assigning different values for already-existing keys, multiple iterators
> on the same unchanging dict do work fine in one or more threads.
> But note that iterators only "read" the dict, don't change it.  If any
> change to the set of keys in the dict happens, all bets are off.
...
> > This is something the _DBWithCursor iteration interface does not currently
> > support due to its use of a single DBCursor internally.
> >
> > _DBWithCursor is currently written such that the cursor is never closed
> > once created.  This leaves tons of potential for deadlock even in single
> > threaded apps.  Reworking _DBWithCursor into a _DBThatUsesCursorsSafely
> > such that each iterator creates its own cursor in an internal pool
> > and other non cursor methods that would write to the db destroy all
> > cursors after saving their current() position so that the iterators can
> > reopen+reposition them is a solution.
> 
> Woof.  I think I understand what you're saying.  However, writing to a
> dict (in the sense of changing the sets of keys) while one is iterating
> on the dict is NOT supported in Python -- basically "undefined behavior"
> (which does NOT include possibilities of crashes and deadlocks, though).
> So, maybe, we could get away with something a bit less rich here?

I just implemented and committed something about that rich.

I believe I could simplify it: have __iter__() and iteritems() return if
their cursor was closed out from underneath them instead of the current
attempt to reopen a cursor, reposition themselves, and keep going [which
could still have unpredictable results since a db modification could
rearrange the keys in some types of databases].

> So, maybe I _should_ just fix popitem that way and see if all tests pass?
> I dunno -- it feels a bit like fixing the symptoms and leaving some deep
> underlying problems intact...

My commit fixed the deadlock problem for the single threaded case and
wrote a test case to prove it.  I opened a SF bug to track fixing the
deadlock possibilities in the multithreaded case (and a memory leak i
believe i added).

-g