On Tue, Oct 28, 2003 at 11:12:21AM +0100, Alex Martelli wrote:
On Monday 27 October 2003 10:56 pm, Gregory P. Smith wrote:
What about the behaviour of multiple iterators for the same dict being used at once (either interleaved or by multiple threads; it shouldn't matter)? I expect that works fine in python.
If the dict is not being modified, or if the only modifications on it are assigning different values for already-existing keys, multiple iterators on the same unchanging dict do work fine in one or more threads. But note that iterators only "read" the dict, don't change it. If any change to the set of keys in the dict happens, all bets are off. ...
This is something the _DBWithCursor iteration interface does not currently support due to its use of a single DBCursor internally.
_DBWithCursor is currently written such that the cursor is never closed once created. This leaves tons of potential for deadlock even in single threaded apps. Reworking _DBWithCursor into a _DBThatUsesCursorsSafely such that each iterator creates its own cursor in an internal pool and other non cursor methods that would write to the db destroy all cursors after saving their current() position so that the iterators can reopen+reposition them is a solution.
Woof. I think I understand what you're saying. However, writing to a dict (in the sense of changing the sets of keys) while one is iterating on the dict is NOT supported in Python -- basically "undefined behavior" (which does NOT include possibilities of crashes and deadlocks, though). So, maybe, we could get away with something a bit less rich here?
I just implemented and committed something about that rich. I believe I could simplify it: have __iter__() and iteritems() return if their cursor was closed out from underneath them instead of the current attempt to reopen a cursor, reposition themselves, and keep going [which could still have unpredictable results since a db modification could rearrange the keys in some types of databases].
So, maybe I _should_ just fix popitem that way and see if all tests pass? I dunno -- it feels a bit like fixing the symptoms and leaving some deep underlying problems intact...
My commit fixed the deadlock problem for the single threaded case and wrote a test case to prove it. I opened a SF bug to track fixing the deadlock possibilities in the multithreaded case (and a memory leak i believe i added). -g