On Monday 27 October 2003 10:40 am, Gregory P. Smith wrote: ...
The big difference i see between 2.3cvs and 2.4cvs that could "explain" it is that Lib/bsddb/__init__.py has been updated to use a private (in memory, single process only) DBEnv with locking and thread support enabled. That explains why db->del() would be doing locking. But not why it would deadlock.
*AH*! I wasn't looking in the right place, silly me. Good job!!! Yes, now that you've pointed it out, the change from 2.3's d = db.DB() to 2.4's e = _openDBEnv() d = db.DB(e) must be the culprit. I still don't quite see how the lock ends up being "held", but, don't mind me -- the intricacy of mixins and wrappings and generators and delegations in those modules is making my head spin anyway, so it's definitely not surprising that I can't quite see what's going on.
How do python dictionaries deal with modifications to the dictionary intermixed with iteration?
In general, Python doesn't deal well with modifications to any iterable in the course of a loop using an iterator on that iterable. The one kind of "modification during the loop" that does work is: for k in somedict: somedict[k] = ...whatever... i.e. one can change the values corresponding to keys, but not change the set of keys in any way -- any changes to the set of keys can cause unending loops or other such misbehavior (not deadlocks nor crashes, though...). However, on a real Python dict, k, v = thedict.iteritems().next() doesn't constitute "a loop" -- the iterator object returned by the iteritems call is dropped since there are no outstanding references to it right after this statement. So, following up with del thedict[k] is quite all right -- the dictionary isn't being "looped on" at that time. Given that in bsddb's case that iteritems() first [and only] next() boils down to a self.first() which in turn does a self.dbc.first() I _still_ don't see exactly what's holding the lock. But the simplest fix would appear to be in __delitem__, i.e., if we have a cursor we should delete through it: def __delitem__(self, key): self._checkOpen() if self.dbc is not None: self.dbc.set(key) self.dbc.delete() else: del self.db[key] ...but this doesn't in fact remove the deadlock on the unit-test for popitem, which just confirms I don't really grasp what's going on, yet!-) Alex