[issue14417] dict RuntimeError workaround

[Guido van Rossum]

Guido van Rossum <guido at python.org> added the comment:

On Sun, Apr 1, 2012 at 1:58 PM, Raymond Hettinger
<report at bugs.python.org> wrote:
[...]
> I'm wary of the original RuntimeError patch because
[...]

I had retorts to most of what you wrote, but decided not to post them.
Instead, I really want to have a serious discussion about this issue,
without falling back on "if it ain't broke don't fix it". There was
definitely *something* wrong with the original code: there was a known
crasher and it had XXX comments in it, and some folks cared enough to
attempt to fix it.

In the end I see this more as a matter of clarifying the semantics and
limitations of dict operations: *should* the language guarantee that
the built-in dict implementation supports correct lookup even if the
in the middle of the lookup the hash table is resized by another
thread? Is this a reasonable guarantee to impose on all Python
implementations? Because I don't see how this can be guaranteed
without a lock; but I haven't tried very hard, so this is not a
rhetorical question.

- - -

On Sun, Apr 1, 2012 at 4:01 PM, Nick Coghlan <report at bugs.python.org> wrote:
> A thought prompted by Raymond's comment: did we ever try just protecting
> the retry as a recursive call? If we can stop the stack blowing up, it
> seems to me we'd get the best of both worlds (that is, crashes become
> RuntimeError, but naive multi-threaded code is unaffected).

Hm... We considered a variety of fixes in
http://bugs.python.org/issue14205 but I don't think we tried that.
(The nastiness is that it's strictly a C-level recursion -- lookdict()
calls lookdict().)

Another thing we didn't try was manually eliminating the tail
recursion in lookdict() using a goto (and perhaps some variable
assignments). I don't see the potential for an infinite loop as a
problem, since one could be created just as easily by putting "while
True: pass" inside a user-defined __hash__().

PS. I'm not surprised your originall hammer_dict.py didn't provoke the
problem -- it only overrides __hash__(), but lookdict() never calls
that. It only calls the compare function; the hash is computed once
and passed into it.

Finally, the guard in lookdict() looks fishy to me:

  if (ep0 == mp->ma_table && ep->me_key == startkey) {

Can't this evaluate to true when the dict has shrunk dramatically in
size? I think an extra test for "mask == mp->ma_mask" ought to be
added.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14417>
_______________________________________