[Python-Dev] Re: Another test_compiler mystery

[Michael Hudson]

Tim Peters <tim.peters at gmail.com> writes:

> Well, this gets nasty.

You're certainly not wrong about that!

[snip Windows details]

> That's
>
> 	if (Py_EnterRecursiveCall(" in cmp"))
> 		return NULL;
>
> in PyObject_RichCompare.  That's just trying to compare two ints.
>
> So NULL gets returned to PyObject_RichCompareBool, which in turn
> returns -1 to lookdict.  AAAARGHGH!  lookdict "isn't allowed" to raise
> exceptions, so it does a PyErr_Clear(), goes on to futilely chase the
> entire dict looking for another match on the hash code, and we've
> effectively turned a MemoryError into a KeyError.  I expect that
> explains a lot about what we see in the release-build runs.

This part happens on Linux, too, it seems.  Running your script:

114 ok
115 ok
116 ok
117 ok
118 ok
119 ok
120 ok
121 ok
122 ok
123 exceptions.RuntimeError maximum recursion depth exceeded
124 exceptions.RuntimeError maximum recursion depth exceeded
125 exceptions.RuntimeError maximum recursion depth exceeded
126 exceptions.KeyError 307
127 exceptions.RuntimeError maximum recursion depth exceeded
128 exceptions.KeyError 306
129 exceptions.RuntimeError maximum recursion depth exceeded
130 exceptions.KeyError 305
131 exceptions.RuntimeError maximum recursion depth exceeded
132 exceptions.KeyError 304
133 exceptions.RuntimeError maximum recursion depth exceeded
134 exceptions.KeyError 303
135 exceptions.KeyError 302
136 exceptions.RuntimeError maximum recursion depth exceeded
137 exceptions.KeyError 301

This is a debug build, so we're not abusing the stack quite so much on
linux as apparently seems to happen in debug builds on windows.

This isn't going to be something we can pin on the new Windows tools,
is it?  It doesn't seem like a feature that a debug build gets a tiny
stack, but, well...

> The question is what we did since 2.3.4 that apparently increases our
> stack demands, and grossly increases them in a debug build(!).
               ^
on windows; I don't see anything so scary on Linux.  

Hmm, maybe that's a lie, I don't see the crazy KeyErrors with 2.3.

> Could be that the compile package is more heavily recursive now too
> (no idea).

Really don't think so.

> test_parser.py in 2,3.4 contained the same deeply nested tuples, so
> that's not what changed.

Something that *has* changed, however, is that we lost the code that
compared objects recursively and now PyObject_RichCompare uses the
more general Py_EnterRecursiveCall mechanism and not its own counter.
So now we're grossly more likely to hit one of these faux KeyErrors in
2.3 than 2.4 (you'd need to invoke a Python __eq__ method or so).

Would it make more sense for PyObject_RichCompare to still use a
private counter but raise RuntimeError if it is exceeded instead of
trying to be clever?  That would make this problem /less/ likely,
though obviously still possible (functions that claim they 'can't
fail' are lying).

Running Misc/find_recursionlimit.py with a debug build yields ...
well, I gave up and killed it at 22000 (hmm, seems the default 'ulimit
-s' is "unlimited"... I'm sure this hasn't always had this effect with
a threaded build).

Cheers,
mwh

-- 
  > Emacs is a fashion statement.
  No, Gnus is a fashion statement.  Emacs is clothing.  Everyone 
  else is running around naked.
        -- Karl Kleinpaste & Jonadab the Unsightly One, gnu.emacs.gnus