[Python-Dev] Re: Another test_compiler mystery
tim.peters at gmail.com
Mon Aug 16 04:50:25 CEST 2004
> I don't remember if it was mentioned here, but maybe we'd better check
> directly whether the C stack is too large rather than (or possibly in addition
> to) using a limit on the number of Python iterations.
How large is too large? The Windows way checks the stack for
impending overflow directly. A long "stack check on Unix" thread here
in 2000 fizzled out in the expected way: no two Unices appear to have
anything in common <0.9 wink>.
> This is not really ANSI C, but I can't think of a setting in which the
> following trick would fail to be a good indication of the amount of stack
> space consumed: save a pointer to a local variable "early", e.g. in
> Py_Initialize(); then in any other function call, the distance between this
> pointer and a pointer to some local equals the amount of stack space consumed
> by Python minus a few bytes.
It fails for threads, whose stacks may be anywhere in relation to the
main thread's. So any gimmick like this has to have a thread-local
"starting point". If "a stack" is always a contiguous slice of
memory, then it can work.
> If this sounds too much of a hack,
No, but I'm not sure how much additional hackery it requires to make
it work in all cases.
> the (usual) recursion limit could be kept to limit nested Python frames; but all
> C-level recursions (including eval_frame) could additionally use the above trick.
> Its advantage is that it is an extremely fast check.
The ways to do it that don't work are indeed extremely fast <wink>.
> If the total stack size is difficult to know in advance,
That's apparently the case across platforms -- and someimes the
main-thread stack is much larger than secondary-thread stacks. The
Windows check doesn't even try to know the current stack's size, but
relies on MS-specific C exception extensions.
> we can still use PyOS_CheckStack(), but more efficiently and less randomly
> than now, by maintaining a "high tide" pointer that records how much stack we
> are sure we have, and calling PyOS_CheckStack() only to attempt to push
> the "high tide" mark further.
Threads again complicate this. AFAICT, the only platform that defines
USE_STACKCHECK now (and actually implements PyOS_CheckStack()) is
32-bit Windows using an MS compiler. I did boost the stack
reservation for that platform in CVS, which can't solve it, but at
least hides it better <wink>.
> While I'm in a hacking mood, there might be a way to prevent PyErr_Clear() to
> clear away "asynchronuous" exceptions like RuntimeError, MemoryError, and
> KeyboardInterrupt: for these exceptions, let's just store them away instead of
> clearing them, and re-raise them at the next occasion. The "next occasion"
> would have to be defined more precisely, but there is probably a way to ensure
> that it will at least be before the next non-trivial opcode operation starts.
> It would overwrite an exception set later, like those spurious KeyError we get
> for dict lookups. It might be a better-than-nothing quick fix to all these
> PyErr_Clear() all around the code base.
Threads probably complicate that too. It's dreadful that serious
problems can get transformed into bogus KeyErrors (in the test driver
I posted, I'm not sure I spelled this out, but the key always *was* in
the dict when a MemoryError got turned into a KeyError; we called a
comparison function to begin with because the hash codes matched, and
since these were mostly small-integer keys, they were identical to
their hash codes -- so these keys were in the dicts, and the KeyErrors
were lying about that). That's just nuts. I don't have spare cycles
to give to it, though, so you'll have to save the world on your own
More information about the Python-Dev