[Python-Dev] Re: Another test_compiler mystery

[Tim Peters]

[Armin Rigo]
> I don't remember if it was mentioned here, but maybe we'd better check
> directly whether the C stack is too large rather than (or possibly in addition
> to) using a limit on the number of Python iterations.

How large is too large?  The Windows way checks the stack for
impending overflow directly.  A long "stack check on Unix" thread here
in 2000 fizzled out in the expected way:  no two Unices appear to have
anything in common <0.9 wink>.

> This is not really ANSI C, but I can't think of a setting in which the
> following trick would fail to be a good indication of the amount of stack
> space consumed: save a pointer to a local variable "early", e.g. in
> Py_Initialize(); then in any other function call, the distance between this
> pointer and a pointer to some local equals the amount of stack space consumed
> by Python minus a few bytes.

It fails for threads, whose stacks may be anywhere in relation to the
main thread's.  So any gimmick like this has to have a thread-local
"starting point".  If "a stack" is always a contiguous slice of
memory, then it can work.

> If this sounds too much of a hack,

No, but I'm not sure how much additional hackery it requires to make
it work in all cases.

> the (usual) recursion limit could be kept to limit nested Python frames; but all
> C-level recursions (including eval_frame) could additionally use the above trick.
>  Its advantage is that it is an extremely fast check.

The ways to do it that don't work are indeed extremely fast <wink>.

> If the total stack size is difficult to know in advance,

That's apparently the case across platforms -- and someimes the
main-thread stack is much larger than secondary-thread stacks.  The
Windows check doesn't even try to know the current stack's size, but
relies on MS-specific C exception extensions.

> we can still use PyOS_CheckStack(), but more efficiently and less randomly
> than now, by maintaining a "high tide" pointer that records how much stack we
> are sure we have, and calling PyOS_CheckStack() only to attempt to push
> the "high tide" mark further.

Threads again complicate this.  AFAICT, the only platform that defines
USE_STACKCHECK now (and actually implements PyOS_CheckStack()) is
32-bit Windows using an MS compiler.  I did boost the stack
reservation for that platform in CVS, which can't solve it, but at
least hides it better <wink>.

> While I'm in a hacking mood, there might be a way to prevent PyErr_Clear() to
> clear away "asynchronuous" exceptions like RuntimeError, MemoryError, and
> KeyboardInterrupt: for these exceptions, let's just store them away instead of
> clearing them, and re-raise them at the next occasion.  The "next occasion"
> would have to be defined more precisely, but there is probably a way to ensure
> that it will at least be before the next non-trivial opcode operation starts.
> It would overwrite an exception set later, like those spurious KeyError we get
> for dict lookups.  It might be a better-than-nothing quick fix to all these
> PyErr_Clear() all around the code base.

Threads probably complicate that too.  It's dreadful that serious
problems can get transformed into bogus KeyErrors (in the test driver
I posted, I'm not sure I spelled this out, but the key always *was* in
the dict when a MemoryError got turned into a KeyError; we called a
comparison function to begin with because the hash codes matched, and
since these were mostly small-integer keys, they were identical to
their hash codes -- so these keys were in the dicts, and the KeyErrors
were lying about that).  That's just nuts.  I don't have spare cycles
to give to it, though, so you'll have to save the world on your own
again.