Another test_compiler mystery

I've noticed several times now, in both debug and release builds, that if I run regrtest.py with -uall, *sometimes* it just stops after running test_compiler: $ python_d regrtest.py -uall test_grammar test_opcodes ... test_compare test_compile test_compiler $ There's no indication of error, it just ends. It's not consistent. Happened once when I was running with -v, and test_compiler's output ended here: ... compiling C:\Code\python\lib\test\test_operator.py compiling C:\Code\python\lib\test\test_optparse.py compiling C:\Code\python\lib\test\test_os.py compiling C:\Code\python\lib\test\test_ossaudiodev.py compiling C:\Code\python\lib\test\test_parser.py In particular, there's no Ran M tests in Ns output, so it doesn't look like unittest (let alone regrtest) ever got control back. Hmm. os.listdir() is in sorted order on NTFS, so test_compiler should be chewing over a lot more files after test_parser.py. *This* I could blame on a blown C stack -- although I'd expect a much nastier symptom then than just premature termination. Anyone else?

Here's a cute one: """ import compiler, sys f = open('../Lib/test/test_parser.py') guts = f.read() f.close() def ouch(n): if n == 0: return compiler.compile(guts, "<string>", "exec") else: return ouch(n-1) for n in range(100, 250): try: ouch(n) msg = 'ok' except Exception, msg: msg = str(sys.exc_info()[0]) + ' ' + str(msg) print n, msg """ Under 2.3.4, that works as expected: when n hit a large enough value, from then on it was all repetitions of recursion-depth exceptions: ,,, 147 ok 148 ok 149 ok 150 exceptions.RuntimeError maximum recursion depth exceeded 151 exceptions.RuntimeError maximum recursion depth exceeded 152 exceptions.RuntimeError maximum recursion depth exceeded ... Under CVS, it's very different. Under a debug build, I get no output *at all*(!). Not even if I change the loop to start at 0. Under a release build: ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 138 exceptions.RuntimeError maximum recursion depth exceeded 139 exceptions.KeyError 300 140 exceptions.RuntimeError maximum recursion depth exceeded 141 exceptions.RuntimeError maximum recursion depth exceeded 142 exceptions.KeyError 299 143 exceptions.RuntimeError maximum recursion depth exceeded 144 exceptions.KeyError 297 145 exceptions.KeyError 296 146 exceptions.RuntimeError maximum recursion depth exceeded 147 exceptions.KeyError 295 148 exceptions.RuntimeError maximum recursion depth exceeded 149 exceptions.KeyError 294 150 exceptions.RuntimeError maximum recursion depth exceeded 151 exceptions.RuntimeError maximum recursion depth exceeded 152 exceptions.RuntimeError maximum recursion depth exceeded 153 exceptions.KeyError 309 154 exceptions.RuntimeError maximum recursion depth exceeded 155 exceptions.RuntimeError maximum recursion depth exceeded 156 exceptions.KeyError 307 157 exceptions.RuntimeError maximum recursion depth exceeded 158 exceptions.KeyError 306 159 exceptions.RuntimeError maximum recursion depth exceeded 160 exceptions.KeyError 305 161 exceptions.RuntimeError maximum recursion depth exceeded 162 exceptions.KeyError 304 163 exceptions.RuntimeError maximum recursion depth exceeded 164 exceptions.KeyError 303 165 exceptions.KeyError 302 166 exceptions.RuntimeError maximum recursion depth exceeded ... and there's a seemingly non-random but hard-to-fathom jumble of recursion-depth and KeyError exceptions thereafter too. So something's really hosed in CVS, or in MSVC 7.1, or ...

[Raymond Hettinger]
Thank you for trying this stuff! You already ruled out my #1 guess, namely that the "catch an exception from alloca()" trick we use on Windows to detect impending stack overflow stopped working in 7.1. But if it sucks under 6.0 too ... Another possibility is that it's something new & evil in the CVS compiler package (when my driver does "import compiler" under 2.3.4, of course it picks up 2.3.4's compiler package). Do you also get no output at all under a 6.0 debug build? Has anyone on Linux tried this yet?

Raymond Hettinger wrote:
Thank you Raymond! You just solved my problem with the totally psychotic behaviour I was getting after building with the free toolkit. I must have had some broken .pyc's lying around, and they were sending ceval.c insane. . . Now maybe I can start being useful around here :) Cheers, Nick. -- Nick Coghlan | Eugene, Oregon Email: ncoghlan@email.com | USA

Well, this gets nasty. In a debug build, and starting the loop at 0, I can't get off the gound in the MS 7.1 debugger. It dies quickly with an access violation in the bowels of ntdll.dll, and I don't have source for that. PyOS_CheckStack on Windows does this to detect stack overflow (it catches an MS exception if the C runtime can't allocate enough room on the stack): alloca(PYOS_STACK_MARGIN * sizeof(void*)); It's trying to see whether there's still room for 2K pointers on the stack. If I multiply that by 2, or by 3, nothing changes. But if I multiply it by 4, everything changes. Then the "oops! we're gonna blow the stack!" exit from PyOS_CheckStack is taken. It returns 1 to _Py_CheckRecursiveCall, which sets a "stack overflow" MemoryError and returns -1 to its caller. That's if (Py_EnterRecursiveCall(" in cmp")) return NULL; in PyObject_RichCompare. That's just trying to compare two ints. So NULL gets returned to PyObject_RichCompareBool, which in turn returns -1 to lookdict. AAAARGHGH! lookdict "isn't allowed" to raise exceptions, so it does a PyErr_Clear(), goes on to futilely chase the entire dict looking for another match on the hash code, and we've effectively turned a MemoryError into a KeyError. I expect that explains a lot about what we see in the release-build runs. If I multiply the stack check by 20, I can finally get some results out of the debug build: 0 exceptions.KeyError 299 1 exceptions.MemoryError Stack overflow 2 exceptions.MemoryError Stack overflow 3 exceptions.MemoryError Stack overflow 4 exceptions.KeyError 295 5 exceptions.MemoryError Stack overflow 6 exceptions.KeyError 294 7 exceptions.MemoryError Stack overflow 8 exceptions.MemoryError Stack overflow 9 exceptions.MemoryError Stack overflow 10 exceptions.MemoryError Stack overflow 11 exceptions.MemoryError Stack overflow 12 exceptions.MemoryError Stack overflow 13 exceptions.MemoryError Stack overflow 14 exceptions.KeyError 309 15 exceptions.KeyError 296 ... So we're blowing the C stack left and right in this test case, and sometimes dict lookup turns that into a KeyError. The question is what we did since 2.3.4 that apparently increases our stack demands, and grossly increases them in a debug build(!). Could be that the compile package is more heavily recursive now too (no idea). test_parser.py in 2,3.4 contained the same deeply nested tuples, so that's not what changed. Back in a release build, and restoring the original Windows stack-check code, but leaving the driver loop starting at 0, I have to sys.setrecursionlimit(16) to avoid getting any KeyErrors. sys.setrecursionlimit(878) is the minimum that allows at least one "ok" to show up: 0 ok 1 exceptions.RuntimeError maximum recursion depth exceeded 2 exceptions.RuntimeError maximum recursion depth exceeded 3 exceptions.RuntimeError maximum recursion depth exceeded 4 exceptions.KeyError 307 5 exceptions.RuntimeError maximum recursion depth exceeded 6 exceptions.KeyError 306 7 exceptions.RuntimeError maximum recursion depth exceeded 8 exceptions.KeyError 305 ... Misc/find_recursionlimit.py in CVS manages to print Limit of 1000 is fine before it craps out in a release build; in a debug build, it doesn't produce *any* output. If I change the limit it starts with to 100, it manages to get to Limit of 400 is fine in a debug build before stopping without a clue. Hmm! But over in 2.3.4 build, a release build also stopped with 1000, and a debug build also exited mysteriously. But after reducing its starting point to 100, it got to Limit of 700 is fine before crapping out. BTW, in 2.3.4 and CVS, when a debug run craps out mysteriously like this, it has an exit code of 128. That's scary: http://support.microsoft.com/support/kb/articles/q184/8/02.asp

Tim Peters <tim.peters@gmail.com> writes:
Well, this gets nasty.
You're certainly not wrong about that! [snip Windows details]
This part happens on Linux, too, it seems. Running your script: 114 ok 115 ok 116 ok 117 ok 118 ok 119 ok 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 This is a debug build, so we're not abusing the stack quite so much on linux as apparently seems to happen in debug builds on windows. This isn't going to be something we can pin on the new Windows tools, is it? It doesn't seem like a feature that a debug build gets a tiny stack, but, well...
Hmm, maybe that's a lie, I don't see the crazy KeyErrors with 2.3.
Could be that the compile package is more heavily recursive now too (no idea).
Really don't think so.
test_parser.py in 2,3.4 contained the same deeply nested tuples, so that's not what changed.
Something that *has* changed, however, is that we lost the code that compared objects recursively and now PyObject_RichCompare uses the more general Py_EnterRecursiveCall mechanism and not its own counter. So now we're grossly more likely to hit one of these faux KeyErrors in 2.3 than 2.4 (you'd need to invoke a Python __eq__ method or so). Would it make more sense for PyObject_RichCompare to still use a private counter but raise RuntimeError if it is exceeded instead of trying to be clever? That would make this problem /less/ likely, though obviously still possible (functions that claim they 'can't fail' are lying). Running Misc/find_recursionlimit.py with a debug build yields ... well, I gave up and killed it at 22000 (hmm, seems the default 'ulimit -s' is "unlimited"... I'm sure this hasn't always had this effect with a threaded build). Cheers, mwh --

FYI, "the stack" under an MS compiler defaults to 1MB. It's easy to experiment with different stack allocations from the cmdline, like this example: editbin /stack:2000000 python.exe The ~2MB there is enough that test_compiler runs to completion normally under a release-build Python, and under a debug-build Python. In a debug-build Windows Python, each MB allocated to the stack allows about 400 recursion levels (according to Misc/find_recursionlimit.py), and running compiler.compile() directly on test_parser.py requires more than 800 levels, so a 2MB stack must be near the insanity limit for the debug build on this test. So the easiest way to get the -uall test suite running on Windows again is to fiddle linker flags to boost the stack size. I'm not sure real apps need it. If they do and we don't "fix it", what happens is that the process mysteriously just vanishes (no error msg, nothing in the system or application event logs either) with a 128 exit code. That's a disaster. Alas, I don't have a theory for how we could be managing to screw up the OS so badly -- but then I guess you really can't spell Windows XP without Windows <0.9 wink>.

Another datapoint: the problem existed prior to 1/15/2004. That eliminates many possibilities. Raymond

"Tim" == Tim Peters <tim.peters@gmail.com> writes:
Tim> I guess you really can't spell Windows XP without Windows Tim> <0.9 wink>. Gee, and on your consistent recommendation, I was just about to upgrade from Linux and Mac OS X!<wink> -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Here's some puzzling evidence (WinXP Pro SP1). The test driver, for convenience: """ import compiler, sys f = open('../Lib/test/test_parser.py') guts = f.read() f.close() def ouch(n): if n == 0: return compiler.compile(guts, "<string>", "exec") else: return ouch(n-1) for n in range(0, 50): try: ouch(n) msg = 'ok' except KeyboardInterrupt: raise except Exception, msg: msg = str(sys.exc_info()[0]) + ' ' + str(msg) print n, msg """ The loop starts at 0 this time, and I'm only looking at a debug-build Python. The stack size for an .exe can be specified to 4-byte granularity. Here's the largest stack size at which that produces no output at all: C:\Code\python\PCbuild>editbin /stack:1310720 python_d.exe Microsoft (R) COFF Binary File Editor Version 6.00.8447 Copyright (C) Microsoft Corp 1992-1998. All rights reserved. C:\Code\python\PCbuild>python_d temp.py C:\Code\python\PCbuild>echo %ERRORLEVEL% 128 Add 4 measly bytes, and it's a world of difference: C:\Code\python\PCbuild>editbin /stack:1310724 python_d.exe Microsoft (R) COFF Binary File Editor Version 6.00.8447 Copyright (C) Microsoft Corp 1992-1998. All rights reserved. C:\Code\python\PCbuild>python_d temp.py 0 ok 1 ok ... 41 ok 42 ok C:\Code\python\PCbuild>echo %ERRORLEVEL% 128 So it still vanishes early, but gets thru the entire compiler.compile() business + 42 additional stacked Python calls! That's awfully impressive for four bytes. That suggests "the problem" isn't in detecting Python-level recursion. Under the debugger, it dies with an access violation at that point. Alas, the C-level Python call stack is no longer anywhere in evidence then. Instead it's 7 levels deep in ntdll.dll, and is "in the middle" of four consecutive Pentium PUSH instructions. That's evidence that the stack has been blown <wink>. The debugger Output window does show ntdll.dll suffering a Stack Overflow exception. It then shows a variety of Access Violations in ntdll.dll trying to read and write various locations, presumably in a failing attempt to report the stack overflow.

Tim Peters <tim.peters@gmail.com> writes:
Has the failure mode of alloca() changed? I take it you're building with VC++ 7.1? What happens for a VC++ 6 build? Hmm, a moment with msdn suggests that there's been no significant changes here, although the documentation is for _alloca(), and Python calls alloca(). That can't make any difference, can it? It still smells like a tool change to me. Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

[Michael Hudson <mwh@python.net]
Has the failure mode of alloca() changed?
No, and the Windows stack-check code works fine. I showed results before from boosting the amount of padding the Windows stack-check code checks for, and it if checks for 20x more (which is ridiculously large) padding than it checks for now, it reliably generates Python stack-overflow MemoryErrors. Indeed, the KeyError exceptions were traced specifically to this: a stack-overflow MemoryError, due to the Windows stack-check code, getting wiped out by lookdict (whose caller took lookdict's NULL return as meaning the key wasn't present in the dict -- although it actually was).
I take it you're building with VC++ 7.1?
Right.
What happens for a VC++ 6 build?
Raymond reported on that earlier. Appeared to be the same as I saw in a release build. He didn't report on a debug build. He's running WinME, so a "hard" failure may look quite different for him.
Right, no difference.
It still smells like a tool change to me.
Not to me. Although it does smell.

More clues. As expected, commenting out the tests in test_parser.py that contain deeply nested tuples makes the problems go away. A surprise: adding code to every function in parsermodule.c to do Py_FatalError() if the Windows stack-check fails doesn't make any difference. So it's not C recursion in parsermodule.c that causes this (although I bet it *could*, it simply doesn't in this test case). Not a surprise: Grzegorz Makarewicz's patch to call the stack-check code in pymalloc catches the problem "in time". But we could put the same patch anywhere in Python that gets called frequently, and catch it in time. That's too expensive to bear. The Windows stack-check code just isn't called often enough, and the recursion limit is "too large" now to live without a reliable stack check. Py_EnterRecursiveCall() in fact rarely calls the Windows stack-check code. At the time the Windows stack-check code first *would* have complained (had it been called then), the tstate recursion depth was 694, and _Py_CheckRecursionLimit was 973. The funky way this code works, that means we can do ~150 more levels of Python call before the Windows stack-check code is called again, and that's too late. We only checked that there's still room for 2K pointers on the stack, and 150 Python-level calls means about 450 C-level calls in this test: the stack at this point is hundreds (& hundreds) of repetitions of PyEval_EvalFrame -> fast_function -> call_function -> PyEval_Frame -> etc etc etc and PyEval_EvalFrame C frames are fat enough on their own for 150 of them to blow what remains of the stack. The easiest workaround for Windows is still to boost the stack size. I'm going to do that if nobody screams. Looks like nobody has an explanation yet for why 2.3.4 consistently yielded MemoryError but 2.4a2 mixes those with spurious KeyError and SyntaxError exceptions. That could be coincidence in this specific test driver, though -- lookdict has always wiped out exceptions. where's-stackless-when-you-need-it-ly y'rs - tim

Tim Peters <tim.peters@gmail.com> wrote:
[...]
The easiest workaround for Windows is still to boost the stack size. I'm going to do that if nobody screams.
Would reducing the recursion limit (for Windows) be a reasonable approach? Charles P.S. Please don't cc: me on list messages. -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

[Charles Cazabon]
Would reducing the recursion limit (for Windows) be a reasonable approach?
Not if that means Windows can't run the standard test suite. Boosting the stack size on Windows is a VM reservation operation, BTW -- it doesn't actually increase the RAM needed, unless the stack actually needs to grow that big.

[Tim]
[Armin]
The reason is that Py_EnterRecursiveCall() was only introduced in 2.4. Comparisons didn't throw RuntimeErrors that easily in 2.3.
Doh! Of course. Recursive compares aren't implicated in the test programs that failed here. Under 2.3.4, all the compares return normally because they're not stack-checking at all, and the program "gets to" recurse deeper then, until a "recursion depth exceeded" exception gets thrown. But in 2.4 it's a crap shoot whether a comparison or a recursive Python call notices first that we're nearing the end of the stack. The recursive Python calls make real stack demands in these tests, but the comparisons are just int-vs-int and string-vs-string, so the stack-check in compare now is "almost always" a nuisance check in these tests. Sounds like a good idea not to run out to stack <wink>.

Hello, On Thu, Aug 12, 2004 at 03:41:37PM -0400, Tim Peters wrote:
Sounds like a good idea not to run out to stack <wink>.
I don't remember if it was mentioned here, but maybe we'd better check directly whether the C stack is too large rather than (or possibly in addition to) using a limit on the number of Python iterations. This is not really ANSI C, but I can't think of a setting in which the following trick would fail to be a good indication of the amount of stack space consumed: save a pointer to a local variable "early", e.g. in Py_Initialize(); then in any other function call, the distance between this pointer and a pointer to some local equals the amount of stack space consumed by Python minus a few bytes. If this sounds too much of a hack, the (usual) recursion limit could be kept to limit nested Python frames; but all C-level recursions (including eval_frame) could additionally use the above trick. Its advantage is that it is an extremely fast check. If the total stack size is difficult to know in advance, we can still use PyOS_CheckStack(), but more efficiently and less randomly than now, by maintaining a "high tide" pointer that records how much stack we are sure we have, and calling PyOS_CheckStack() only to attempt to push the "high tide" mark further. While I'm in a hacking mood, there might be a way to prevent PyErr_Clear() to clear away "asynchronuous" exceptions like RuntimeError, MemoryError, and KeyboardInterrupt: for these exceptions, let's just store them away instead of clearing them, and re-raise them at the next occasion. The "next occasion" would have to be defined more precisely, but there is probably a way to ensure that it will at least be before the next non-trivial opcode operation starts. It would overwrite an exception set later, like those spurious KeyError we get for dict lookups. It might be a better-than-nothing quick fix to all these PyErr_Clear() all around the code base. A bientôt, Armin.

[Armin Rigo]
How large is too large? The Windows way checks the stack for impending overflow directly. A long "stack check on Unix" thread here in 2000 fizzled out in the expected way: no two Unices appear to have anything in common <0.9 wink>.
It fails for threads, whose stacks may be anywhere in relation to the main thread's. So any gimmick like this has to have a thread-local "starting point". If "a stack" is always a contiguous slice of memory, then it can work.
If this sounds too much of a hack,
No, but I'm not sure how much additional hackery it requires to make it work in all cases.
The ways to do it that don't work are indeed extremely fast <wink>.
If the total stack size is difficult to know in advance,
That's apparently the case across platforms -- and someimes the main-thread stack is much larger than secondary-thread stacks. The Windows check doesn't even try to know the current stack's size, but relies on MS-specific C exception extensions.
Threads again complicate this. AFAICT, the only platform that defines USE_STACKCHECK now (and actually implements PyOS_CheckStack()) is 32-bit Windows using an MS compiler. I did boost the stack reservation for that platform in CVS, which can't solve it, but at least hides it better <wink>.
Threads probably complicate that too. It's dreadful that serious problems can get transformed into bogus KeyErrors (in the test driver I posted, I'm not sure I spelled this out, but the key always *was* in the dict when a MemoryError got turned into a KeyError; we called a comparison function to begin with because the hash codes matched, and since these were mostly small-integer keys, they were identical to their hash codes -- so these keys were in the dicts, and the KeyErrors were lying about that). That's just nuts. I don't have spare cycles to give to it, though, so you'll have to save the world on your own again.

Hello Tim, On Sun, Aug 15, 2004 at 10:50:25PM -0400, Tim Peters wrote:
Yes. Here is a patch attempting to do what I described: http://www.python.org/sf/1009929 It's an extension of the asynchronous exception mecanism used to signal between threads. PyErr_Clear() can send some exceptions to its own thread using this mecanism. (So it is thread-safe.) Armin

On Aug 16, 2004, at 7:29 AM, Armin Rigo wrote:
Yes. Here is a patch attempting to do what I described: http://www.python.org/sf/1009929
From the patch description:
It seems to me that something similar to what Java has would be a good idea. Namely, a new top level exception (from which all others would derive) called "Raisable", analogous to Java's Throwable. This then has two subclasses: "Exception", and "FatalError". I'm not sure FatalError is a good name, but *some* new name needs to be thought up for Java's "Error" class, because lots of python exceptions end in "Error" but belong under the "Exception" hierarchy (for example "KeyError"). The criteria for whether a given exception should go under "Exception" or "FatalError" is whether users' code should generally catch the exception. Thus, at least "SystemExit", "KeyboardInterrupt", and "MemoryError" should go under "FatalError". Catching those is nearly never what you wanted to do when you write "except Exception:". There's likely more -- I have not gone through all the exceptions in Python to classify them. One issue is that creating a new category of Exceptions doesn't help people who do "except:" instead of "except Exception:". It is unlikely that person meant to catch things like MemoryError, rather, they were just being lazy. I suppose that syntax could be deprecated, at least in example code and documentation, in favor of "except Exception" for the usual case, and "except Raisable" for the cases where you do actually want to catch everything*. James * Except, of course, old string exceptions which have been deprecated for ages.

On Aug 16, 2004, at 8:07 PM, James Y Knight wrote:
* basestr could inherit from "Raisable" ;) The big problem with "Raisable" is that both raiseable and raisable seem to be correct spellings, and I don't think either are in many abridged dictionaries (the OS X spell checker doesn't like either, for example). -bob

On Mon, Aug 16, 2004, James Y Knight wrote:
We've already got StandardError; I think it makes more sense to rearrange the exception hierarchy a bit to support your suggestion rather than creating a whole new base class. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com

Hm, Java needs the distinction because some exceptions must be declared and others mustn't. But Python doesn't have that distinction. I'm not sure that you can always treat the same set of exceptions as fatal. E.g. in many situations, AttributeError, TypeError and NameError are all indicative of programming bugs (typically typos), but in other contexts these are recoverable. So rather than giving an arbitrary definition of fatality, let's refrain from defining the concept.
Calling SystemExit and KeyboardInterrupt fatal strikes me as particularly odd, as I routinely catch these.
--Guido van Rossum (home page: http://www.python.org/~guido/)

On Aug 16, 2004, at 9:32 PM, Guido van Rossum wrote:
Well, actually, java has three categories of exceptions: 1) Error - not necessary to declare as thrown, only *serious* errors. Contains: a) things that can "never happen" with a program compiled by the java compiler (no such field, unknown class file format, etc), b) machine errors (out of memory, stack overflow, etc) c) assert() failure d) ThreadDeath (similar to a KeyboardInterrupt/SystemExit) 2) Exception - normal run of the mill exception, needs to be declared as thrown. 3) RuntimeException - subclass of Exception, does not need to be declared as thrown. (e.g. IndexOutOfBoundsException, NoSuchFieldException, ClassCastException) The distinction you refer to above is really the difference between Exception and RuntimeException. Translated to java, AttributeError, TypeError and NameError would be RuntimeExceptions. So, I agree with you - I don't believe python needs that distinction. I do believe python needs the distinction between Exception and Error.
Calling SystemExit and KeyboardInterrupt fatal strikes me as particularly odd, as I routinely catch these.
I'll agree: I don't think the name "FatalError" is particularly great. However, I hope it gets the idea across better than "XXXErrorXXXRenameMeXXX" which was my other choice of name. ;) I do think the categorization is correct. While you may sometimes catch KeyboardInterrupt/SystemExit, most of the time you really do not want to, even if you are catching "everything". If you do want to catch KeyboardInterrupt, you are also likely to be catching it explicitly by its name, anyhow. James

[Armin Rigo]
I'm sorry that I haven't had time to look at this. But since I didn't and don't, let's try to complicate it <wink>. Some exceptions should never be suppressed unless named explicitly, and a real bitch is that some user-defined exceptions can fit in that category too. The ones that give me (and my employer) the most grief are the tree of exceptions deriving from ZODB's ConflictError. ConflictError is a serious thing: it essentially means the current transaction cannot succeed, and the app should give up (and maybe retry the current transaction from its start). Suppressing ConflictError by accident-- even inside a hasattr() call! --can grossly reduce efficiency, and has a long history too of provoking subtle, catastrophic, database corruption bugs. I would like to see Python's exception hierarchy grow more sophisticated in this respect. MemoryError, SystemExit, and KeyboardInterrupt are things that should not be caught by "except Exception:", neither by a bare "except:", nor by hasattr() or C-level dict lookup. ZODB's ConflictError is another of that ilk. I'd like to see "except Exception:" become synonymous with bare "except:", and move the "dangerous exceptions" to subclass off a new branch of the exception hierarchy. It could be that something like your patch is the only practical way to make this work in the C implementation, so I'm keen on it.

In article <1f7befae04090422024afaee58@mail.gmail.com>, Tim Peters <tim.peters@gmail.com> wrote:
It's not really the same subject, but the exception that gives me the most grief is StopIteration. I have to keep remembering to never call .next() without catching it; if I forget, I get bugs where some loop several levels back in the call tree mysteriously exits. -- David Eppstein Computer Science Dept., Univ. of California, Irvine http://www.ics.uci.edu/~eppstein/

Are you sure? This sounds like superstition to me, since that's not how loops work. Raising StopIteration in the middle of a loop does not break out of the loop -- only raising StopIteration from a next() breaks a loop. Or are you talking about nested next() calls? That's the only case where the behavior you are citing occurs. -- --Guido van Rossum (home page: http://www.python.org/~guido/) Ask me about gmail.

In article <ca471dc2040906184648d95e55@mail.gmail.com>, Guido van Rossum <gvanrossum@gmail.com> wrote:
I don't remember, it could have been nested next()s. -- David Eppstein Computer Science Dept., Univ. of California, Irvine http://www.ics.uci.edu/~eppstein/

On Sun, 5 Sep 2004 01:02:35 -0400, Tim Peters <tim.peters@gmail.com> wrote:
The current exception hierarchy isn't too far from what you suggest. We just got the names wrong. That is, there is a base class, StandardException, that captures most exceptions other than MemoryError, SystemError, and KeyboardInterrupt. If we renamed that Exception, then we'd be 90% of the way there. You could also change your code, right now, to say "except StandardError:" and avoid the problem entirely. Make sure ConflictError does not inherit from StandardError, of course. And make sure you're happy that ImportError is not a StandardError either. I'm not sure what I think of the change to "except:" It's often the case that someone who has written "except:" really means "except Something:", but I expect that very often Something != StandardError and issubclass(Something, StandardError). In that case, the change doesn't really help them. The code is still wrong. Jeremy

On Mon, 6 Sep 2004 09:44:12 +0200, Fredrik Lundh <fredrik@pythonware.com> wrote:
I misread the very long output of pydoc exceptions :-). I don't understand the hierarchy, either, or I would have noticed that the results don't make sense. Why isn't StopIteration a StandardError? I'll second Tim's suggestion that some errors -- like SystemError, MemoryError, and KeyboardInterrupt belong in a different category. I think it would be easier in principle to put them at a different place in the class hierarchy than to make them some special kind of uncatchable exception. Jeremy

Jeremy Hylton wrote: ...
Note that we don't want uncatchable exceptions. Rather, we want exceptions that aren't caught by bare excepts or very broad excepts. In many cases, we want certain knowledgeable code to be able to catch these exceptions. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Tue, 07 Sep 2004 09:43:53 -0400, Jim Fulton <jim@zope.com> wrote:
I agree with half the cause. There ought to be a decent organization of the exception class hierarchy so that exceptions like KeyboardInterrupt are in a special category. Then an "except NormalError:" <wink> would catch only the normal errors and not the special ones. I don't think bare exception should change it's meaning; you just shouldn't use it unless it's *really* what you mean. I think backwards compatibility is a really hard issue for any of these changes. It's probably hard to re-arrange the class hierarchy, but I don't know what practical solution there is to these problems that doesn't involve breaking some code. It's even harder to change bare except, but I don't think that's necessary. Jeremy

On Tue, 2004-09-07 at 09:43, Jim Fulton wrote:
I don't agree about having exceptions that pass bare excepts. A typical /valid/ use of bare excepts are in frameworks such as transaction processing, where you need to do some extra work when an exception occurs, then re-raise the original exception, e.g.: try: do_something() except: database.rollback() raise else: database.commit() Even exceptions like SystemError, MemoryError, or KeyboardInterrupt want to adhere to this simple idiom. Bare except should continue to catch all exceptions. Code that wanted to do otherwise should /not/ use a bare except, and +1 on some form of exception hierarchy restructuring that would make that clearer. -Barry

Barry Warsaw wrote:
Fair enough. I also agree with jeremy's points re backward compatability. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Barry Warsaw wrote:
My policy for bare excepts is that without significant justification they _must_ either re-raise the original exception or raise another exception. There are very few circumstances where I have allowed my team to write pure bare excepts. I haven't checked, but a warning for violations of this rule may be a nice addition to pychecker or pylint. -Kevin

On Tue, 2004-09-07 at 11:11, Kevin Jacobs wrote:
The other case I've seen are for command-shell like loops, where you might print the exception in the bare except, but not re-raise the exception. Think about the main interactive interpreter loop. But yeah I agree, you want strong justification for any use of bare except. -Barry

After a bit more thought (and it's hard to measure how little), I'd like to see "bare except" deprecated. That doesn't mean no way to catch all exceptions, it means being explicit about intent. Only a few of the bare excepts I've seen in my Python life did what was actually intended, and there's something off in the design when the easiest thing to say usually does a wrong thing. I think Java has a saner model in this particular respect: Throwable Exception Error Java's distinction between "checked" and "unchecked" exceptions is a distinct layer of complication on top of that. All exceptions derive from Throwable. A "catch" clause requires specifying a class (there's no "bare except"). "An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch". That includes AssertionError and VirtualMachineError. Those are exceptions that should never occur. It also includes ThreadDeath, which is expected to occur, but The class ThreadDeath is specifically a subclass of Error rather than Exception, even though it is a "normal occurrence", because many applications catch all occurrences of Exception and then discard the exception. and it's necessary for ThreadDeath to reach the top level else the thread never really dies. In that respect, it's interesting that SystemExit and KeyboardInterrupt are *intended* to "reach the top level" too, but can't be relied on to do so because of ubiquitous bare excepts and even pseudo-careful "except Exception:"s now. If people changed those to "except StandardError:", SystemExit would make it to the top but KeyboardInterrupt still wouldn't. Raisable Exception Stubborn ControlFlow KeyboardInterrupt StopIteration SystemExit MemoryError introduces a class of stubborn exceptions, those that wouldn't be caught by "except Exception:", and with the intent that there's no way you should get the effect of "except Raisable" without explictly saying just that (once bare except's deprecation is complete). Oh well. We should elect a benevolent dictator for Python!

Tim Peters wrote:
Giving it same amount of thought as Tim, I like this idea as well. I don't think the burden of having to always specify an exception to catch is that much work and would definitely be more explicit. Plus it is not hard to teach to newbies; just tell them to catch the exception all of the "safe" exceptions inherit from. Is it PEP time? We have now had the idea of reorganizing the exception hierarchy come up in this thread and in early August (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is... for the thread on reorganizing for the purpose of using new-style classes and the idea of having all raisable objects inherit from a specific base class). It seems like a serious enough of an idea that it will happen for Python 3000 and thus should be discussed; never too early to start. But maybe it at least deserves to be mentioned in PEP 3000 (Guido?)?
Oh well. We should elect a benevolent dictator for Python!
Here, here! =) -Brett

Tim Peters wrote:
I have copied stack check into PyObject_Malloc and after X calls to PyEval_EvalFrame, fast_function python dies in malloc(msvcrt) and HeapAlloc (ntdll) as described by Tim. After this patch most ocurring exception is MemoryError, but for 141, 171, 201 and 231 python raises SyntaxError - always witout traceback :( mak relevant part from obmalloc.c: PyObject_Malloc(size_t nbytes) { block *bp; poolp pool; poolp next; uint size; #ifdef USE_STACKCHECK if (PyOS_CheckStack()) { return NULL; } #endif

This suggests to me that the very concept of a function that's not allowed to return errors is dubious, since even just calling another function could provoke a MemoryError. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[Tim Peters]
[Greg Ewing]
I believe I had a more elegant way to say that: AAAARGHGH! <wink> Alas, 10 billion call sites, including in external extensions, rely on that a NULL return from PyDict_GetItem() or PyDict_GetItemString() is the unexceptional "nope, the key's not there". So there's no realistic chance of repairing this before Python 3.0 (if even then). In the meantime, KeyError is confusable with almost all exceptions. For some real fun, ask on a Zope list about hasattr() suppressing all exceptions. Not that Guido ever expected a hasttr() call to try to mess with persistent objects across a networked database as side effect. Python originally had only string-keyed dicts, and the basic dict API still reflects that a dict lookup simply "can't fail" for any reason other than "key ain't there". That used to be true. For that matter, hasattr() was originally no worse than a string-keyed dict lookup too, so "can't fail" also made good sense there at the time.

On Mon, 9 Aug 2004, Tim Peters wrote: [...]
Has anyone on Linux tried this yet?
On Linux, I get precisely the same output from Python CVS that you posted, at least as far as 148 (didn't check any further): [...] 118 ok 119 ok 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 138 exceptions.RuntimeError maximum recursion depth exceeded 139 exceptions.KeyError 300 140 exceptions.RuntimeError maximum recursion depth exceeded 141 exceptions.RuntimeError maximum recursion depth exceeded 142 exceptions.KeyError 299 143 exceptions.RuntimeError maximum recursion depth exceeded 144 exceptions.KeyError 297 145 exceptions.KeyError 296 146 exceptions.RuntimeError maximum recursion depth exceeded 147 exceptions.KeyError 295 148 exceptions.RuntimeError maximum recursion depth exceeded [...] John

Hello Tim, On Mon, Aug 09, 2004 at 09:48:37PM -0400, Tim Peters wrote:
Has anyone on Linux tried this yet?
I get this: in a release build, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.RuntimeError maximum recursion depth exceeded 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.RuntimeError maximum recursion depth exceeded ... In a release 21st of March version with the fresh generator expression patch applied, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 305 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 304 129 exceptions.RuntimeError maximum recursion depth exceeded ... In a recent debug build, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded ... So there is no particular problem running debug builds under Linux, but still why we get this behavior is anybody's guess. Also note that even though the compiler package may only have changed a bit, the parser C module definitely has, e.g. when genexprs and decorators were introduced. I wonder what effect this has on the depth of concrete syntax trees. I also wonder if the parser module tries to be safe again C stack overflow when building hugely nested data structures. Armin

Hello again, On Tue, Aug 10, 2004 at 08:16:38PM +0100, Armin Rigo wrote:
Sorry, my mistake. This was after I removed from object.c the two Py_Enter/LeaveRecursiveCall(" in cmp"). Not suprizingly, I then only get "regular" recursion errors. A pure CVS version gets me the same results as you and John. Armin

On Tuesday 10 August 2004 03:16 pm, Armin Rigo wrote:
syntax trees. I also wonder if the parser module tries to be safe again C stack overflow when building hugely nested data structures.
No, it really doesn't, though it probably should be a little careful. Switching to a less recursive approach may be a good idea, but not sure what would cause the most pain in implementation. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On Tue, 10 Aug 2004 20:16:38 +0100, Armin Rigo <arigo@tunes.org> wrote:
I don't think the compiler package has changed much at all, but it consumes the deeply nested concrete syntax trees created by the parser module. The compiler.transformer module uses recursion a lot. Several releases ago I added some hacks to reduce the number of frames (basically did tail call optimization by hand in a few spots). It may be that those hacks are no longer sufficient. where's-that-ast-branch-when-you-need-it-ly y'rs, Jeremy

Here's a cute one: """ import compiler, sys f = open('../Lib/test/test_parser.py') guts = f.read() f.close() def ouch(n): if n == 0: return compiler.compile(guts, "<string>", "exec") else: return ouch(n-1) for n in range(100, 250): try: ouch(n) msg = 'ok' except Exception, msg: msg = str(sys.exc_info()[0]) + ' ' + str(msg) print n, msg """ Under 2.3.4, that works as expected: when n hit a large enough value, from then on it was all repetitions of recursion-depth exceptions: ,,, 147 ok 148 ok 149 ok 150 exceptions.RuntimeError maximum recursion depth exceeded 151 exceptions.RuntimeError maximum recursion depth exceeded 152 exceptions.RuntimeError maximum recursion depth exceeded ... Under CVS, it's very different. Under a debug build, I get no output *at all*(!). Not even if I change the loop to start at 0. Under a release build: ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 138 exceptions.RuntimeError maximum recursion depth exceeded 139 exceptions.KeyError 300 140 exceptions.RuntimeError maximum recursion depth exceeded 141 exceptions.RuntimeError maximum recursion depth exceeded 142 exceptions.KeyError 299 143 exceptions.RuntimeError maximum recursion depth exceeded 144 exceptions.KeyError 297 145 exceptions.KeyError 296 146 exceptions.RuntimeError maximum recursion depth exceeded 147 exceptions.KeyError 295 148 exceptions.RuntimeError maximum recursion depth exceeded 149 exceptions.KeyError 294 150 exceptions.RuntimeError maximum recursion depth exceeded 151 exceptions.RuntimeError maximum recursion depth exceeded 152 exceptions.RuntimeError maximum recursion depth exceeded 153 exceptions.KeyError 309 154 exceptions.RuntimeError maximum recursion depth exceeded 155 exceptions.RuntimeError maximum recursion depth exceeded 156 exceptions.KeyError 307 157 exceptions.RuntimeError maximum recursion depth exceeded 158 exceptions.KeyError 306 159 exceptions.RuntimeError maximum recursion depth exceeded 160 exceptions.KeyError 305 161 exceptions.RuntimeError maximum recursion depth exceeded 162 exceptions.KeyError 304 163 exceptions.RuntimeError maximum recursion depth exceeded 164 exceptions.KeyError 303 165 exceptions.KeyError 302 166 exceptions.RuntimeError maximum recursion depth exceeded ... and there's a seemingly non-random but hard-to-fathom jumble of recursion-depth and KeyError exceptions thereafter too. So something's really hosed in CVS, or in MSVC 7.1, or ...

[Raymond Hettinger]
Thank you for trying this stuff! You already ruled out my #1 guess, namely that the "catch an exception from alloca()" trick we use on Windows to detect impending stack overflow stopped working in 7.1. But if it sucks under 6.0 too ... Another possibility is that it's something new & evil in the CVS compiler package (when my driver does "import compiler" under 2.3.4, of course it picks up 2.3.4's compiler package). Do you also get no output at all under a 6.0 debug build? Has anyone on Linux tried this yet?

Raymond Hettinger wrote:
Thank you Raymond! You just solved my problem with the totally psychotic behaviour I was getting after building with the free toolkit. I must have had some broken .pyc's lying around, and they were sending ceval.c insane. . . Now maybe I can start being useful around here :) Cheers, Nick. -- Nick Coghlan | Eugene, Oregon Email: ncoghlan@email.com | USA

Well, this gets nasty. In a debug build, and starting the loop at 0, I can't get off the gound in the MS 7.1 debugger. It dies quickly with an access violation in the bowels of ntdll.dll, and I don't have source for that. PyOS_CheckStack on Windows does this to detect stack overflow (it catches an MS exception if the C runtime can't allocate enough room on the stack): alloca(PYOS_STACK_MARGIN * sizeof(void*)); It's trying to see whether there's still room for 2K pointers on the stack. If I multiply that by 2, or by 3, nothing changes. But if I multiply it by 4, everything changes. Then the "oops! we're gonna blow the stack!" exit from PyOS_CheckStack is taken. It returns 1 to _Py_CheckRecursiveCall, which sets a "stack overflow" MemoryError and returns -1 to its caller. That's if (Py_EnterRecursiveCall(" in cmp")) return NULL; in PyObject_RichCompare. That's just trying to compare two ints. So NULL gets returned to PyObject_RichCompareBool, which in turn returns -1 to lookdict. AAAARGHGH! lookdict "isn't allowed" to raise exceptions, so it does a PyErr_Clear(), goes on to futilely chase the entire dict looking for another match on the hash code, and we've effectively turned a MemoryError into a KeyError. I expect that explains a lot about what we see in the release-build runs. If I multiply the stack check by 20, I can finally get some results out of the debug build: 0 exceptions.KeyError 299 1 exceptions.MemoryError Stack overflow 2 exceptions.MemoryError Stack overflow 3 exceptions.MemoryError Stack overflow 4 exceptions.KeyError 295 5 exceptions.MemoryError Stack overflow 6 exceptions.KeyError 294 7 exceptions.MemoryError Stack overflow 8 exceptions.MemoryError Stack overflow 9 exceptions.MemoryError Stack overflow 10 exceptions.MemoryError Stack overflow 11 exceptions.MemoryError Stack overflow 12 exceptions.MemoryError Stack overflow 13 exceptions.MemoryError Stack overflow 14 exceptions.KeyError 309 15 exceptions.KeyError 296 ... So we're blowing the C stack left and right in this test case, and sometimes dict lookup turns that into a KeyError. The question is what we did since 2.3.4 that apparently increases our stack demands, and grossly increases them in a debug build(!). Could be that the compile package is more heavily recursive now too (no idea). test_parser.py in 2,3.4 contained the same deeply nested tuples, so that's not what changed. Back in a release build, and restoring the original Windows stack-check code, but leaving the driver loop starting at 0, I have to sys.setrecursionlimit(16) to avoid getting any KeyErrors. sys.setrecursionlimit(878) is the minimum that allows at least one "ok" to show up: 0 ok 1 exceptions.RuntimeError maximum recursion depth exceeded 2 exceptions.RuntimeError maximum recursion depth exceeded 3 exceptions.RuntimeError maximum recursion depth exceeded 4 exceptions.KeyError 307 5 exceptions.RuntimeError maximum recursion depth exceeded 6 exceptions.KeyError 306 7 exceptions.RuntimeError maximum recursion depth exceeded 8 exceptions.KeyError 305 ... Misc/find_recursionlimit.py in CVS manages to print Limit of 1000 is fine before it craps out in a release build; in a debug build, it doesn't produce *any* output. If I change the limit it starts with to 100, it manages to get to Limit of 400 is fine in a debug build before stopping without a clue. Hmm! But over in 2.3.4 build, a release build also stopped with 1000, and a debug build also exited mysteriously. But after reducing its starting point to 100, it got to Limit of 700 is fine before crapping out. BTW, in 2.3.4 and CVS, when a debug run craps out mysteriously like this, it has an exit code of 128. That's scary: http://support.microsoft.com/support/kb/articles/q184/8/02.asp

Tim Peters <tim.peters@gmail.com> writes:
Well, this gets nasty.
You're certainly not wrong about that! [snip Windows details]
This part happens on Linux, too, it seems. Running your script: 114 ok 115 ok 116 ok 117 ok 118 ok 119 ok 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 This is a debug build, so we're not abusing the stack quite so much on linux as apparently seems to happen in debug builds on windows. This isn't going to be something we can pin on the new Windows tools, is it? It doesn't seem like a feature that a debug build gets a tiny stack, but, well...
Hmm, maybe that's a lie, I don't see the crazy KeyErrors with 2.3.
Could be that the compile package is more heavily recursive now too (no idea).
Really don't think so.
test_parser.py in 2,3.4 contained the same deeply nested tuples, so that's not what changed.
Something that *has* changed, however, is that we lost the code that compared objects recursively and now PyObject_RichCompare uses the more general Py_EnterRecursiveCall mechanism and not its own counter. So now we're grossly more likely to hit one of these faux KeyErrors in 2.3 than 2.4 (you'd need to invoke a Python __eq__ method or so). Would it make more sense for PyObject_RichCompare to still use a private counter but raise RuntimeError if it is exceeded instead of trying to be clever? That would make this problem /less/ likely, though obviously still possible (functions that claim they 'can't fail' are lying). Running Misc/find_recursionlimit.py with a debug build yields ... well, I gave up and killed it at 22000 (hmm, seems the default 'ulimit -s' is "unlimited"... I'm sure this hasn't always had this effect with a threaded build). Cheers, mwh --

FYI, "the stack" under an MS compiler defaults to 1MB. It's easy to experiment with different stack allocations from the cmdline, like this example: editbin /stack:2000000 python.exe The ~2MB there is enough that test_compiler runs to completion normally under a release-build Python, and under a debug-build Python. In a debug-build Windows Python, each MB allocated to the stack allows about 400 recursion levels (according to Misc/find_recursionlimit.py), and running compiler.compile() directly on test_parser.py requires more than 800 levels, so a 2MB stack must be near the insanity limit for the debug build on this test. So the easiest way to get the -uall test suite running on Windows again is to fiddle linker flags to boost the stack size. I'm not sure real apps need it. If they do and we don't "fix it", what happens is that the process mysteriously just vanishes (no error msg, nothing in the system or application event logs either) with a 128 exit code. That's a disaster. Alas, I don't have a theory for how we could be managing to screw up the OS so badly -- but then I guess you really can't spell Windows XP without Windows <0.9 wink>.

Another datapoint: the problem existed prior to 1/15/2004. That eliminates many possibilities. Raymond

"Tim" == Tim Peters <tim.peters@gmail.com> writes:
Tim> I guess you really can't spell Windows XP without Windows Tim> <0.9 wink>. Gee, and on your consistent recommendation, I was just about to upgrade from Linux and Mac OS X!<wink> -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Here's some puzzling evidence (WinXP Pro SP1). The test driver, for convenience: """ import compiler, sys f = open('../Lib/test/test_parser.py') guts = f.read() f.close() def ouch(n): if n == 0: return compiler.compile(guts, "<string>", "exec") else: return ouch(n-1) for n in range(0, 50): try: ouch(n) msg = 'ok' except KeyboardInterrupt: raise except Exception, msg: msg = str(sys.exc_info()[0]) + ' ' + str(msg) print n, msg """ The loop starts at 0 this time, and I'm only looking at a debug-build Python. The stack size for an .exe can be specified to 4-byte granularity. Here's the largest stack size at which that produces no output at all: C:\Code\python\PCbuild>editbin /stack:1310720 python_d.exe Microsoft (R) COFF Binary File Editor Version 6.00.8447 Copyright (C) Microsoft Corp 1992-1998. All rights reserved. C:\Code\python\PCbuild>python_d temp.py C:\Code\python\PCbuild>echo %ERRORLEVEL% 128 Add 4 measly bytes, and it's a world of difference: C:\Code\python\PCbuild>editbin /stack:1310724 python_d.exe Microsoft (R) COFF Binary File Editor Version 6.00.8447 Copyright (C) Microsoft Corp 1992-1998. All rights reserved. C:\Code\python\PCbuild>python_d temp.py 0 ok 1 ok ... 41 ok 42 ok C:\Code\python\PCbuild>echo %ERRORLEVEL% 128 So it still vanishes early, but gets thru the entire compiler.compile() business + 42 additional stacked Python calls! That's awfully impressive for four bytes. That suggests "the problem" isn't in detecting Python-level recursion. Under the debugger, it dies with an access violation at that point. Alas, the C-level Python call stack is no longer anywhere in evidence then. Instead it's 7 levels deep in ntdll.dll, and is "in the middle" of four consecutive Pentium PUSH instructions. That's evidence that the stack has been blown <wink>. The debugger Output window does show ntdll.dll suffering a Stack Overflow exception. It then shows a variety of Access Violations in ntdll.dll trying to read and write various locations, presumably in a failing attempt to report the stack overflow.

Tim Peters <tim.peters@gmail.com> writes:
Has the failure mode of alloca() changed? I take it you're building with VC++ 7.1? What happens for a VC++ 6 build? Hmm, a moment with msdn suggests that there's been no significant changes here, although the documentation is for _alloca(), and Python calls alloca(). That can't make any difference, can it? It still smells like a tool change to me. Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

[Michael Hudson <mwh@python.net]
Has the failure mode of alloca() changed?
No, and the Windows stack-check code works fine. I showed results before from boosting the amount of padding the Windows stack-check code checks for, and it if checks for 20x more (which is ridiculously large) padding than it checks for now, it reliably generates Python stack-overflow MemoryErrors. Indeed, the KeyError exceptions were traced specifically to this: a stack-overflow MemoryError, due to the Windows stack-check code, getting wiped out by lookdict (whose caller took lookdict's NULL return as meaning the key wasn't present in the dict -- although it actually was).
I take it you're building with VC++ 7.1?
Right.
What happens for a VC++ 6 build?
Raymond reported on that earlier. Appeared to be the same as I saw in a release build. He didn't report on a debug build. He's running WinME, so a "hard" failure may look quite different for him.
Right, no difference.
It still smells like a tool change to me.
Not to me. Although it does smell.

More clues. As expected, commenting out the tests in test_parser.py that contain deeply nested tuples makes the problems go away. A surprise: adding code to every function in parsermodule.c to do Py_FatalError() if the Windows stack-check fails doesn't make any difference. So it's not C recursion in parsermodule.c that causes this (although I bet it *could*, it simply doesn't in this test case). Not a surprise: Grzegorz Makarewicz's patch to call the stack-check code in pymalloc catches the problem "in time". But we could put the same patch anywhere in Python that gets called frequently, and catch it in time. That's too expensive to bear. The Windows stack-check code just isn't called often enough, and the recursion limit is "too large" now to live without a reliable stack check. Py_EnterRecursiveCall() in fact rarely calls the Windows stack-check code. At the time the Windows stack-check code first *would* have complained (had it been called then), the tstate recursion depth was 694, and _Py_CheckRecursionLimit was 973. The funky way this code works, that means we can do ~150 more levels of Python call before the Windows stack-check code is called again, and that's too late. We only checked that there's still room for 2K pointers on the stack, and 150 Python-level calls means about 450 C-level calls in this test: the stack at this point is hundreds (& hundreds) of repetitions of PyEval_EvalFrame -> fast_function -> call_function -> PyEval_Frame -> etc etc etc and PyEval_EvalFrame C frames are fat enough on their own for 150 of them to blow what remains of the stack. The easiest workaround for Windows is still to boost the stack size. I'm going to do that if nobody screams. Looks like nobody has an explanation yet for why 2.3.4 consistently yielded MemoryError but 2.4a2 mixes those with spurious KeyError and SyntaxError exceptions. That could be coincidence in this specific test driver, though -- lookdict has always wiped out exceptions. where's-stackless-when-you-need-it-ly y'rs - tim

Tim Peters <tim.peters@gmail.com> wrote:
[...]
The easiest workaround for Windows is still to boost the stack size. I'm going to do that if nobody screams.
Would reducing the recursion limit (for Windows) be a reasonable approach? Charles P.S. Please don't cc: me on list messages. -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

[Charles Cazabon]
Would reducing the recursion limit (for Windows) be a reasonable approach?
Not if that means Windows can't run the standard test suite. Boosting the stack size on Windows is a VM reservation operation, BTW -- it doesn't actually increase the RAM needed, unless the stack actually needs to grow that big.

[Tim]
[Armin]
The reason is that Py_EnterRecursiveCall() was only introduced in 2.4. Comparisons didn't throw RuntimeErrors that easily in 2.3.
Doh! Of course. Recursive compares aren't implicated in the test programs that failed here. Under 2.3.4, all the compares return normally because they're not stack-checking at all, and the program "gets to" recurse deeper then, until a "recursion depth exceeded" exception gets thrown. But in 2.4 it's a crap shoot whether a comparison or a recursive Python call notices first that we're nearing the end of the stack. The recursive Python calls make real stack demands in these tests, but the comparisons are just int-vs-int and string-vs-string, so the stack-check in compare now is "almost always" a nuisance check in these tests. Sounds like a good idea not to run out to stack <wink>.

Hello, On Thu, Aug 12, 2004 at 03:41:37PM -0400, Tim Peters wrote:
Sounds like a good idea not to run out to stack <wink>.
I don't remember if it was mentioned here, but maybe we'd better check directly whether the C stack is too large rather than (or possibly in addition to) using a limit on the number of Python iterations. This is not really ANSI C, but I can't think of a setting in which the following trick would fail to be a good indication of the amount of stack space consumed: save a pointer to a local variable "early", e.g. in Py_Initialize(); then in any other function call, the distance between this pointer and a pointer to some local equals the amount of stack space consumed by Python minus a few bytes. If this sounds too much of a hack, the (usual) recursion limit could be kept to limit nested Python frames; but all C-level recursions (including eval_frame) could additionally use the above trick. Its advantage is that it is an extremely fast check. If the total stack size is difficult to know in advance, we can still use PyOS_CheckStack(), but more efficiently and less randomly than now, by maintaining a "high tide" pointer that records how much stack we are sure we have, and calling PyOS_CheckStack() only to attempt to push the "high tide" mark further. While I'm in a hacking mood, there might be a way to prevent PyErr_Clear() to clear away "asynchronuous" exceptions like RuntimeError, MemoryError, and KeyboardInterrupt: for these exceptions, let's just store them away instead of clearing them, and re-raise them at the next occasion. The "next occasion" would have to be defined more precisely, but there is probably a way to ensure that it will at least be before the next non-trivial opcode operation starts. It would overwrite an exception set later, like those spurious KeyError we get for dict lookups. It might be a better-than-nothing quick fix to all these PyErr_Clear() all around the code base. A bientôt, Armin.

[Armin Rigo]
How large is too large? The Windows way checks the stack for impending overflow directly. A long "stack check on Unix" thread here in 2000 fizzled out in the expected way: no two Unices appear to have anything in common <0.9 wink>.
It fails for threads, whose stacks may be anywhere in relation to the main thread's. So any gimmick like this has to have a thread-local "starting point". If "a stack" is always a contiguous slice of memory, then it can work.
If this sounds too much of a hack,
No, but I'm not sure how much additional hackery it requires to make it work in all cases.
The ways to do it that don't work are indeed extremely fast <wink>.
If the total stack size is difficult to know in advance,
That's apparently the case across platforms -- and someimes the main-thread stack is much larger than secondary-thread stacks. The Windows check doesn't even try to know the current stack's size, but relies on MS-specific C exception extensions.
Threads again complicate this. AFAICT, the only platform that defines USE_STACKCHECK now (and actually implements PyOS_CheckStack()) is 32-bit Windows using an MS compiler. I did boost the stack reservation for that platform in CVS, which can't solve it, but at least hides it better <wink>.
Threads probably complicate that too. It's dreadful that serious problems can get transformed into bogus KeyErrors (in the test driver I posted, I'm not sure I spelled this out, but the key always *was* in the dict when a MemoryError got turned into a KeyError; we called a comparison function to begin with because the hash codes matched, and since these were mostly small-integer keys, they were identical to their hash codes -- so these keys were in the dicts, and the KeyErrors were lying about that). That's just nuts. I don't have spare cycles to give to it, though, so you'll have to save the world on your own again.

Hello Tim, On Sun, Aug 15, 2004 at 10:50:25PM -0400, Tim Peters wrote:
Yes. Here is a patch attempting to do what I described: http://www.python.org/sf/1009929 It's an extension of the asynchronous exception mecanism used to signal between threads. PyErr_Clear() can send some exceptions to its own thread using this mecanism. (So it is thread-safe.) Armin

On Aug 16, 2004, at 7:29 AM, Armin Rigo wrote:
Yes. Here is a patch attempting to do what I described: http://www.python.org/sf/1009929
From the patch description:
It seems to me that something similar to what Java has would be a good idea. Namely, a new top level exception (from which all others would derive) called "Raisable", analogous to Java's Throwable. This then has two subclasses: "Exception", and "FatalError". I'm not sure FatalError is a good name, but *some* new name needs to be thought up for Java's "Error" class, because lots of python exceptions end in "Error" but belong under the "Exception" hierarchy (for example "KeyError"). The criteria for whether a given exception should go under "Exception" or "FatalError" is whether users' code should generally catch the exception. Thus, at least "SystemExit", "KeyboardInterrupt", and "MemoryError" should go under "FatalError". Catching those is nearly never what you wanted to do when you write "except Exception:". There's likely more -- I have not gone through all the exceptions in Python to classify them. One issue is that creating a new category of Exceptions doesn't help people who do "except:" instead of "except Exception:". It is unlikely that person meant to catch things like MemoryError, rather, they were just being lazy. I suppose that syntax could be deprecated, at least in example code and documentation, in favor of "except Exception" for the usual case, and "except Raisable" for the cases where you do actually want to catch everything*. James * Except, of course, old string exceptions which have been deprecated for ages.

On Aug 16, 2004, at 8:07 PM, James Y Knight wrote:
* basestr could inherit from "Raisable" ;) The big problem with "Raisable" is that both raiseable and raisable seem to be correct spellings, and I don't think either are in many abridged dictionaries (the OS X spell checker doesn't like either, for example). -bob

On Mon, Aug 16, 2004, James Y Knight wrote:
We've already got StandardError; I think it makes more sense to rearrange the exception hierarchy a bit to support your suggestion rather than creating a whole new base class. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com

Hm, Java needs the distinction because some exceptions must be declared and others mustn't. But Python doesn't have that distinction. I'm not sure that you can always treat the same set of exceptions as fatal. E.g. in many situations, AttributeError, TypeError and NameError are all indicative of programming bugs (typically typos), but in other contexts these are recoverable. So rather than giving an arbitrary definition of fatality, let's refrain from defining the concept.
Calling SystemExit and KeyboardInterrupt fatal strikes me as particularly odd, as I routinely catch these.
--Guido van Rossum (home page: http://www.python.org/~guido/)

On Aug 16, 2004, at 9:32 PM, Guido van Rossum wrote:
Well, actually, java has three categories of exceptions: 1) Error - not necessary to declare as thrown, only *serious* errors. Contains: a) things that can "never happen" with a program compiled by the java compiler (no such field, unknown class file format, etc), b) machine errors (out of memory, stack overflow, etc) c) assert() failure d) ThreadDeath (similar to a KeyboardInterrupt/SystemExit) 2) Exception - normal run of the mill exception, needs to be declared as thrown. 3) RuntimeException - subclass of Exception, does not need to be declared as thrown. (e.g. IndexOutOfBoundsException, NoSuchFieldException, ClassCastException) The distinction you refer to above is really the difference between Exception and RuntimeException. Translated to java, AttributeError, TypeError and NameError would be RuntimeExceptions. So, I agree with you - I don't believe python needs that distinction. I do believe python needs the distinction between Exception and Error.
Calling SystemExit and KeyboardInterrupt fatal strikes me as particularly odd, as I routinely catch these.
I'll agree: I don't think the name "FatalError" is particularly great. However, I hope it gets the idea across better than "XXXErrorXXXRenameMeXXX" which was my other choice of name. ;) I do think the categorization is correct. While you may sometimes catch KeyboardInterrupt/SystemExit, most of the time you really do not want to, even if you are catching "everything". If you do want to catch KeyboardInterrupt, you are also likely to be catching it explicitly by its name, anyhow. James

[Armin Rigo]
I'm sorry that I haven't had time to look at this. But since I didn't and don't, let's try to complicate it <wink>. Some exceptions should never be suppressed unless named explicitly, and a real bitch is that some user-defined exceptions can fit in that category too. The ones that give me (and my employer) the most grief are the tree of exceptions deriving from ZODB's ConflictError. ConflictError is a serious thing: it essentially means the current transaction cannot succeed, and the app should give up (and maybe retry the current transaction from its start). Suppressing ConflictError by accident-- even inside a hasattr() call! --can grossly reduce efficiency, and has a long history too of provoking subtle, catastrophic, database corruption bugs. I would like to see Python's exception hierarchy grow more sophisticated in this respect. MemoryError, SystemExit, and KeyboardInterrupt are things that should not be caught by "except Exception:", neither by a bare "except:", nor by hasattr() or C-level dict lookup. ZODB's ConflictError is another of that ilk. I'd like to see "except Exception:" become synonymous with bare "except:", and move the "dangerous exceptions" to subclass off a new branch of the exception hierarchy. It could be that something like your patch is the only practical way to make this work in the C implementation, so I'm keen on it.

In article <1f7befae04090422024afaee58@mail.gmail.com>, Tim Peters <tim.peters@gmail.com> wrote:
It's not really the same subject, but the exception that gives me the most grief is StopIteration. I have to keep remembering to never call .next() without catching it; if I forget, I get bugs where some loop several levels back in the call tree mysteriously exits. -- David Eppstein Computer Science Dept., Univ. of California, Irvine http://www.ics.uci.edu/~eppstein/

Are you sure? This sounds like superstition to me, since that's not how loops work. Raising StopIteration in the middle of a loop does not break out of the loop -- only raising StopIteration from a next() breaks a loop. Or are you talking about nested next() calls? That's the only case where the behavior you are citing occurs. -- --Guido van Rossum (home page: http://www.python.org/~guido/) Ask me about gmail.

In article <ca471dc2040906184648d95e55@mail.gmail.com>, Guido van Rossum <gvanrossum@gmail.com> wrote:
I don't remember, it could have been nested next()s. -- David Eppstein Computer Science Dept., Univ. of California, Irvine http://www.ics.uci.edu/~eppstein/

On Sun, 5 Sep 2004 01:02:35 -0400, Tim Peters <tim.peters@gmail.com> wrote:
The current exception hierarchy isn't too far from what you suggest. We just got the names wrong. That is, there is a base class, StandardException, that captures most exceptions other than MemoryError, SystemError, and KeyboardInterrupt. If we renamed that Exception, then we'd be 90% of the way there. You could also change your code, right now, to say "except StandardError:" and avoid the problem entirely. Make sure ConflictError does not inherit from StandardError, of course. And make sure you're happy that ImportError is not a StandardError either. I'm not sure what I think of the change to "except:" It's often the case that someone who has written "except:" really means "except Something:", but I expect that very often Something != StandardError and issubclass(Something, StandardError). In that case, the change doesn't really help them. The code is still wrong. Jeremy

On Mon, 6 Sep 2004 09:44:12 +0200, Fredrik Lundh <fredrik@pythonware.com> wrote:
I misread the very long output of pydoc exceptions :-). I don't understand the hierarchy, either, or I would have noticed that the results don't make sense. Why isn't StopIteration a StandardError? I'll second Tim's suggestion that some errors -- like SystemError, MemoryError, and KeyboardInterrupt belong in a different category. I think it would be easier in principle to put them at a different place in the class hierarchy than to make them some special kind of uncatchable exception. Jeremy

Jeremy Hylton wrote: ...
Note that we don't want uncatchable exceptions. Rather, we want exceptions that aren't caught by bare excepts or very broad excepts. In many cases, we want certain knowledgeable code to be able to catch these exceptions. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Tue, 07 Sep 2004 09:43:53 -0400, Jim Fulton <jim@zope.com> wrote:
I agree with half the cause. There ought to be a decent organization of the exception class hierarchy so that exceptions like KeyboardInterrupt are in a special category. Then an "except NormalError:" <wink> would catch only the normal errors and not the special ones. I don't think bare exception should change it's meaning; you just shouldn't use it unless it's *really* what you mean. I think backwards compatibility is a really hard issue for any of these changes. It's probably hard to re-arrange the class hierarchy, but I don't know what practical solution there is to these problems that doesn't involve breaking some code. It's even harder to change bare except, but I don't think that's necessary. Jeremy

On Tue, 2004-09-07 at 09:43, Jim Fulton wrote:
I don't agree about having exceptions that pass bare excepts. A typical /valid/ use of bare excepts are in frameworks such as transaction processing, where you need to do some extra work when an exception occurs, then re-raise the original exception, e.g.: try: do_something() except: database.rollback() raise else: database.commit() Even exceptions like SystemError, MemoryError, or KeyboardInterrupt want to adhere to this simple idiom. Bare except should continue to catch all exceptions. Code that wanted to do otherwise should /not/ use a bare except, and +1 on some form of exception hierarchy restructuring that would make that clearer. -Barry

Barry Warsaw wrote:
Fair enough. I also agree with jeremy's points re backward compatability. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Barry Warsaw wrote:
My policy for bare excepts is that without significant justification they _must_ either re-raise the original exception or raise another exception. There are very few circumstances where I have allowed my team to write pure bare excepts. I haven't checked, but a warning for violations of this rule may be a nice addition to pychecker or pylint. -Kevin

On Tue, 2004-09-07 at 11:11, Kevin Jacobs wrote:
The other case I've seen are for command-shell like loops, where you might print the exception in the bare except, but not re-raise the exception. Think about the main interactive interpreter loop. But yeah I agree, you want strong justification for any use of bare except. -Barry

After a bit more thought (and it's hard to measure how little), I'd like to see "bare except" deprecated. That doesn't mean no way to catch all exceptions, it means being explicit about intent. Only a few of the bare excepts I've seen in my Python life did what was actually intended, and there's something off in the design when the easiest thing to say usually does a wrong thing. I think Java has a saner model in this particular respect: Throwable Exception Error Java's distinction between "checked" and "unchecked" exceptions is a distinct layer of complication on top of that. All exceptions derive from Throwable. A "catch" clause requires specifying a class (there's no "bare except"). "An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch". That includes AssertionError and VirtualMachineError. Those are exceptions that should never occur. It also includes ThreadDeath, which is expected to occur, but The class ThreadDeath is specifically a subclass of Error rather than Exception, even though it is a "normal occurrence", because many applications catch all occurrences of Exception and then discard the exception. and it's necessary for ThreadDeath to reach the top level else the thread never really dies. In that respect, it's interesting that SystemExit and KeyboardInterrupt are *intended* to "reach the top level" too, but can't be relied on to do so because of ubiquitous bare excepts and even pseudo-careful "except Exception:"s now. If people changed those to "except StandardError:", SystemExit would make it to the top but KeyboardInterrupt still wouldn't. Raisable Exception Stubborn ControlFlow KeyboardInterrupt StopIteration SystemExit MemoryError introduces a class of stubborn exceptions, those that wouldn't be caught by "except Exception:", and with the intent that there's no way you should get the effect of "except Raisable" without explictly saying just that (once bare except's deprecation is complete). Oh well. We should elect a benevolent dictator for Python!

Tim Peters wrote:
Giving it same amount of thought as Tim, I like this idea as well. I don't think the burden of having to always specify an exception to catch is that much work and would definitely be more explicit. Plus it is not hard to teach to newbies; just tell them to catch the exception all of the "safe" exceptions inherit from. Is it PEP time? We have now had the idea of reorganizing the exception hierarchy come up in this thread and in early August (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is... for the thread on reorganizing for the purpose of using new-style classes and the idea of having all raisable objects inherit from a specific base class). It seems like a serious enough of an idea that it will happen for Python 3000 and thus should be discussed; never too early to start. But maybe it at least deserves to be mentioned in PEP 3000 (Guido?)?
Oh well. We should elect a benevolent dictator for Python!
Here, here! =) -Brett

Tim Peters wrote:
I have copied stack check into PyObject_Malloc and after X calls to PyEval_EvalFrame, fast_function python dies in malloc(msvcrt) and HeapAlloc (ntdll) as described by Tim. After this patch most ocurring exception is MemoryError, but for 141, 171, 201 and 231 python raises SyntaxError - always witout traceback :( mak relevant part from obmalloc.c: PyObject_Malloc(size_t nbytes) { block *bp; poolp pool; poolp next; uint size; #ifdef USE_STACKCHECK if (PyOS_CheckStack()) { return NULL; } #endif

This suggests to me that the very concept of a function that's not allowed to return errors is dubious, since even just calling another function could provoke a MemoryError. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[Tim Peters]
[Greg Ewing]
I believe I had a more elegant way to say that: AAAARGHGH! <wink> Alas, 10 billion call sites, including in external extensions, rely on that a NULL return from PyDict_GetItem() or PyDict_GetItemString() is the unexceptional "nope, the key's not there". So there's no realistic chance of repairing this before Python 3.0 (if even then). In the meantime, KeyError is confusable with almost all exceptions. For some real fun, ask on a Zope list about hasattr() suppressing all exceptions. Not that Guido ever expected a hasttr() call to try to mess with persistent objects across a networked database as side effect. Python originally had only string-keyed dicts, and the basic dict API still reflects that a dict lookup simply "can't fail" for any reason other than "key ain't there". That used to be true. For that matter, hasattr() was originally no worse than a string-keyed dict lookup too, so "can't fail" also made good sense there at the time.

On Mon, 9 Aug 2004, Tim Peters wrote: [...]
Has anyone on Linux tried this yet?
On Linux, I get precisely the same output from Python CVS that you posted, at least as far as 148 (didn't check any further): [...] 118 ok 119 ok 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301 138 exceptions.RuntimeError maximum recursion depth exceeded 139 exceptions.KeyError 300 140 exceptions.RuntimeError maximum recursion depth exceeded 141 exceptions.RuntimeError maximum recursion depth exceeded 142 exceptions.KeyError 299 143 exceptions.RuntimeError maximum recursion depth exceeded 144 exceptions.KeyError 297 145 exceptions.KeyError 296 146 exceptions.RuntimeError maximum recursion depth exceeded 147 exceptions.KeyError 295 148 exceptions.RuntimeError maximum recursion depth exceeded [...] John

Hello Tim, On Mon, Aug 09, 2004 at 09:48:37PM -0400, Tim Peters wrote:
Has anyone on Linux tried this yet?
I get this: in a release build, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.RuntimeError maximum recursion depth exceeded 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.RuntimeError maximum recursion depth exceeded ... In a release 21st of March version with the fresh generator expression patch applied, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 305 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 304 129 exceptions.RuntimeError maximum recursion depth exceeded ... In a recent debug build, ... 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded ... So there is no particular problem running debug builds under Linux, but still why we get this behavior is anybody's guess. Also note that even though the compiler package may only have changed a bit, the parser C module definitely has, e.g. when genexprs and decorators were introduced. I wonder what effect this has on the depth of concrete syntax trees. I also wonder if the parser module tries to be safe again C stack overflow when building hugely nested data structures. Armin

Hello again, On Tue, Aug 10, 2004 at 08:16:38PM +0100, Armin Rigo wrote:
Sorry, my mistake. This was after I removed from object.c the two Py_Enter/LeaveRecursiveCall(" in cmp"). Not suprizingly, I then only get "regular" recursion errors. A pure CVS version gets me the same results as you and John. Armin
participants (23)
-
Aahz
-
Armin Rigo
-
Barry Warsaw
-
Bob Ippolito
-
Brett C.
-
Charles Cazabon
-
David Eppstein
-
Fred L. Drake, Jr.
-
Fredrik Lundh
-
Greg Ewing
-
Grzegorz Makarewicz
-
Guido van Rossum
-
Guido van Rossum
-
James Y Knight
-
Jeremy Hylton
-
Jim Fulton
-
John J Lee
-
Kevin Jacobs
-
Michael Hudson
-
Nick Coghlan
-
Raymond Hettinger
-
Stephen J. Turnbull
-
Tim Peters