Hi py-dev, I got a problem report for Stackless today, that it seems to leak with tracebacks. After trying other Python versions, I found out that this is a "feature" of Python and not related to Stackless. The problem becomes only more visible, since people are keeping thousands of threads alive. Here the problem: When an exception has been raised in a frame, and it already is handled in an except clause, the exception is not cleared out from tstate and also stays alive in the frame object. Only when the frame is left, eval_frame calls reset_exc_info(tstate); which clear all these, breaking cycles. Question: Does this need to be so, and for what reason? Would it be equivalent if I cleared error info in the context of a finally: ? If not, please give me advice how to solve this problem. It exists in all long-running frames which have seen exceptions. Thanks a lot - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
If I understand correctly what you are referring to, this is a feature. You're supposed to say: try: return f() except: pass sys.exc_info() i.e. the exc_info() must remain accessible. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
If I understand correctly what you are referring to, this is a feature. You're supposed to say:
try: return f() except: pass sys.exc_info()
i.e. the exc_info() must remain accessible.
Aha! Yup! Thanks. Then, it is probably not a bug but a feature. Would it be ok then, if I write a function that exposes reset_exc_info to the user? sys.exc_clear() # maybe? Or a function which returns *and* clears the exc? tup = sys.exc_info_reset() ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Guido van Rossum wrote:
If I understand correctly what you are referring to, this is a feature. You're supposed to say:
try: return f() except: pass sys.exc_info()
i.e. the exc_info() must remain accessible.
[Christian]
Aha! Yup! Thanks.
Then, it is probably not a bug but a feature. Would it be ok then, if I write a function that exposes reset_exc_info to the user?
sys.exc_clear() # maybe?
Or a function which returns *and* clears the exc?
tup = sys.exc_info_reset()
Sure, as long as it is part of Stackless and not of Python proper. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, 24 Feb 2003, Guido van Rossum wrote:
[Christian]
Then, it is probably not a bug but a feature. Would it be ok then, if I write a function that exposes reset_exc_info to the user?
sys.exc_clear() # maybe?
Or a function which returns *and* clears the exc?
tup = sys.exc_info_reset()
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core? Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core?
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, 24 Feb 2003, Guido van Rossum wrote:
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core?
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
This has nothing to do with Stackless, and I'm happy to have it for Python 2.3 (or 2.4 if it is too risky to add post-a2). Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core?
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
This has nothing to do with Stackless,
Well, Christian explained that it was a problem because they have 1000s of threads.
and I'm happy to have it for Python 2.3 (or 2.4 if it is too risky to add post-a2).
I don't have time to write the code, but I'll entertain a patch before 2.3b1. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, 24 Feb 2003, Guido van Rossum wrote:
This has nothing to do with Stackless,
Well, Christian explained that it was a problem because they have 1000s of threads.
My problem relates to logging the "last exception" in a catch-all error handler. We sometimes see the same exception when an error occurs and no new exception is thrown.
and I'm happy to have it for Python 2.3 (or 2.4 if it is too risky to add post-a2).
I don't have time to write the code, but I'll entertain a patch before 2.3b1.
No problem. I'll upload it to SF, along with documentation and a new unit test. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Guido van Rossum wrote:
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core?
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
This has nothing to do with Stackless,
Well, Christian explained that it was a problem because they have 1000s of threads.
Yeah. But standard CPython can exploit the same thing, with a handful of real threads, which happen to catch exceptions from, say, a very deep, very memory consming chain of frames. These are all kept alive, and from the documentation, people don't expect this! ...
I don't have time to write the code, but I'll entertain a patch before 2.3b1.
I was about to supply a patch, that's why I asked for the right syntax. Do you want an extra function of which name, or do you like a default arg to exc_info? Both patches should not take me more than 1/2 hour, or I should better give up on programming and become a farmer. no-offense-to-farmers,-they-are-probably-the-happier-people -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Guido van Rossum wrote:
Sure, as long as it is part of Stackless and not of Python proper.
Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core?
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
Huh? What makes you believe that? I'm talking of 2.3, of course. Sure, it is possible for long-running functions to wrap any exception-raising stuff into an extra wrapping function that always returns after the exception has happened, and to let the main worker function run without try..except. But is this very cool? I would also like to point out that the documentation of sys.exc_info is very misleading, and I always programmed acording to this false information: """
print sys.exc_info.__doc__ exc_info() -> (type, value, traceback)
Return information about the exception that is currently being handled. This should be called from inside an except clause only.
"""
Where I have to say that I'm in favor of doing like the documentation claims. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
I'm not completely clear on the problem being described here, but:
Sure, it is possible for long-running functions to wrap any exception-raising stuff into an extra wrapping function that always returns after the exception has happened, and to let the main worker function run without try..except. But is this very cool?
I point out that the following code: import sys def a(): try: 1/0 except: t, v, tb = sys.exc_info() if __name__=='__main__': while 1: a() When garbage collection is not enabled will cause huge memory leaks. The problem is the "tb" variable - the traceback holds a reference to the stack frame, which holds references to the locals, which holds a reference back to "tb". gc clears these cycles, so later Python versions work fine - does stackless enable gc? Explicitly setting "tb" to None in the exception handler also fixes it. No idea if it is related or not, but this problem caused me such grief years ago that I thought it worth mentioning. Mark.
Mark Hammond wrote:
I'm not completely clear on the problem being described here, but:
Sure, it is possible for long-running functions to wrap any exception-raising stuff into an extra wrapping function that always returns after the exception has happened, and to let the main worker function run without try..except. But is this very cool?
I point out that the following code:
import sys def a(): try: 1/0 except: t, v, tb = sys.exc_info()
if __name__=='__main__': while 1: a()
When garbage collection is not enabled will cause huge memory leaks. The problem is the "tb" variable - the traceback holds a reference to the stack frame, which holds references to the locals, which holds a reference back to "tb".
gc clears these cycles, so later Python versions work fine - does stackless enable gc? Explicitly setting "tb" to None in the exception handler also fixes it.
Thanks a lot. Yes, I know about this and advised them to clear t, v, tb after usage, which breaks the cycle. The problem is just that they don't like to use an extra wrapper function around the exception, but want to handle it in their top-level worker loop. When an exception happens there, it isn't cleared until another one happens, or the function leaves. But the latter never happens... ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[Guido]
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
[Christian]
Huh? What makes you believe that?
I thought quite a while ago you announced you were giving up the old stackless code. I must have misunderstood.
I'm talking of 2.3, of course.
But without GC, right? Or did I misunderstand that too? I thought you told me that your customers didn't want GC enabled? But in 2.3 GC can't be disabled. So I guess I'm unclear on what you want.
I would also like to point out that the documentation of sys.exc_info is very misleading, and I always programmed acording to this false information:
"""
print sys.exc_info.__doc__ exc_info() -> (type, value, traceback)
Return information about the exception that is currently being handled. This should be called from inside an except clause only.
"""
It's pretty complex to explain the actual behavior, so I gave a recipe that's a little more restricted but is guaranteed to work.
Where I have to say that I'm in favor of doing like the documentation claims.
Too bad, that would definitely break existing code. [And later, Guido]
Well, Christian explained that it was a problem because they have 1000s of threads.
[Christian]
Yeah. But standard CPython can exploit the same thing, with a handful of real threads, which happen to catch exceptions from, say, a very deep, very memory consming chain of frames. These are all kept alive, and from the documentation, people don't expect this!
Who reads documentation. :-)
...
I don't have time to write the code, but I'll entertain a patch before 2.3b1.
I was about to supply a patch, that's why I asked for the right syntax. Do you want an extra function of which name, or do you like a default arg to exc_info? Both patches should not take me more than 1/2 hour, or I should better give up on programming and become a farmer.
You and Kevin Jacobs can argue about the syntax. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
[Guido]
I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy.
I am not talking about adding a feature but removing a wart.
[Christian]
Huh? What makes you believe that?
I thought quite a while ago you announced you were giving up the old stackless code. I must have misunderstood.
And introduced the new, much better, Stackless code, of course. This is what I'm talking about at PyCon. You were in the comittee.
I'm talking of 2.3, of course.
But without GC, right? Or did I misunderstand that too? I thought you told me that your customers didn't want GC enabled? But in 2.3 GC can't be disabled. So I guess I'm unclear on what you want.
I have asked them about GC. Both were less negative about GC than they claimed a year ago. But this is not the point. The code they have cannot be made working by GC, since there is a life cycle supported by prolonged survival of exceptions that cannot be solved by GC. There is *no* way to clear an exception other than raising another one, or by leaving the current function that received the exception. Since it is easy to catch an exception in the except clause my assingning sys.exc_info() to some locals, there is no need to make these implicitly survive longer than needed, with the extra surprize that there is no sane way to clean them up. ...
It's pretty complex to explain the actual behavior, so I gave a recipe that's a little more restricted but is guaranteed to work.
It is actually quite simple to explain, since I know almost all of ceval.c by heart: There is only one single function in the whole system that both clears the exception saved in tstate, *and* the extra reference which is captured in the current frame: It is reset_exc_info(), a static local function in ceval.c, which is exactly only called when eval_frame returns.
Where I have to say that I'm in favor of doing like the documentation claims.
Too bad, that would definitely break existing code.
Break it, break it! Code that does not adhere to the documentation is bad code that should be broken, even if it is your own. Examplify. Relying on side effects which are not documented have *never* been an argument for you to support abuse.
[And later, Guido] ...
Who reads documentation. :-)
You, I suppose. :-) ...
You and Kevin Jacobs can argue about the syntax.
Proposals, in decreasing orders of delight: 1) Implement exeptions according to the documentation. Cleaning exceptions after they are handled is what most users would expect, last but not least since the documentation suggests it. 2) Add an optional boolean clear argument to sys.exc_info() This is lame, but it gives the user one way out. 3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature". 4) Keep things as they are, and expect the Spanish Inquision. In case of 4), I assure you to start a thread that will not be less than the PEP308 mess. In the other cases, I will happily submit a validated patch. Please don't disappoint me by defending a questionable design. sincerely your's -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Tue, 25 Feb 2003, Christian Tismer wrote:
You and Kevin Jacobs can argue about the syntax.
Proposals, in decreasing orders of delight:
1) Implement exeptions according to the documentation. Cleaning exceptions after they are handled is what most users would expect, last but not least since the documentation suggests it.
This is not a practical option -- while I don't think the current behavior is ideal, we have a great deal of code that relies on this behavior. I'm sure that many other Python developers are in the same boat. If this is to change, we need to do a lot more work to justify it, and to give at least full one version of notice before enforcing it.
2) Add an optional boolean clear argument to sys.exc_info() This is lame, but it gives the user one way out.
exc_info(...) exc_info() -> (type, value, traceback) Return information about the exception that is currently being handled. This should be called from inside an except clause only. Adding an argument to this function reeks of a kludge. Its job is to return information, and should not take any magic arguments to make it a 'one-shot'.
3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature".
This one gets my vote (if this were a democracy). "Explicit is better than implicit". The new symbol in the sys namespace also makes it easy to test for the capability (without a version number check). Maybe it should be called 'clear_exc' or 'reset_exc_info' (to match the C API name).
4) Keep things as they are, and expect the Spanish Inquision.
I'm even okay with the status quo and can continue to use my current hack to clear exception data. Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs wrote:
On Tue, 25 Feb 2003, Christian Tismer wrote:
You and Kevin Jacobs can argue about the syntax.
Proposals, in decreasing orders of delight:
1) Implement exeptions according to the documentation. Cleaning exceptions after they are handled is what most users would expect, last but not least since the documentation suggests it.
This is not a practical option -- while I don't think the current behavior is ideal, we have a great deal of code that relies on this behavior. I'm sure that many other Python developers are in the same boat. If this is to change, we need to do a lot more work to justify it, and to give at least full one version of notice before enforcing it.
Why this? sys.exc_info() is well documented since a couple of versions. I was even so dumb to code according to it that I wasn't even aware of the problem. (But I should have known better having read all the code). ...
3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature".
This one gets my vote (if this were a democracy). "Explicit is better than implicit". The new symbol in the sys namespace also makes it easy to test for the capability (without a version number check). Maybe it should be called 'clear_exc' or 'reset_exc_info' (to match the C API name).
I don't care, but to get that crappy problem off my desk.
4) Keep things as they are, and expect the Spanish Inquision.
I'm even okay with the status quo and can continue to use my current hack to clear exception data.
I have received lots of complaints about "bad Stackless behavior", and I would be happy if I were causing the problem, in the first place. The problem exists since centuries, but is exploited by the fact that people can have so many threads now. I hate to get prosecuted for the consequences of other people's decisions, in a way... ... but I'm far from being a language designer, for heaven's sake :-)) unseriously yours - chgris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
"KJ" == Kevin Jacobs <jacobs@penguin.theopalgroup.com> writes:
KJ> This one gets my vote (if this were a democracy). "Explicit KJ> is better than implicit". The new symbol in the sys namespace KJ> also makes it easy to test for the capability (without a KJ> version number check). Maybe it should be called 'clear_exc' KJ> or 'reset_exc_info' (to match the C API name). Or exc_clear() for symmetry with exc_info() ? -Barry
1) Implement exeptions according to the documentation. Cleaning exceptions after they are handled is what most users would expect, last but not least since the documentation suggests it.
This is not a practical option -- while I don't think the current behavior is ideal, we have a great deal of code that relies on this behavior. I'm sure that many other Python developers are in the same boat. If this is to change, we need to do a lot more work to justify it, and to give at least full one version of notice before enforcing it.
Indeed. Not to mention that a new opcode would have to be invented to signal the end of the except block.
2) Add an optional boolean clear argument to sys.exc_info() This is lame, but it gives the user one way out.
exc_info(...) exc_info() -> (type, value, traceback)
Return information about the exception that is currently being handled. This should be called from inside an except clause only.
Adding an argument to this function reeks of a kludge. Its job is to return information, and should not take any magic arguments to make it a 'one-shot'.
Agreed; -1 on this particular API.
3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature".
This one gets my vote (if this were a democracy). "Explicit is better than implicit". The new symbol in the sys namespace also makes it easy to test for the capability (without a version number check). Maybe it should be called 'clear_exc' or 'reset_exc_info' (to match the C API name).
Let's call it exc_reset() or exc_clear().
4) Keep things as they are, and expect the Spanish Inquision.
I'm even okay with the status quo and can continue to use my current hack to clear exception data.
You can always clear it by raising and catching another exception. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, 25 Feb 2003, Guido van Rossum wrote:
3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature".
This one gets my vote (if this were a democracy). "Explicit is better than implicit". The new symbol in the sys namespace also makes it easy to test for the capability (without a version number check). Maybe it should be called 'clear_exc' or 'reset_exc_info' (to match the C API name).
Let's call it exc_reset() or exc_clear().
I'm going with exc_clear(), since 'clearing' seems to be the verb most people are using when talking about the action it performed. Implementation question: I'm planning to reuse the reset_exc_info function defined statically in ceval.c. Which of these options do you prefer: 1) Just remove the static declaration from reset_exc_info and add the prototype as an extern declaration to sysmodule.c. 2) Remove the static declaration from reset_exc_info and add the prototype to an include file. If so, which include file is the most appropriate place? 3) Move the implementation of reset_exc_info out of ceval to another, more public, file. If so, which one? I ask only because I usually submit bug fixes and minor changes, and am not familar enough with this part of the Python development culture.
4) Keep things as they are, and expect the Spanish Inquision.
I'm even okay with the status quo and can continue to use my current hack to clear exception data.
You can always clear it by raising and catching another exception. :-)
This is what I do, though I have a nice 10 line comment in the source to ward off much squinting and head scratching from our junior developers. Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
I'm going with exc_clear(), since 'clearing' seems to be the verb most people are using when talking about the action it performed.
OK.
Implementation question: I'm planning to reuse the reset_exc_info function defined statically in ceval.c. Which of these options do you prefer:
1) Just remove the static declaration from reset_exc_info and add the prototype as an extern declaration to sysmodule.c.
It has to be renamed to have a proper Py*** name too.
2) Remove the static declaration from reset_exc_info and add the prototype to an include file. If so, which include file is the most appropriate place?
ceval.h.
3) Move the implementation of reset_exc_info out of ceval to another, more public, file. If so, which one?
I see no need.
I ask only because I usually submit bug fixes and minor changes, and am not familar enough with this part of the Python development culture.
Looks like you're catching on fast. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I'm going with exc_clear(), since 'clearing' seems to be the verb most people are using when talking about the action it performed.
I agree. clear is in fact better than reset, since there is no exception resetted to something, but it is cleared away. ...
I ask only because I usually submit bug fixes and minor changes, and am not familar enough with this part of the Python development culture.
Great, Kevin, that you are implementing this. I was already starting, but this is fine, if I can save some time. Will instead work a little on making GC optional, again, as promised a while ago. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Tue, 25 Feb 2003, Christian Tismer wrote:
Guido van Rossum wrote:
I'm going with exc_clear(), since 'clearing' seems to be the verb most people are using when talking about the action it performed.
I agree. clear is in fact better than reset, since there is no exception resetted to something, but it is cleared away.
I ask only because I usually submit bug fixes and minor changes, and am not familar enough with this part of the Python development culture.
Great, Kevin, that you are implementing this. I was already starting, but this is fine, if I can save some time.
I've uploaded a patch to SF: http://www.python.org/sf/693195 It turns out that ceval.c:reset_exc_info was a red herring and did not do what we wanted. Luckily, the correct behavior is fairly trivial (although the interaction between set_exc_info and reset_exc_info had me seeing cross-eyed for a few minutes). -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs wrote: ...
I've uploaded a patch to SF:
http://www.python.org/sf/693195
It turns out that ceval.c:reset_exc_info was a red herring and did not do what we wanted. Luckily, the correct behavior is fairly trivial (although the interaction between set_exc_info and reset_exc_info had me seeing cross-eyed for a few minutes).
I had a look at your patch. Here a small analysis: In essence, it is doing exactly the same as PyErr_Restore(NULL, NULL, NULL) from errors.c, with the difference that it is acting on sys.exc_(type, value, traceback) instead of sys.curexc_(type, value, traceback). I just want to make sure that this is what we want to do. The rest of reset_exc_info restores an exception that has been captured in the frame, before. The counterpart is set_exc_info, which is called only if an exception handler is executed. The frame then saves a prior exception from tstate to local variables and sets a new exception into tstate. I have no real clue what should happen if this occours multiple times -- the frame will only accept one such exception to save. Ah, I see: This is meant to be an exception which originated in the caller of the function, and the frame is just supposed to save the caller's exception, in case it is itself called in an exception handler! The action on return is to drop the current exception, since it has been handled, and to restore the caller's exception, if this exists. There is also a small trick here: set_exc_info assigns Py_None to tstate->exc_type, if it has been NULL. This is then assigned to the local variables of the frame. The side effect of this trick is, that after set_exc_info has been called once, reset_exc_info always will think that it has to restore, even if there is just a None-Exception. The side effect is, that the frame's variables always will be cleared out on a return. Well, this code could stand a few more comments ;-) Ok, what I understood now is, that this extra action of reset_exc_info is really not what we need here, this is about restoring the caller's exception. But we are not concerned with the caller here, since we stay in the current frame. We just want to clear the current exception in tstate, and I think your code does the right thing. +1 cheers -- chris p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO. -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Tue, 25 Feb 2003, Christian Tismer wrote:
We just want to clear the current exception in tstate, and I think your code does the right thing.
+1
Thanks for checking through the logic.
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs wrote:
On Tue, 25 Feb 2003, Christian Tismer wrote:
We just want to clear the current exception in tstate, and I think your code does the right thing.
+1
Thanks for checking through the logic.
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
No, I didn't refer to your code, but just to the implementation of (re)set_exc_info. I would have understood this much easier, if the saved exception were saved in the caller's frame. I'm not proposing a change, but maybe a comment, why this must be saved. Yes, probably it is most efficient to do it as it is. (Although functions called in an exception context are probably not the normal case which needs to be optimal). Saving and restoring things in the callee which belong to the caller just doesn't look very clean, and I now also understand certain problems with old Stackless, where this probably caused errors which were never resolved, simply because I didn't understand this. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Wed, 26 Feb 2003, Christian Tismer wrote:
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
No, I didn't refer to your code, but just to the implementation of (re)set_exc_info.
Actually, we're on the same page.
I would have understood this much easier, if the saved exception were saved in the caller's frame. I'm not proposing a change, but maybe a comment, why this must be saved. Yes, probably it is most efficient to do it as it is. (Although functions called in an exception context are probably not the normal case which needs to be optimal).
Exactly. This is why I'm fairly certain that nobody looks at the frame.f_exc_* values, since they make no sense in the context of that frame. It should be trivial to dereference the traceback to find the generating frame and stow the values there. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs wrote:
On Wed, 26 Feb 2003, Christian Tismer wrote:
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
No, I didn't refer to your code, but just to the implementation of (re)set_exc_info.
Actually, we're on the same page.
I understand.
I would have understood this much easier, if the saved exception were saved in the caller's frame. I'm not proposing a change, but maybe a comment, why this must be saved. Yes, probably it is most efficient to do it as it is. (Although functions called in an exception context are probably not the normal case which needs to be optimal).
Exactly. This is why I'm fairly certain that nobody looks at the frame.f_exc_* values, since they make no sense in the context of that frame. It should be trivial to dereference the traceback to find the generating frame and stow the values there.
Yeah, but Guido had some point there, which I still have to investigate. I'm not quite sure, yet. Will get back to it in the other thread. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
Watch out though. There are situations where an exception needs to be stored but no frame is available (when executing purely in C). There is always a thread state. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
Watch out though. There are situations where an exception needs to be stored but no frame is available (when executing purely in C). There is always a thread state.
I've been sitting a while over this puzzle now. tstate has two different kinds of exceptions: There are tstate->exc_XXX and tstate->curexc_XXX. I have been searching through the whole source trunk to validate my thought: All internal stuff is only concerned with handling tstate->curexc_XXX. The tstate->exc_XXX is *only* used in ceval.c . References to tstate->exc_XXX are only in pystate.c (clearing stuff) and sysmodule.c (accessing stuff). The only place where tstate->exc_XXX is filled with life is ceval.c, which indicates that this is purely interpreter- -related and has nothing to do with the internal exception state. It is eval_frame which checks for exceptions, normalizes them and turns them into interpreter-level exceptions, around line 2360 of ceval.c . After stating that, I conclude that tstate.exc_XXX can only be in use if there is an existing interpreter with an existing frame. Nobody else makes use of this structure. So, whenever you have to save this, you can expect a valid frame waiting in f_back that will be able to take it. (This all under the maybe false assumption that I'm not wrong). Still not proposing a change. But thanks for the time, I understood quite a lot more of the internals, now. all the best -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
(Picking up an old thread.) [Guido]
Watch out though. There are situations where an exception needs to be stored but no frame is available (when executing purely in C). There is always a thread state.
[Christian]
I've been sitting a while over this puzzle now.
tstate has two different kinds of exceptions: There are tstate->exc_XXX and tstate->curexc_XXX.
I have been searching through the whole source trunk to validate my thought:
All internal stuff is only concerned with handling tstate->curexc_XXX.
Correct. This is the "hot" exception that is set by PyErr_SetString() c.s., cleared by PyErr_Clear(), and so on.
The tstate->exc_XXX is *only* used in ceval.c .
Once an exception is caught by an except clause, it is transferred from tstate->curexc_XXX to tstate->exc_XXX, from which sys.exc_info() can pick it up.
References to tstate->exc_XXX are only in pystate.c (clearing stuff) and sysmodule.c (accessing stuff). The only place where tstate->exc_XXX is filled with life is ceval.c, which indicates that this is purely interpreter- -related and has nothing to do with the internal exception state. It is eval_frame which checks for exceptions, normalizes them and turns them into interpreter-level exceptions, around line 2360 of ceval.c .
Correct.
After stating that, I conclude that tstate.exc_XXX can only be in use if there is an existing interpreter with an existing frame. Nobody else makes use of this structure. So, whenever you have to save this, you can expect a valid frame waiting in f_back that will be able to take it.
Right. Now let me explain the complicated dance with frame->f_exc_XXX. Long ago, when none of this existed, there were just a few globals: one set corresponding to the "hot" exception, and one set corresponding to sys.exc_type etc. The problem was that in code like this: try: "something that may fail" except "some exception": "do something else first" "print the exception from sys.exc_type etc." if "do something else first" invoked something that raised and caught an exception, sys.exc_type etc. were overwritten. That was a frequent cause of subtle bugs. I fixed this by changing the semantics as follows: - Within one frame, sys.exc_XXX will hold the last exception caught *in that frame*. - But initially, and as long as no exception is caught in a given frame, sys.exc_XXX will hold the last exception caught in the previous frame (or the frame before that, etc.). The first bullet fixed the bug in the above example. The second bullet was for backwards compatibility: it was (and is) common to have a function that is called when an exception is caught, and to have that function access the caught exception via sys.exc_XXX. (Example: traceback.print_exc()). At the same time I fixed the problem that sys.exc_type and friends weren't thread-safe, by introducing sys.exc_info() which gets it from tstate; but that's really a separate improvement. The reset_exc_info() function in ceval.c restores the tstate->exc_XXX variables to what they were before the current frame was called. The set_exc_info() function saves them on the frame so that reset_exc_info() can restore them. The invariant is that frame->f_exc_XXX is NULL iff the current frame never caught an exception (where "catching" an exception applies only to successful except clauses); and if the current frame ever caught an exception, frame->f_exc_XXX is the exception that was stored in tstate->exc_XXX at the start of the current frame. Now I hope you'll understand why this was never documented exactly. :-)
(This all under the maybe false assumption that I'm not wrong).
No; I guess I was wrong in the quoted text at the top. :-)
Still not proposing a change. But thanks for the time, I understood quite a lot more of the internals, now.
Great! Hope this message has shed some additional light. Kevin, I'll try to get to your patch next. --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido van Rossum" <guido@python.org> wrote in message news:200303010152.h211qwZ11517@pcp02138704pcs.reston01.va.comcast.net. .. [explanation of traceback info storage]
Great! Hope this message has shed some additional light.
It would be a shame for this to be lost in the archives. If there were a directory of ImplementationNotes somewhere (or an interpreter wiki), this would belong there. And responders to "where are the docs on the implementation" could be told more than "read the source". TJR
Great! Hope this message has shed some additional light.
It would be a shame for this to be lost in the archives. If there were a directory of ImplementationNotes somewhere (or an interpreter wiki), this would belong there. And responders to "where are the docs on the implementation" could be told more than "read the source".
Good idea. I hate separating implementation notes from the code by more than absolutely necessary (Zope's cobweb of Wikis drives me nuts :-), so I added the essence of that message to ceval.c as a big comment block. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Great! Hope this message has shed some additional light.
It would of course, two years earlier. When I wrote my message, I already had triple-checked that there was no way to contradict me :-)
It would be a shame for this to be lost in the archives. If there were a directory of ImplementationNotes somewhere (or an interpreter wiki), this would belong there. And responders to "where are the docs on the implementation" could be told more than "read the source".
Put the whole message into the comments, and all is just fine.
Good idea. I hate separating implementation notes from the code by more than absolutely necessary (Zope's cobweb of Wikis drives me nuts :-), so I added the essence of that message to ceval.c as a big comment block.
Hey, that's just great! Guess how often I had to re-read that code, finally concluding that it is all-right that way, but always thinking that I could have saved quite some time by taking some notes :-) The hardest thing to remember always was the fact that the callee is saving the caller's state for the exceptions. I always have to go through analysis again to get it right, and I always think this is not the way it should be. but-this-keeps-me-young -- cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Terry> [explanation of traceback info storage] >> Great! Hope this message has shed some additional light. Terry> It would be a shame for this to be lost in the archives. If Terry> there were a directory of ImplementationNotes somewhere (or an Terry> interpreter wiki), this would belong there. Feel free to add it to http://manatee.mojam.com/pyvmwiki Skip
On Tue, 25 Feb 2003, Guido van Rossum wrote:
p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO.
I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards.
Watch out though. There are situations where an exception needs to be stored but no frame is available (when executing purely in C). There is always a thread state.
Absolutely. I'm looking to store exception information into the right frame when it is currently stored into the current frame. This won't change how exceptions are also stored in the thread state. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Christian Tismer wrote: [me, lots of rubbish, ending up in...]
Please don't disappoint me by defending a questionable design.
Oops, I'm very sorry about that. I again tried to clone myself, but now that I see the mess that the tis-bot wrote last night, I have to put him down again. I will give him more unit tests, next time :-) ciao - chris (the oiginal) -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer wrote:
Guido van Rossum wrote:
If I understand correctly what you are referring to, this is a feature. You're supposed to say:
try: return f() except: pass sys.exc_info()
i.e. the exc_info() must remain accessible.
Proposal: changing sys.exc_info into def exc_info(reset=False): ## if reset is true, also clear the error. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
participants (7)
-
barry＠python.org -
Christian Tismer -
Guido van Rossum -
Kevin Jacobs -
Mark Hammond -
Skip Montanaro -
Terry Reedy