Re: [Python-ideas] raising an exception type doesn't instantiate it until it gets caught
On Tue, Nov 1, 2011 at 1:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I did not know that. Is there a good reason for doing this? It seems unnecessarily complicated. -gps
On Mon, Nov 7, 2011 at 11:38 AM, Gregory P. Smith <greg@krypto.org> wrote:
You'll have to ask Guido that one - it's been like that since long before I got involved in hacking on the interpreter. It's possibly a lingering artifact of the old "exceptions are just strings" design, since traceback storage didn't get added to exception instances until Py3k. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Nov 6, 2011 at 5:46 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
No, it was actually introduced when exceptions became classes. It is an optimization that we deemed important at the time: to avoid instantiating the class when it's going to be caught by C code that doesn't care about the instance. A common example is StopIteration, but there are probably plenty of other situations like it. We may even have benchmarked for-loops with and without this -- while the exception only happens once per for-loop, there are a lot of for-loops, many of which iterate over small sequences, and it adds up. I'm not sure that it's still that important -- in fact I'm not sure Python 3 still has this behavior. -- --Guido van Rossum (python.org/~guido)
On Mon, Nov 7, 2011 at 11:52 AM, Guido van Rossum <guido@python.org> wrote:
Ah, thanks - interesting to know.
I'm not sure that it's still that important -- in fact I'm not sure Python 3 still has this behavior.
I'm fairly sure it does (it was the 3.x version of ceval that I was reading to remind myself of how the stack unwinding process actually works and the call to PyErr_NormalizeException() is still in there). It's just that without the multiple argument forms of the raise statement, pure Python code can only exploit it for exceptions without any arguments (so it still works for the StopIteration optimisation). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 7 Nov 2011 12:03:46 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
It's too late for me to go and take a look, but I'm not sure it does. Unless the performance loss is significant, I would be in favour of simplifying all this and always instantiating exceptions. Exception handling in the eval loop and exception "normalization" is a nasty area of the interpreter. Regards Antoine.
On 07Nov2011 04:04, Antoine Pitrou <solipsis@pitrou.net> wrote: | On Mon, 7 Nov 2011 12:03:46 +1000 | Nick Coghlan <ncoghlan@gmail.com> wrote: | > > I'm not sure that it's still that important -- in fact I'm not sure Python 3 | > > still has this behavior. | > | > I'm fairly sure it does (it was the 3.x version of ceval that I was | > reading to remind myself of how the stack unwinding process actually | > works and the call to PyErr_NormalizeException() is still in there). | > It's just that without the multiple argument forms of the raise | > statement, pure Python code can only exploit it for exceptions without | > any arguments (so it still works for the StopIteration optimisation). | | It's too late for me to go and take a look, but I'm not sure it does. | Unless the performance loss is significant, I would be in favour of | simplifying all this and always instantiating exceptions. Exception | handling in the eval loop and exception "normalization" is a nasty area | of the interpreter. I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Login incorrect. Only perfect spellers may enter this system. - Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html
Cameron Simpson wrote:
I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking.
Are you asking about what it should be, or what it is? Either way:
-- Steven
I wrote, naively: | > I presume StopIteration would get instantiated to a singleton, like | > NoneType to None? Just asking. On 07Nov2011 22:01, Nick Coghlan <ncoghlan@gmail.com> wrote: | Even without the traceback issue Antoine mentioned, it's already the | case that StopIteration isn't a singleton in 2.x. Various pieces of | code (e.g. contextlib.contextmanager) rely on being able to tell | whether they're getting a specific StopIteration instance back or a | new one. Interesting. Off topic digression: I've been slightly uncomfortable about exceptions as control flow for a while, basicly when writing code like this: try: x = G.next() except StopIteration: # G is empty! in that I don't entirely know that the StopIteration came from G of from some buggy code deeper inside G that let a StopIteration out, eg by mangling a try/except like the above. In most circumstances with other exceptions, while you might _expect_ the exception to come from the source you expect you don't care so much because it will indicate failure of the operation anyway. Report or die, you don't proceed as if the op was good. But with StopIteration one is reading "G is empty" directly into the situation and acting as though it is normal (exit the event loop or whatever it may imply). On 07Nov2011 11:35, Antoine Pitrou <solipsis@pitrou.net> wrote: | It is impossible to use singletons for exception instances now that the | traceback is stored on them. Ah. I had somehow thought the exception itself and the traceback were distinct items, presented in a tuple. On 07Nov2011 21:15, Steven D'Aprano <steve@pearwood.info> wrote: | Are you asking about what it should be, or what it is? The former. | Either way: | >>> a = StopIteration('spam') | >>> b = StopIteration('ham') | >>> a is b | False Since my question was about the proposed new behaviour when just a type was raised, the above test wouldn't educate me. Though it does address the behaviour of the type instantation in general. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Carpe Datum - John Sloan <jsloan@ncar.ucar.edu>
On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs@zip.com.au> wrote:
Agreed. Use of exceptions for this in the language feels like it was a convenient way to do it but as the conditions aren't really *exceptional*at all it'd be nice if there were a lighter weight mechanism that could skip the unneeded parts of the exception raising and handling mechanism for the implementation. We don't need the traceback to be stored in these situations. This existing logic to instantiate and associate the traceback with it only if caught is one way to implement doing exactly that. Any other ideas? Hackish things like a class attribute on classes being raised as an exception, or a LightweightException class being part of its class heirarchy used to signify if that exception should take the full path or the fast path come to mind but could be considered equally surprising. I'm not sure any of this is worth it but it would simplify the eval loop. We're speaking implementation details of CPython here, not an actual change to the language itself. (*) -gps (*) Please beat anybody who writes code that depends on this somewhat odd exception instantiation timing behavior side effect over the head with a frozen herring.
On 7 November 2011 21:43, Gregory P. Smith <greg@krypto.org> wrote:
Having the interpreter instantiate the exception for you allows you to do wonderful things like this:
(I know this has nothing to do with the topic being debated but for some reason this code tickles me. Plus it used to segfault Python 3...) All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 7 November 2011 23:05, Michael Foord <fuzzyman@gmail.com> wrote:
Ooh... this one segfaults Python 3.2 - I wonder if that's been fixed yet.
All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 7 November 2011 23:06, Michael Foord <fuzzyman@gmail.com> wrote:
Yeah, fixed on 3.2 branch and trunk. Sorry for all the noise. I finally managed to get Python head to compile on OS X Lion, yay! Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mon, 7 Nov 2011 18:52:55 +1100 Cameron Simpson <cs@zip.com.au> wrote:
It is impossible to use singletons for exception instances now that the traceback is stored on them. (there was a bug with MemoryError which was a singleton: a raised memory error would make its traceback immortal, making memory pressure even worse) Regards Antoine.
On Mon, Nov 7, 2011 at 5:52 PM, Cameron Simpson <cs@zip.com.au> wrote:
I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking.
Even without the traceback issue Antoine mentioned, it's already the case that StopIteration isn't a singleton in 2.x. Various pieces of code (e.g. contextlib.contextmanager) rely on being able to tell whether they're getting a specific StopIteration instance back or a new one. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 06, 2011, at 05:52 PM, Guido van Rossum wrote:
I did some performance testing back when this was introduced. They numbers aren't relevant any more, but the basic idea was to time things like dictionary access with .get() vs. __getitem__() both when the key was in the dictionary and when it was missing. There were some other timing tests IIRC. IIRC, instantiating the exception in every case was a fairly significant hit. It would be good to see some updated tests and numbers before anything was changed in the interpreter. -Barry
On Mon, Nov 7, 2011 at 11:38 AM, Gregory P. Smith <greg@krypto.org> wrote:
You'll have to ask Guido that one - it's been like that since long before I got involved in hacking on the interpreter. It's possibly a lingering artifact of the old "exceptions are just strings" design, since traceback storage didn't get added to exception instances until Py3k. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Nov 6, 2011 at 5:46 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
No, it was actually introduced when exceptions became classes. It is an optimization that we deemed important at the time: to avoid instantiating the class when it's going to be caught by C code that doesn't care about the instance. A common example is StopIteration, but there are probably plenty of other situations like it. We may even have benchmarked for-loops with and without this -- while the exception only happens once per for-loop, there are a lot of for-loops, many of which iterate over small sequences, and it adds up. I'm not sure that it's still that important -- in fact I'm not sure Python 3 still has this behavior. -- --Guido van Rossum (python.org/~guido)
On Mon, Nov 7, 2011 at 11:52 AM, Guido van Rossum <guido@python.org> wrote:
Ah, thanks - interesting to know.
I'm not sure that it's still that important -- in fact I'm not sure Python 3 still has this behavior.
I'm fairly sure it does (it was the 3.x version of ceval that I was reading to remind myself of how the stack unwinding process actually works and the call to PyErr_NormalizeException() is still in there). It's just that without the multiple argument forms of the raise statement, pure Python code can only exploit it for exceptions without any arguments (so it still works for the StopIteration optimisation). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 7 Nov 2011 12:03:46 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
It's too late for me to go and take a look, but I'm not sure it does. Unless the performance loss is significant, I would be in favour of simplifying all this and always instantiating exceptions. Exception handling in the eval loop and exception "normalization" is a nasty area of the interpreter. Regards Antoine.
On 07Nov2011 04:04, Antoine Pitrou <solipsis@pitrou.net> wrote: | On Mon, 7 Nov 2011 12:03:46 +1000 | Nick Coghlan <ncoghlan@gmail.com> wrote: | > > I'm not sure that it's still that important -- in fact I'm not sure Python 3 | > > still has this behavior. | > | > I'm fairly sure it does (it was the 3.x version of ceval that I was | > reading to remind myself of how the stack unwinding process actually | > works and the call to PyErr_NormalizeException() is still in there). | > It's just that without the multiple argument forms of the raise | > statement, pure Python code can only exploit it for exceptions without | > any arguments (so it still works for the StopIteration optimisation). | | It's too late for me to go and take a look, but I'm not sure it does. | Unless the performance loss is significant, I would be in favour of | simplifying all this and always instantiating exceptions. Exception | handling in the eval loop and exception "normalization" is a nasty area | of the interpreter. I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Login incorrect. Only perfect spellers may enter this system. - Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html
Cameron Simpson wrote:
I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking.
Are you asking about what it should be, or what it is? Either way:
-- Steven
I wrote, naively: | > I presume StopIteration would get instantiated to a singleton, like | > NoneType to None? Just asking. On 07Nov2011 22:01, Nick Coghlan <ncoghlan@gmail.com> wrote: | Even without the traceback issue Antoine mentioned, it's already the | case that StopIteration isn't a singleton in 2.x. Various pieces of | code (e.g. contextlib.contextmanager) rely on being able to tell | whether they're getting a specific StopIteration instance back or a | new one. Interesting. Off topic digression: I've been slightly uncomfortable about exceptions as control flow for a while, basicly when writing code like this: try: x = G.next() except StopIteration: # G is empty! in that I don't entirely know that the StopIteration came from G of from some buggy code deeper inside G that let a StopIteration out, eg by mangling a try/except like the above. In most circumstances with other exceptions, while you might _expect_ the exception to come from the source you expect you don't care so much because it will indicate failure of the operation anyway. Report or die, you don't proceed as if the op was good. But with StopIteration one is reading "G is empty" directly into the situation and acting as though it is normal (exit the event loop or whatever it may imply). On 07Nov2011 11:35, Antoine Pitrou <solipsis@pitrou.net> wrote: | It is impossible to use singletons for exception instances now that the | traceback is stored on them. Ah. I had somehow thought the exception itself and the traceback were distinct items, presented in a tuple. On 07Nov2011 21:15, Steven D'Aprano <steve@pearwood.info> wrote: | Are you asking about what it should be, or what it is? The former. | Either way: | >>> a = StopIteration('spam') | >>> b = StopIteration('ham') | >>> a is b | False Since my question was about the proposed new behaviour when just a type was raised, the above test wouldn't educate me. Though it does address the behaviour of the type instantation in general. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Carpe Datum - John Sloan <jsloan@ncar.ucar.edu>
On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs@zip.com.au> wrote:
Agreed. Use of exceptions for this in the language feels like it was a convenient way to do it but as the conditions aren't really *exceptional*at all it'd be nice if there were a lighter weight mechanism that could skip the unneeded parts of the exception raising and handling mechanism for the implementation. We don't need the traceback to be stored in these situations. This existing logic to instantiate and associate the traceback with it only if caught is one way to implement doing exactly that. Any other ideas? Hackish things like a class attribute on classes being raised as an exception, or a LightweightException class being part of its class heirarchy used to signify if that exception should take the full path or the fast path come to mind but could be considered equally surprising. I'm not sure any of this is worth it but it would simplify the eval loop. We're speaking implementation details of CPython here, not an actual change to the language itself. (*) -gps (*) Please beat anybody who writes code that depends on this somewhat odd exception instantiation timing behavior side effect over the head with a frozen herring.
On 7 November 2011 21:43, Gregory P. Smith <greg@krypto.org> wrote:
Having the interpreter instantiate the exception for you allows you to do wonderful things like this:
(I know this has nothing to do with the topic being debated but for some reason this code tickles me. Plus it used to segfault Python 3...) All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 7 November 2011 23:05, Michael Foord <fuzzyman@gmail.com> wrote:
Ooh... this one segfaults Python 3.2 - I wonder if that's been fixed yet.
All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 7 November 2011 23:06, Michael Foord <fuzzyman@gmail.com> wrote:
Yeah, fixed on 3.2 branch and trunk. Sorry for all the noise. I finally managed to get Python head to compile on OS X Lion, yay! Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mon, 7 Nov 2011 18:52:55 +1100 Cameron Simpson <cs@zip.com.au> wrote:
It is impossible to use singletons for exception instances now that the traceback is stored on them. (there was a bug with MemoryError which was a singleton: a raised memory error would make its traceback immortal, making memory pressure even worse) Regards Antoine.
On Mon, Nov 7, 2011 at 5:52 PM, Cameron Simpson <cs@zip.com.au> wrote:
I presume StopIteration would get instantiated to a singleton, like NoneType to None? Just asking.
Even without the traceback issue Antoine mentioned, it's already the case that StopIteration isn't a singleton in 2.x. Various pieces of code (e.g. contextlib.contextmanager) rely on being able to tell whether they're getting a specific StopIteration instance back or a new one. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 06, 2011, at 05:52 PM, Guido van Rossum wrote:
I did some performance testing back when this was introduced. They numbers aren't relevant any more, but the basic idea was to time things like dictionary access with .get() vs. __getitem__() both when the key was in the dictionary and when it was missing. There were some other timing tests IIRC. IIRC, instantiating the exception in every case was a fairly significant hit. It would be good to see some updated tests and numbers before anything was changed in the interpreter. -Barry
participants (8)
-
Antoine Pitrou -
Barry Warsaw -
Cameron Simpson -
Gregory P. Smith -
Guido van Rossum -
Michael Foord -
Nick Coghlan -
Steven D'Aprano