Extending traceback to (optionally) format and show locals.

For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936. I have two questions I'm hoping to get answered through this thread: - does the change in question need a PEP? Antoine seemed to think it didn't, as long as it was opt-in (via the unittest CLI etc) - Is my implementation approach sound (for traceback, unittest I think I have covered :))? Implementation wise, I think its useful to work in the current traceback module layout - that is to alter extract_stack to (optionally) include rendered data about locals and then look for that and format it in format_list. I'm sure there is code out there that depends on the quadruple nature of extract_stack though, so I think we need to preserve that. Three strategies occured to me; one is to have parallel functions, one quadruple, one quintuple. A second one is to have the return value of extract_stack be a quintuple when a new keyword parameter include_locals is included. Lastly, and this is my preferred one, if we return a tuple subclass with an attribute containing a dict with the rendered data on the locals; this can be present but None, or even just absent when extract_stack was not asked to include locals. The last option is my preferred one because the other two both imply having a data structure which is likely to break existing code - and while you'd have to opt into having them it seems likely to require a systematic set of upgrades vs having an optional attribute that can be looked up. So - thoughts? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On Nov 26, 2014, at 15:45, Robert Collins <robertc@robertcollins.net> wrote:
There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?

On 27 November 2014 at 14:12, Andrew Barnert <abarnert@yahoo.com> wrote:
On Nov 26, 2014, at 15:45, Robert Collins <robertc@robertcollins.net> wrote:
Because backwards compatibility. Moving to a namedtuple is fine - changing the length of the tuple is a problem. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On Nov 27, 2014, at 16:48, Robert Collins <robertc@robertcollins.net> wrote:
That's the whole point: you're _not_ changing the length. Again, look at the examples that are already in the stdlib: they have a fixed length as a tuple, with extra fields accessible by name only. And it's dead easy to do with structseq.

On Thu, Nov 27, 2014, at 19:48, Robert Collins wrote:
Er, but what is being suggested is to do the same backwards-compatible thing: move to a namedtuple-like object with extra non-tuple fields, just like those others. I'm confused as to what is the conflict here.

On 1 December 2014 at 10:30, <random832@fastmail.us> wrote:
The thing I was missing is that Andrew was referring to a C only API - AFAICT there is no Python equivalent to PyStructSequence (other than implementing __len__ etc oneself - which is fine, but its not structseq then, AIUI. NamedTuple would imply changing the length - and there's no reason to reimplement traceback as C, so I'd rather not do that. Anyhow, looks like there is a strong desire for a fresh API anyway in 17911, so I'm just going to do that. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On 27 November 2014 at 09:45, Robert Collins <robertc@robertcollins.net> wrote:
For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936.
Another useful bit of context is the current RFE to have a way to extract and save a traceback *summary* that omits the local variable details: http://bugs.python.org/issue17911 Our inclination to resolve that one was to design a new higher level traceback manipulation API, which seems relevant to this proposal as well.
I agree this isn't a PEP level change, just a normal RFE. That said, if the API design details get too confusing, a PEP may still end up being a useful way of clarifying specific details.
For 17911, we're not so sure about that - there's a strong case to be made for exposing a new object-oriented API, rather than continuing with the C-style "records + functions that work on them" model that the traceback module currently uses. We made similar additions over the last couple of releases for both inspect (via inspect.Signature & inspect.Parameter) and dis (via dis.ByteCode & dis.Instruction).
Fourth, expand the new ExceptionSummary API proposed in 17911 to also include the ability to *optionally* preserve the local variable data from the stack frame, rather than always discarding it.
I don't see a lot of value in adhering too strictly to the current API design model - I think it would make more sense to design a new higher level API, and then look at including some elements to make it easy to adopt the new tools in existing code without having to rewrite the world (e.g. the inspect module work in Python 3.4 that switched the legacy APIs to actually call the new inspect.signature API internally, greatly expanding the scope of the types they could handle). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 November 2014 at 16:08, Nick Coghlan <ncoghlan@gmail.com> wrote:
AIUI the specific desire is to allow a minimal cost capture of tracebacks without frames (so as to allow gc), with enough data to render a traceback later should the thing turn out to escape the system. So largely skipping linecache lookup etc. I think thats a good thing to do :).
There's a nice functional feel there more than a C thing, IMO - in that there is no hidden state, and everything is laid out for direct view. OTOH I've no objection to a more objects-and-methods feel, though we'll want a thunk through to the new code which will essentially just be constructing objects just-in-time. Seems straight forward enough to write though.
A couple of other related things I should be clear about: I want something I can backport successfully via traceback2, for use in unittest2. I don't see any issue with the proposal so far, other than the linecache API change needed to support __loader__, which implies a linecache2 backport as well.
Sure. AIUI noone is actively pushing on the new thing, so I'll put my hand up for it now and we'll see where I get to in my available cycles. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On Nov 26, 2014, at 15:45, Robert Collins <robertc@robertcollins.net> wrote:
There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?

On 27 November 2014 at 14:12, Andrew Barnert <abarnert@yahoo.com> wrote:
On Nov 26, 2014, at 15:45, Robert Collins <robertc@robertcollins.net> wrote:
Because backwards compatibility. Moving to a namedtuple is fine - changing the length of the tuple is a problem. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On Nov 27, 2014, at 16:48, Robert Collins <robertc@robertcollins.net> wrote:
That's the whole point: you're _not_ changing the length. Again, look at the examples that are already in the stdlib: they have a fixed length as a tuple, with extra fields accessible by name only. And it's dead easy to do with structseq.

On Thu, Nov 27, 2014, at 19:48, Robert Collins wrote:
Er, but what is being suggested is to do the same backwards-compatible thing: move to a namedtuple-like object with extra non-tuple fields, just like those others. I'm confused as to what is the conflict here.

On 1 December 2014 at 10:30, <random832@fastmail.us> wrote:
The thing I was missing is that Andrew was referring to a C only API - AFAICT there is no Python equivalent to PyStructSequence (other than implementing __len__ etc oneself - which is fine, but its not structseq then, AIUI. NamedTuple would imply changing the length - and there's no reason to reimplement traceback as C, so I'd rather not do that. Anyhow, looks like there is a strong desire for a fresh API anyway in 17911, so I'm just going to do that. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On 27 November 2014 at 09:45, Robert Collins <robertc@robertcollins.net> wrote:
For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936.
Another useful bit of context is the current RFE to have a way to extract and save a traceback *summary* that omits the local variable details: http://bugs.python.org/issue17911 Our inclination to resolve that one was to design a new higher level traceback manipulation API, which seems relevant to this proposal as well.
I agree this isn't a PEP level change, just a normal RFE. That said, if the API design details get too confusing, a PEP may still end up being a useful way of clarifying specific details.
For 17911, we're not so sure about that - there's a strong case to be made for exposing a new object-oriented API, rather than continuing with the C-style "records + functions that work on them" model that the traceback module currently uses. We made similar additions over the last couple of releases for both inspect (via inspect.Signature & inspect.Parameter) and dis (via dis.ByteCode & dis.Instruction).
Fourth, expand the new ExceptionSummary API proposed in 17911 to also include the ability to *optionally* preserve the local variable data from the stack frame, rather than always discarding it.
I don't see a lot of value in adhering too strictly to the current API design model - I think it would make more sense to design a new higher level API, and then look at including some elements to make it easy to adopt the new tools in existing code without having to rewrite the world (e.g. the inspect module work in Python 3.4 that switched the legacy APIs to actually call the new inspect.signature API internally, greatly expanding the scope of the types they could handle). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 November 2014 at 16:08, Nick Coghlan <ncoghlan@gmail.com> wrote:
AIUI the specific desire is to allow a minimal cost capture of tracebacks without frames (so as to allow gc), with enough data to render a traceback later should the thing turn out to escape the system. So largely skipping linecache lookup etc. I think thats a good thing to do :).
There's a nice functional feel there more than a C thing, IMO - in that there is no hidden state, and everything is laid out for direct view. OTOH I've no objection to a more objects-and-methods feel, though we'll want a thunk through to the new code which will essentially just be constructing objects just-in-time. Seems straight forward enough to write though.
A couple of other related things I should be clear about: I want something I can backport successfully via traceback2, for use in unittest2. I don't see any issue with the proposal so far, other than the linecache API change needed to support __loader__, which implies a linecache2 backport as well.
Sure. AIUI noone is actively pushing on the new thing, so I'll put my hand up for it now and we'll see where I get to in my available cycles. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
participants (4)
-
Andrew Barnert
-
Nick Coghlan
-
random832@fastmail.us
-
Robert Collins