Hi everyone, I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope. *Conceptual TL;DR: *Add a function which attaches a string label to the current interpreter frame. (This would go away when you either explicitly cleared it, or the frame ended, i.e. the current function returned) You can then use it for: - *Debugging: *Print this value out, if set, in the traceback formatters. This can be used e.g. in a server, by putting a unique request identifier as the label for the top-level handler method; then if something in the particular query triggers a crash, it's *much* easier to see what caused it. (When I was at Google, we did something very similar to this for the servers in search. It completely transformed the way we detected "queries of death," and made hunting down data-dependent bugs an order of magnitude easier.) - *CPU and heap profiling: *If a profiler or tracer stores this information when it's grabbing the stack trace, this can be used for two different kinds of analysis, depending on user needs: - *Splitting: *Consider two stack traces different if they differ in any of the labels. This lets you separate flows which seem to be going through the same part of the logic, but are actually doing it on very distinct paths. - *Joining: *Group stack traces by the label; this lets you identify "call with this label value and its descendants," which lets you very easily establish a user-defined aggregation for total CPU or heap usage. - *Network request tracing: *If you were feeling really fancy, you could attach this to a cross-network-request trace ID and propagate the same value of this across multiple servers in a complicated request chain. Then you could perform all of the above joinings, plus similar joinings from cross-server profiling systems, as well. *Implementation: *Pretty simple: add a new field PyObject *f_label to PyFrameObject which contains an optional Unicode string; the frame owns a reference. Add get and set methods, exposing those in the Python API, and probably also a simple context manager to set and restore the value of this field because that's going to be the 90% way it's used. (Based on experience with doing similar things in C++) Modify the traceback, profile, cProfile, and tracemalloc libraries (and maybe others?) to use this value as well. What do people think? Yonatan
On Jul 17, 2019, at 18:41, Yonatan Zunger <zunger@humu.com> wrote:
I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope.
Conceptual TL;DR: Add a function which attaches a string label to the current interpreter frame. (This would go away when you either explicitly cleared it, or the frame ended, i.e. the current function returned)
Can you get 90% of this by creating a global WeakKeyDictionary mapping frames to names, and monkey patching the relevant modules to use it? It might be too inefficient for some uses, but it seems like it should be good enough to demo the idea and show off some of the ways it could be used.
On Jul 17, 2019, at 20:23, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Actually, it looks like frame objects can’t be weakref’d. So, I guess a regular dict, and a settrace function that removes the entry on return is probably the best way to do this in current Python? Still probably good enough for demo purposes.
Even that isn't so simple, because these need to vanish when the frame does (you wouldn't want the dict to hold a reference to the frame!). Also, most of the libraries that would be using this (cprofile, tracemalloc, traceback, the new profiler I'm working on) are in C, so it wouldn't be a straightforward monkeypatch. It would probably be easier to do a real implementation than a demo. What would the goal of an effective demo be? On Wed, Jul 17, 2019 at 9:07 PM Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 18, 2019, at 14:50, Yonatan Zunger <zunger@humu.com> wrote:
Even that isn't so simple, because these need to vanish when the frame does (you wouldn't want the dict to hold a reference to the frame!).
Yes, and settrace takes care of that: the dict _can_ hold a reference to a frame, because you get a “return” trace action exactly when a frame is going away, so you can remove the reference then. (That’s also exactly what weakrefs are for, but unfortunately you can’t weakref a frame, which is why I suggested settrace.) Obviously settrace has a performance cost. Plus, using it for scope painting interferes with using it for anything else, which could be a problem for some of the kinds of uses you’re talking about. So, it’s not a production solution. But considering how trivial it should be slap together something that works in Python 3.7, it may be good enough for a demo.
Also, most of the libraries that would be using this (cprofile, tracemalloc, traceback, the new profiler I'm working on) are in C, so it wouldn't be a straightforward monkeypatch.
traceback is in Python. tracemalloc is also in Python (with C support from _tracemalloc, but the stuff you want to patch is in Python). cProfile is in C, but you can use profile instead, which is in Python. (On this one, the performance may mean it’s not acceptable for many uses, but it would still be usable for some—which is why profile is there, after all. In fact, “easier to extend” is its documented reason for existence.) Whatever new profiler you’re working on is new code not part of Python 3.7, so it doesn’t need to be monkeypatched in the first place. Also, you don’t have to patch _everyrhing_ for a demo, just the modules needed for that demo. Something that stashes network peer names in tracebacks for logging (I think that was one of your uses?) will work fine without patching profile or tracemalloc, right? Maybe it would be easier for me to just slap together a quick&dirty prototype than to try to explain it?
What would the goal of an effective demo be?
To show people actual output rather than describing it in general terms and hoping they can see why it might be helpful. To let them use it on their own code and see whether it can do what they want from it. All the usual reasons a demo is useful when trying to sell people on a feature.
On Thu, Jul 18, 2019 at 3:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Nope, I see what you're saying now; with settrace it would make sense. (Although I'm still not sure it would work for the cases I'm thinking of; the C helpers are doing a lot of the lifting in this case)
I guess the real question is, who would the intended audience for this demo be, and how might they want it presented? (I've never built a demo for this audience before so I want to make sure I cover the things people care about)
Op 18 jul. 2019 om 03:41 heeft Yonatan Zunger <zunger@humu.com> het volgende geschreven:
Hi everyone,
I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope.
Would contextvars as introduced in 3.7 be helpful for you? This seems to cover at least some of the use cases you mention. Ronald
On Jul 17, 2019, at 18:41, Yonatan Zunger <zunger@humu.com> wrote:
I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope.
Conceptual TL;DR: Add a function which attaches a string label to the current interpreter frame. (This would go away when you either explicitly cleared it, or the frame ended, i.e. the current function returned)
Can you get 90% of this by creating a global WeakKeyDictionary mapping frames to names, and monkey patching the relevant modules to use it? It might be too inefficient for some uses, but it seems like it should be good enough to demo the idea and show off some of the ways it could be used.
On Jul 17, 2019, at 20:23, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Actually, it looks like frame objects can’t be weakref’d. So, I guess a regular dict, and a settrace function that removes the entry on return is probably the best way to do this in current Python? Still probably good enough for demo purposes.
Even that isn't so simple, because these need to vanish when the frame does (you wouldn't want the dict to hold a reference to the frame!). Also, most of the libraries that would be using this (cprofile, tracemalloc, traceback, the new profiler I'm working on) are in C, so it wouldn't be a straightforward monkeypatch. It would probably be easier to do a real implementation than a demo. What would the goal of an effective demo be? On Wed, Jul 17, 2019 at 9:07 PM Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 18, 2019, at 14:50, Yonatan Zunger <zunger@humu.com> wrote:
Even that isn't so simple, because these need to vanish when the frame does (you wouldn't want the dict to hold a reference to the frame!).
Yes, and settrace takes care of that: the dict _can_ hold a reference to a frame, because you get a “return” trace action exactly when a frame is going away, so you can remove the reference then. (That’s also exactly what weakrefs are for, but unfortunately you can’t weakref a frame, which is why I suggested settrace.) Obviously settrace has a performance cost. Plus, using it for scope painting interferes with using it for anything else, which could be a problem for some of the kinds of uses you’re talking about. So, it’s not a production solution. But considering how trivial it should be slap together something that works in Python 3.7, it may be good enough for a demo.
Also, most of the libraries that would be using this (cprofile, tracemalloc, traceback, the new profiler I'm working on) are in C, so it wouldn't be a straightforward monkeypatch.
traceback is in Python. tracemalloc is also in Python (with C support from _tracemalloc, but the stuff you want to patch is in Python). cProfile is in C, but you can use profile instead, which is in Python. (On this one, the performance may mean it’s not acceptable for many uses, but it would still be usable for some—which is why profile is there, after all. In fact, “easier to extend” is its documented reason for existence.) Whatever new profiler you’re working on is new code not part of Python 3.7, so it doesn’t need to be monkeypatched in the first place. Also, you don’t have to patch _everyrhing_ for a demo, just the modules needed for that demo. Something that stashes network peer names in tracebacks for logging (I think that was one of your uses?) will work fine without patching profile or tracemalloc, right? Maybe it would be easier for me to just slap together a quick&dirty prototype than to try to explain it?
What would the goal of an effective demo be?
To show people actual output rather than describing it in general terms and hoping they can see why it might be helpful. To let them use it on their own code and see whether it can do what they want from it. All the usual reasons a demo is useful when trying to sell people on a feature.
On Thu, Jul 18, 2019 at 3:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Nope, I see what you're saying now; with settrace it would make sense. (Although I'm still not sure it would work for the cases I'm thinking of; the C helpers are doing a lot of the lifting in this case)
I guess the real question is, who would the intended audience for this demo be, and how might they want it presented? (I've never built a demo for this audience before so I want to make sure I cover the things people care about)
Op 18 jul. 2019 om 03:41 heeft Yonatan Zunger <zunger@humu.com> het volgende geschreven:
Hi everyone,
I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope.
Would contextvars as introduced in 3.7 be helpful for you? This seems to cover at least some of the use cases you mention. Ronald
participants (3)
-
Andrew Barnert -
Ronald Oussoren -
Yonatan Zunger