[Python-Dev] Encoding of PyFrameObject members

[Gregory P. Smith]

To get at the Python thread state from a signal handler (using 2.7 as a
reference here; but i don't believe 3.4 has changed this part much) you
need to modify the interpreter to expose pystate.c's "autoTLSkey" and
thread.c's "struct key" as well as "keyhead" and "keymutex".

>From there, in your signal handler you must try to acquire the newly
exposed keymutex and do nothing if you were unable to get it.  If you did
acquire it (rare not to), you can walk the keyhead list looking for
autoTLSkey to find the current valid thread state.

I had an intern (hi Elena!) write a signal sampling based low overhead
Python CPU profiler based on that last summer. I believe there are still
bugs to shaken out (if they are even possible to fix... Armin's comments
are true: signal handler code is super limited). I am stating this here
because I want someone to pester me at PyCon if I haven't released our work
as a proof of concept by then. The important take away: From what I could
figure out, you need to modify the CPython interpreter to be more amenable
to such introspection.

A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs
within the Python interpreter, stdlib, and your own code will show up in
your application. So until those are fixed (hooray for Antoine's PEP!), it
may not be practical for use on production processes which is sort of the
entire point of a low overhead sampling profiler...

I'd like to get a buildbot setup that runs the testsuite while a continual
barrage of signals are being generated. We really don't stress test that
stuff (as evidence by the EINTR mishandling issues that are rampant) as
non-fatal signals are so rare for most things... until they aren't.

As a side note and encouragement: I wonder what PyPy could do for
dynamically enabled and disabled low overhead CPU profiling. (take that as
a hint that I want someone else to get extremely creative!)

-gps

On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Maciej Fijalkowski wrote:
> > However, you can't access thread
> > locals from signal handlers (since in some cases it mallocs, thread
> > locals are built lazily if you're inside the .so, e.g. if python is
> > built with --shared)
>
> You might be able to use Py_AddPendingCall to schedule
> what you want done outside the context of the signal
> handler.
>
> The call will be made by the main thread, though,
> so if you need to access the frame of whatever thread
> was running when the signal occured, you will have
> to track down its PyThreadState somehow and get the
> frame from there. Not sure what would be involved
> in doing that.
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150208/12cc99ef/attachment.html>