[Python-Dev] Encoding of PyFrameObject members

[Maciej Fijalkowski]

I'm working on vmprof (github.com/vmprof/vmprof-python) which works
for both cpython and pypy (pypy has special support, cpython is
patched on-the fly)

On Sun, Feb 8, 2015 at 6:39 AM, Gregory P. Smith <greg at krypto.org> wrote:
> To get at the Python thread state from a signal handler (using 2.7 as a
> reference here; but i don't believe 3.4 has changed this part much) you need
> to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's
> "struct key" as well as "keyhead" and "keymutex".
>
> From there, in your signal handler you must try to acquire the newly exposed
> keymutex and do nothing if you were unable to get it.  If you did acquire it
> (rare not to), you can walk the keyhead list looking for autoTLSkey to find
> the current valid thread state.
>
> I had an intern (hi Elena!) write a signal sampling based low overhead
> Python CPU profiler based on that last summer. I believe there are still
> bugs to shaken out (if they are even possible to fix... Armin's comments are
> true: signal handler code is super limited). I am stating this here because
> I want someone to pester me at PyCon if I haven't released our work as a
> proof of concept by then. The important take away: From what I could figure
> out, you need to modify the CPython interpreter to be more amenable to such
> introspection.
>
> A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs
> within the Python interpreter, stdlib, and your own code will show up in
> your application. So until those are fixed (hooray for Antoine's PEP!), it
> may not be practical for use on production processes which is sort of the
> entire point of a low overhead sampling profiler...
>
> I'd like to get a buildbot setup that runs the testsuite while a continual
> barrage of signals are being generated. We really don't stress test that
> stuff (as evidence by the EINTR mishandling issues that are rampant) as
> non-fatal signals are so rare for most things... until they aren't.
>
> As a side note and encouragement: I wonder what PyPy could do for
> dynamically enabled and disabled low overhead CPU profiling. (take that as a
> hint that I want someone else to get extremely creative!)
>
> -gps
>
> On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
>>
>> Maciej Fijalkowski wrote:
>> > However, you can't access thread
>> > locals from signal handlers (since in some cases it mallocs, thread
>> > locals are built lazily if you're inside the .so, e.g. if python is
>> > built with --shared)
>>
>> You might be able to use Py_AddPendingCall to schedule
>> what you want done outside the context of the signal
>> handler.
>>
>> The call will be made by the main thread, though,
>> so if you need to access the frame of whatever thread
>> was running when the signal occured, you will have
>> to track down its PyThreadState somehow and get the
>> frame from there. Not sure what would be involved
>> in doing that.
>>
>> --
>> Greg
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>