[Cython] Debugging Cython programs with PDB

Stefan Behnel stefan_ml at behnel.de
Mon Dec 31 05:28:25 EST 2018


Hi!

Nice idea.

Prakhar Goel schrieb am 29.12.18 um 15:02:
> I wanted to test the waters on this idea.
> 
> The idea is to allow debugging Cython programs with PDB. This relies
> on making call-backs to trigger the sys.trace functionality every now
> and then. It is very similar to how profiling is handled so I figured
> that a bunch of the infra for this is already present. It involves a
> lot of overhead so it would only be enabled for a special debug mode
> but could be very useful in that capacity since it would allow using
> identical tools for both the Python and Cython modules. Additionally
> other people have built debugging tools on top of sys.settrace so some
> of the more advanced debugging facilities would also become at least
> partially available for Cython modules.
> 
> It would be a fair bit of work. I'm not asking you to do this work of
> course! Just looking for some feedback here. The rough pieces as far
> as I can tell:
> 
>> Adding in a flag for a special debug mode that calls Python's trace functions appropriately.
> 
>> code-gen for all the trace calls.

Most of this should already be in place. I wouldn't want a special flag for
it, just extend the existing tracing support (which works for coverage
analysis, mostly).


>> Ideally we want some kind of wrapped call-frame that exports the current set of variables. Cython knows quite a bit about the variables currently in scope thanks to the cdef declarations.

Not only those, it actually knows all defined names in the current scope
(except for the global scope, which is extensible at runtime).


>> Ideally we should export these over to the Python side (with appropriate wrappers perhaps for the raw-c structs?). This is probably the most work but I'm hoping that getting a MVP here wouldn't be too hard...

There is support for locals(), which does most of what you need here.

Regarding frames, Cython's tracing code uses frames already. I think the
important steps would be:

1) Make sure there is only one frame per function execution. Currently,
there are cases where we use one per source code line, which is an ugly
hack in lack of proper Cython line number reporting support. This can be
achieved by finalising the lnotab support in PR-93, and then cleaning up
the way we create code objects and frames for tracing.

https://github.com/cython/cython/pull/93

2) Extend the meta-data that the code objects (which we already create,
just mostly empty) provide for each of the functions, since that is where
PDB gets its introspection details from.

3) Make the frame refer to the current "locals", Preferably in a way that
only generates the dict on request, not for all frames. In the worst case,
that could be something to enable with a C compile time define, as we do
for tracing in general.

4) Trial and error fixing. :)

So, yeah, there is a bit of work involved, but it seems doable and worth
doing. Are you interested in giving this a try?

Stefan


More information about the cython-devel mailing list