Mailman 3 December 2018 - cython-devel

Debugging Cython programs with PDB
by Prakhar Goel Dec. 31, 2018

Dec. 31, 2018

I wanted to test the waters on this idea. The idea is to allow debugging Cython programs with PDB. This relies on making call-backs to trigger the sys.trace functionality every now and then. It is very similar to how profiling is handled so I figured that a bunch of the infra for this is already present. It involves a lot of overhead so it would only be enabled for a special debug mode but could be very useful in that capacity since it would allow using identical tools for both the Python and Cython modules. Additionally other people have built debugging tools on top of sys.settrace so some of the more advanced debugging facilities would also become at least partially available for Cython modules. It would be a fair bit of work. I'm not asking you to do this work of course! Just looking for some feedback here. The rough pieces as far as I can tell: > Adding in a flag for a special debug mode that calls Python's trace functions appropriately. > code-gen for all the trace calls. > Ideally we want some kind of wrapped call-frame that exports the current set of variables. Cython knows quite a bit about the variables currently in scope thanks to the cdef declarations. Ideally we should export these over to the Python side (with appropriate wrappers perhaps for the raw-c structs?). This is probably the most work but I'm hoping that getting a MVP here wouldn't be too hard... Thoughts? Comments? Suggestions? Thanks. -- ________________________ Warm Regards Prakhar Goel

3 2

Python subinterpreters support problem in v0.29
by Ricardo Dias Dec. 12, 2018

Dec. 12, 2018

Hi Cython developers, In the recent Cython 0.29 version was introduced a commit [1] that hinders the usage of python subinterpreters. I discovered this the hard way when suddenly a component I was working on started to crash. The component in question is the ceph-mgr daemon from the Ceph project [2]. Python subinterpreters are the basic building block for the plugin/module architecture of ceph-mgr. Each "manager module" runs in its own python subinterpreter. Furthermore, all python bindings for the client libraries of Ceph, such as librados, librbd, libcephfs, and librgw, are implemented as Cython modules, and in the particular case of librados, all ceph-mgr plugin modules import the rados Cython module upon initialization. In practice, with Cython 0.29 we can only load one module, because the following modules will refuse to load. After discovering this issue, we "temporarily" prevent the issue by restricting the version of Cython as a dependency [3]. But we don't want to keep this restriction indefinitely and would prefer a fix from the Cython side. Do you think it's feasible to implement a flag to disable the safe guard introduced in [1]? That way we could re-enable subinterpreters at our own risk. [1] https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8… [2] https://github.com/ceph/ceph [3] https://github.com/ceph/ceph/pull/25328 -- Ricardo Dias Senior Software Engineer - Storage Team SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

2 3

Re: [Cython] Python subinterpreters support problem in v0.29
by Nathaniel Smith Dec. 11, 2018

Dec. 11, 2018

(resending to cython-devel list since my first attempt bounced) On Tue, Dec 11, 2018, 11:56 Nathaniel Smith <njs(a)pobox.com wrote: > FYI – you should be aware that subinterpreters are poorly tested (AFAIK > ceph is the third project to try using them, ever), not well supported in > general, and there has even been some discussion of removing them from > CPython. For example, numpy has never supported being used with > subinterpreters, and currently has no plans to fix this. I suspect other > extension modules are in similar positions, but since the bugs that > subinterpreters trigger are often really hard to detect or debug, no one > really knows. > > On Tue, Dec 11, 2018, 11:11 Ricardo Dias <rdias(a)suse.com wrote: > >> Hi Cython developers, >> >> In the recent Cython 0.29 version was introduced a commit [1] that >> hinders the usage of python subinterpreters. >> >> I discovered this the hard way when suddenly a component I was working >> on started to crash. The component in question is the ceph-mgr daemon >> from the Ceph project [2]. >> >> Python subinterpreters are the basic building block for the >> plugin/module architecture of ceph-mgr. Each "manager module" runs in >> its own python subinterpreter. Furthermore, all python bindings for the >> client libraries of Ceph, such as librados, librbd, libcephfs, and >> librgw, are implemented as Cython modules, and in the particular case of >> librados, all ceph-mgr plugin modules import the rados Cython module >> upon initialization. >> >> In practice, with Cython 0.29 we can only load one module, because the >> following modules will refuse to load. >> >> After discovering this issue, we "temporarily" prevent the issue by >> restricting the version of Cython as a dependency [3]. But we don't want >> to keep this restriction indefinitely and would prefer a fix from the >> Cython side. >> >> Do you think it's feasible to implement a flag to disable the safe guard >> introduced in [1]? That way we could re-enable subinterpreters at our >> own risk. >> >> >> [1] >> >> https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8… >> [2] https://github.com/ceph/ceph >> [3] https://github.com/ceph/ceph/pull/25328 >> >> -- >> Ricardo Dias >> Senior Software Engineer - Storage Team >> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, >> HRB 21284 >> (AG Nürnberg) >> >> >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel(a)python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> >

1 0