[Cython] Python subinterpreters support problem in v0.29

Nathaniel Smith njs at vorpus.org
Tue Dec 11 15:07:27 EST 2018

(resending to cython-devel list since my first attempt bounced)

On Tue, Dec 11, 2018, 11:56 Nathaniel Smith <njs at pobox.com wrote:

> FYI – you should be aware that subinterpreters are poorly tested (AFAIK
> ceph is the third project to try using them, ever), not well supported in
> general, and there has even been some discussion of removing them from
> CPython. For example, numpy has never supported being used with
> subinterpreters, and currently has no plans to fix this. I suspect other
> extension modules are in similar positions, but since the bugs that
> subinterpreters trigger are often really hard to detect or debug, no one
> really knows.
> On Tue, Dec 11, 2018, 11:11 Ricardo Dias <rdias at suse.com wrote:
>> Hi Cython developers,
>> In the recent Cython 0.29 version was introduced a commit [1] that
>> hinders the usage of python subinterpreters.
>> I discovered this the hard way when suddenly a component I was working
>> on started to crash. The component in question is the ceph-mgr daemon
>> from the Ceph project [2].
>> Python subinterpreters are the basic building block for the
>> plugin/module architecture of ceph-mgr. Each "manager module" runs in
>> its own python subinterpreter. Furthermore, all python bindings for the
>> client libraries of Ceph, such as librados, librbd, libcephfs, and
>> librgw, are implemented as Cython modules, and in the particular case of
>> librados, all ceph-mgr plugin modules import the rados Cython module
>> upon initialization.
>> In practice, with Cython 0.29 we can only load one module, because the
>> following modules will refuse to load.
>> After discovering this issue, we "temporarily" prevent the issue by
>> restricting the version of Cython as a dependency [3]. But we don't want
>> to keep this restriction indefinitely and would prefer a fix from the
>> Cython side.
>> Do you think it's feasible to implement a flag to disable the safe guard
>> introduced in [1]? That way we could re-enable subinterpreters at our
>> own risk.
>> [1]
>> https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50
>> [2] https://github.com/ceph/ceph
>> [3] https://github.com/ceph/ceph/pull/25328
>> --
>> Ricardo Dias
>> Senior Software Engineer - Storage Team
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>> HRB 21284
>> (AG Nürnberg)
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20181211/be8fb34c/attachment.html>

More information about the cython-devel mailing list