[Cython] Python subinterpreters support problem in v0.29

[Nathaniel Smith]

(resending to cython-devel list since my first attempt bounced)

On Tue, Dec 11, 2018, 11:56 Nathaniel Smith <njs at pobox.com wrote:

> FYI – you should be aware that subinterpreters are poorly tested (AFAIK
> ceph is the third project to try using them, ever), not well supported in
> general, and there has even been some discussion of removing them from
> CPython. For example, numpy has never supported being used with
> subinterpreters, and currently has no plans to fix this. I suspect other
> extension modules are in similar positions, but since the bugs that
> subinterpreters trigger are often really hard to detect or debug, no one
> really knows.
>
> On Tue, Dec 11, 2018, 11:11 Ricardo Dias <rdias at suse.com wrote:
>
>> Hi Cython developers,
>>
>> In the recent Cython 0.29 version was introduced a commit [1] that
>> hinders the usage of python subinterpreters.
>>
>> I discovered this the hard way when suddenly a component I was working
>> on started to crash. The component in question is the ceph-mgr daemon
>> from the Ceph project [2].
>>
>> Python subinterpreters are the basic building block for the
>> plugin/module architecture of ceph-mgr. Each "manager module" runs in
>> its own python subinterpreter. Furthermore, all python bindings for the
>> client libraries of Ceph, such as librados, librbd, libcephfs, and
>> librgw, are implemented as Cython modules, and in the particular case of
>> librados, all ceph-mgr plugin modules import the rados Cython module
>> upon initialization.
>>
>> In practice, with Cython 0.29 we can only load one module, because the
>> following modules will refuse to load.
>>
>> After discovering this issue, we "temporarily" prevent the issue by
>> restricting the version of Cython as a dependency [3]. But we don't want
>> to keep this restriction indefinitely and would prefer a fix from the
>> Cython side.
>>
>> Do you think it's feasible to implement a flag to disable the safe guard
>> introduced in [1]? That way we could re-enable subinterpreters at our
>> own risk.
>>
>>
>> [1]
>>
>> https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50
>> [2] https://github.com/ceph/ceph
>> [3] https://github.com/ceph/ceph/pull/25328
>>
>> --
>> Ricardo Dias
>> Senior Software Engineer - Storage Team
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>> HRB 21284
>> (AG Nürnberg)
>>
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20181211/be8fb34c/attachment.html>