On 24.06.2015 10:22, Nick Coghlan wrote:
On 24 June 2015 at 15:33, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Sun, Jun 21, 2015 at 7:06 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Nick Coghlan schrieb am 21.06.2015 um 03:28:
* there may be restrictions on some extension modules that limit them to "main interpreter only" (e.g. if the extension module itself isn't thread-safe, then it will need to remain fully protected by the GIL)
Just an idea, but C extensions could opt-in to this. Calling into them has to go through some kind of callable type, usually PyCFunction. We could protect all calls to extension types and C functions with a global runtime lock (per process, not per interpreter) and Extensions could set a flag on their functions and methods (or get it inherited from their extension types etc.) that says "I don't need the lock". That allows for a very fine-grained transition.
Exactly. PEP 489 helps facilitate opting in as well, right?
Yep, as PEP 489 requires subinterpreter compatibility as a precondition for using multi-phase initialisation :)
Cheers, Nick.
P.S. Technically, what it actually requires is support for "multiple instances of the module existing in the same process at the same time", as it really recreates the module if you remove it from sys.modules and import it again, unlike single phase initialisation. But that's a mouthful, so "must support subinterpreters" is an easier shorthand.
Note that extension modules often interface to other C libraries which typically use some setup logic that is not thread safe, but is used to initialize the other thread safe parts. E.g. setting up locks and shared memory for all threads to use is a typical scenario you find in such libs. A requirement to be able to import modules multiple times would pretty much kill the idea for those modules. That said, I don't think this is really needed. Modules would only have to be made aware that there is a global first time setup phase and a later shutdown/reinit phase. As a result, the module DLL would load only once, but then use the new module setup logic to initialize its own state multiple times. That said, I still think the multiple-process is a better one (more robust, more compatible, fewer problems). We'd just need a way more efficient approach to sharing objects between the Python processes than using pickle and shared memory or pipes :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 24 2015)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015-07-20: EuroPython 2015, Bilbao, Spain ... 26 days to go 2015-07-29: Python Meeting Duesseldorf ... 35 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/