[Python-Dev] Pre-PEP: Redesigning extension modules

Nick Coghlan ncoghlan at gmail.com
Sun Sep 1 16:10:08 CEST 2013

On 1 September 2013 23:03, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 1 Sep 2013 11:28:36 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
>> in order to support reloading. All module state (including type
>> cross-references) is stored in hidden state (e.g. an instance of a custom
>> type not exposed to Python, with a reference stored on each custom type
>> object defined in the module, and any module level "functions" actually
>> being methods of a hidden object). Still doesn't support loading a *fresh*
>> copy due to the hidden PEP 3121 module cache.
> Not sure what you mean by that:
>>>> import atexit
>>>> id(atexit)
> 140031896222680
>>>> import sys
>>>> del sys.modules['atexit']
>>>> import atexit
>>>> id(atexit)
> 140031896221400

Ah, you're right - I misremembered the exact problem that broke
xml.etree.ElementTree testing. PyModule_GetState is actually fine
(since that pointer is hidden state on the module object), it's only
PyState_GetModule that is broken when you import a second copy. So,
here, when the second import happens, it breaks the original atexit
module's callbacks, even though the two callback registries are
properly isolated:

$ ./python
Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import atexit
>>> atexit.register(print, "Hello World!")
<built-in function print>
>>> import sys
>>> del sys.modules["atexit"]
>>> import atexit as atexit2
>>> atexit2.register(print, "Goodbye World!")
<built-in function print>
Goodbye World!

So I think PEP 3121 is actually as good as we can get on the hidden
state front, but the important point is that it is the
*PyState_GetModule* API that can't handle fresh imports - the second
import will always replace the first one. So anyone affected needs to
find some other way of passing the state, like using bound methods of
a hidden type rather than ordinary callables. If you have to
interoperate with a C API that only accepts a C callback without
allowing additional state arguments, you're going to have trouble.

I think atexit serves as a good example, though - that _Py_PyAtExit
call will *always* be destructive (even if you still have a reference
to the original module), so there should be a way for the module to
explicitly indicate to the import system "you can only create this
module once, and then you're committed - unloading it and importing it
again won't work properly due to side effects on the process state".


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list