[Python-Dev] module shutdown procedure

[Stefan Behnel]

Georg Brandl, 22.07.2010 16:13:
> Am 22.07.2010 13:29, schrieb Antoine Pitrou:
>> Le jeudi 22 juillet 2010 à 07:23 -0500, Benjamin Peterson a écrit :
>>> 2010/7/22 Antoine Pitrou:
>>>> Brett Cannon  wrote:
>>>>> Basically the whole setting a module's globals to None was done before gc
>>>>> came into the language. Now that it's there it seems that it might work to
>>>>> simply let gc clean up the module itself.
>>>>
>>>> There is a patch at http://bugs.python.org/issue812369 for GC-based
>>>> module shutdown, but it doesn't actually remove the setting of module
>>>> globals to None. I think further testing and experimentation would be
>>>> required to validate it.
>>>
>>> Also, it seems to have been stalled by static globals in extension
>>> modules that the gc doesn't know about.
>>
>> Is it the reason why? With the new module creation API in 3.x, extension
>> modules should be able to handle deletion of their own internal
>> resources.
>
> Yes, but as Martin noted at the summit, nobody since went through all the
> extension modules and changed them to use a struct instead of globals.

The Cython project has had this on the agenda ever since the early days of 
3.0, but we never got around to investing the time.

http://trac.cython.org/cython_trac/ticket/173
http://trac.cython.org/cython_trac/ticket/218

We already generate optional module cleanup code as an atexit function to 
help with valgrind, so much of the code required for m_clear() is already 
there, but getting it right (i.e. making all module globals local to an 
instance) would require some effort on our side that is not currently 
considered worth it (just ask yourself what you want first: blazingly fast 
generator functions? Or Py3-only module cleanup code?).

It wouldn't be hard to do, though, and adapting Cython here would 
immediately migrate a whole bunch of extension modules to the new module 
handling API. Note that this may not mean that all of these modules will 
work out of the box. The current cleanup code is disabled by default 
because module globals can be used in finalisers, which means that the 
module cleanup can crash if the globals are cleaned up in the wrong order. 
This could be worked around, though, as we could potentially detect 
required globals at compile time and either change the order of their 
cleanup or emit a warning that there is no such order in the face of cycles.

So, as always, it's all just a matter of investing the time to get this 
implemented.

Stefan