[Python-Dev] SVN rev. 85392 broke module handling in py3k

[exarkun at twistedmatrix.com]

On 02:13 pm, stefan_ml at behnel.de wrote:
>Benjamin Peterson, 22.10.2010 16:03:
>>2010/10/22 Stefan Behnel:
>>>since SVN rev. 85392, Cython's installation fails on the py3k branch 
>>>with a
>>>weird globals error. I think it is related to some sys.modules magic 
>>>that we
>>>do in order to support running Cython in Python 3 using lib2to3.
>>>
>>>Basically, what we do is, we import some parts of Cython at the 
>>>beginning
>>>that are Py3 clean, specifically some distutils build_ext replacement 
>>>for
>>>building Cython modules. Then we start up distutils, which first runs
>>>lib2to3 on Cython's sources to convert them into Py3 code. When it 
>>>then gets
>>>to building the binary modules, we remove all Cython modules and 
>>>packages
>>>from sys.modules and reimport their 2to3-ed sources so that we can 
>>>run the
>>>complete compiler during the installation (to bootstrap parts of 
>>>Cython into
>>>binary modules).
>>>
>>>Since the above revision, this process bails out with an error when
>>>accessing "os.path" because "os" is None. The "os" module is imported
>>>globally in our early-imported build_ext module, more or less like 
>>>this:
>>>
>>>     import os
>>>
>>>     from distutils.command import build_ext as _build_ext
>>>
>>>     class build_ext(_build_ext.build_ext):
>>>
>>>         def build_extensions(self):
>>>             print(os) # prints None!
>>>
>>>I suspect that the fact that we remove the modules from sys.modules 
>>>somehow
>>>triggers the cleanup of these modules while there are still objects 
>>>from
>>>these modules alive that refer to their globals. So, what I think is
>>>happening is that the module cleanup sets the module's globals to 
>>>None
>>>before the objects from that module that refer to these globals have
>>>actually gone out of scope.

Instances of classes don't refer to the module their class is defined 
in.  It seems more likely that the reason the module is garbage 
collected is that there really is nothing which refers to it anymore.

The behavior of setting the attributes of a module being freed to None 
has been in place for a long time, r85392 only restored it after a brief 
absence.

Perhaps Cython itself should keep the modules alive that it wants kept 
alive.  Alternatively, if Cython owns the code that's running into the 
zapped global, you could change it to not use globals.

Jean-Paul