[Python-Dev] SVN rev. 85392 broke module handling in py3k

[Stefan Behnel]

Benjamin Peterson, 22.10.2010 16:17:
> 2010/10/22 Stefan Behnel:
>> Benjamin Peterson, 22.10.2010 16:03:
>>> 2010/10/22 Stefan Behnel:
>>>>
>>>> since SVN rev. 85392, Cython's installation fails on the py3k branch with
>>>> a
>>>> weird globals error. I think it is related to some sys.modules magic that
>>>> we
>>>> do in order to support running Cython in Python 3 using lib2to3.
>>>>
>>>> Basically, what we do is, we import some parts of Cython at the beginning
>>>> that are Py3 clean, specifically some distutils build_ext replacement for
>>>> building Cython modules. Then we start up distutils, which first runs
>>>> lib2to3 on Cython's sources to convert them into Py3 code. When it then
>>>> gets
>>>> to building the binary modules, we remove all Cython modules and packages
>>>> from sys.modules and reimport their 2to3-ed sources so that we can run
>>>> the
>>>> complete compiler during the installation (to bootstrap parts of Cython
>>>> into binary modules).
>>>>
>>>> Since the above revision, this process bails out with an error when
>>>> accessing "os.path" because "os" is None. The "os" module is imported
>>>> globally in our early-imported build_ext module, more or less like this:
>>>>
>>>>     import os
>>>>
>>>>     from distutils.command import build_ext as _build_ext
>>>>
>>>>     class build_ext(_build_ext.build_ext):
>>>>
>>>>         def build_extensions(self):
>>>>             print(os) # prints None!
>>>>
>>>> I suspect that the fact that we remove the modules from sys.modules
>>>> somehow
>>>> triggers the cleanup of these modules while there are still objects from
>>>> these modules alive that refer to their globals. So, what I think is
>>>> happening is that the module cleanup sets the module's globals to None
>>>> before the objects from that module that refer to these globals have
>>>> actually gone out of scope.
>>>>
>>>> Could someone (benjamin?) please look into this?
>>>
>>> Is this broken before 2.7, ie 2.6 and 2.6?
>>
>> I can't tell. Py2 doesn't need 2to3, so we don't unload the modules there.
>
> What about 3.1.0 then?

The 3.1.2 release was fine, but the current 3.1 SVN branch is not. I assume 
you have applied the change there, too?

py3.1 branch:
https://sage.math.washington.edu:8091/hudson/job/cython-devel-build-py31/524/console

py3k branch:
https://sage.math.washington.edu:8091/hudson/job/cython-devel-build-py3k/613/console

We have our CI tests running against both branches, as well as all Py2 
branches starting from 2.3. That's how I spotted it.

Stefan