[Distutils] Odd problem with distribute 0.6.26 and 2to3

[Vinay Sajip]

I'm having an odd problem with distribute and 2to3 when working with the venv 
branch of Python 3.3 (PEP 405 - http://bugs.python.org/issue14712), and I'm 
hoping one of the Fellowship of the Packaging can help.

The problem: I created a venv with no Distribute installed. Then I downloaded 
and extracted distribute-0.6.26.tar.gz in a scratch folder, and ran setup.py 
install using the venv's python. This gave errors because some of the setuptools 
sources still have 2.x syntax, despite having been processed by 2to3.

To investigate further, I added logging to distutils refactoring code to print 
what was being refactored. I won't list the entire log output [1], but when 
looking at just one file which caused a SyntaxError - 
setuptools/tests/test_packageindex.py - I see the following output from the 
"setup.py install" operation:

creating build
creating build/src
...
copying setuptools/tests/test_packageindex.py -> build/src/setuptools/tests
...
copying launcher.c -> build/src
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
refactored build/src/setuptools/script template (dev).py
...
refactored build/src/setuptools/tests/test_packageindex.py
...
refactored build/src/release.py
Before install bootstrap.
Scanning installed packages
No setuptools distribution found
running install
running build
running build_py
creating build/lib
...
copying setuptools/tests/test_packageindex.py -> build/lib/setuptools/tests
...
creating /tmp/venv/lib/python3.3/site-packages/setuptools/tests
copying build/lib/setuptools/tests/test_packageindex.py -> 
/tmp/venv/lib/python3.3/site-packages/setuptools/tests
...
running install_lib
...
creating /tmp/venv/lib/python3.3/site-packages/setuptools
creating /tmp/venv/lib/python3.3/site-packages/setuptools/tests
copying build/lib/setuptools/tests/test_packageindex.py -> 
/tmp/venv/lib/python3.3/site-packages/setuptools/tests
...
byte-compiling /tmp/venv/lib/python3.3/site-
packages/setuptools/tests/test_packageindex.py to test_packageindex.cpython-
33.pyc
...
running install_egg_info
Writing /tmp/venv/lib/python3.3/site-packages/distribute-0.6.26-py3.3.egg-info
setup.py:139: ResourceWarning: unclosed file <_io.TextIOWrapper 
name='README.txt' mode='r' encoding='UTF-8'>
  long_description = open('README.txt').read() + open('CHANGES.txt').read(),
setup.py:139: ResourceWarning: unclosed file <_io.TextIOWrapper 
name='CHANGES.txt' mode='r' encoding='UTF-8'>
  long_description = open('README.txt').read() + open('CHANGES.txt').read(),
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'test_suite'
  warnings.warn(msg)
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'entry_points'
  warnings.warn(msg)
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'zip_safe'
  warnings.warn(msg)
  File "/tmp/venv/lib/python3.3/site-
packages/setuptools/tests/test_packageindex.py", line 17
    except Exception, v:
                    ^
SyntaxError: invalid syntax

Now the first part seems straightfoward: files are copied to build/src and then 
2to3 is run over them. But then, following the "No setuptools distribution 
found", files are copied to build/lib not from the 2to3-processed output at 
build/src, but rather the actual 2.x sources shipped with distribute. That looks 
wrong; at the point where setup is called, src_root is correctly set to 
build/src, and moreover, that's the first entry on sys.path. I'm not sure why 
there's no error immediately following the "byte-compiling ..." line when 
running install_lib, but the error seems to occur a little later, when running 
install_egg_info ...  

Can anyone shed any light on what's happening here?

Regards,


Vinay Sajip

[1] https://gist.github.com/2725747