[Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation

[Stefan Behnel]

Hi,

sorry for asking here instead of filing a bug, but given that 3.2 final is 
pretty close, I wanted to make sure this gets considered.

A Cython user noticed that the installation (setup.py install or bdist) 
puts several .pyc files into the installed source directory, instead of 
moving them into __pycache__. The latter is still used, and most of the 
.pyc files end up there, but not all of them. Some even end up in both 
directories, once with a ".cpython-32.pyc" suffix in __pycache__ and simply 
as ".pyc" next to the sources.

A specialty of the Cython installation is that a tiny part of Cython gets 
imported at the beginning, then setup() is called. Next, 2to3 is run over 
the sources, and during the extension building phase, the initially 
imported part is removed from sys.modules and Cython is imported completely 
from the converted sources and runs to compile parts of itself. Finally, 
the installation continues and copies/byte-compiles the 2to3 converted .py 
files.

What I think is happening here, is that the normal import mechanism byte 
compiles the 2to3 converted sources into the __pycache__ directory (when 
invoked at extension building time), but then distutils' byte compilation 
seems to decide that it's better to keep the .pyc files where the sources 
are. And thus a mix of both ends up in the installation.

Wouldn't it be better to let distutils *always* byte-compile the .py files 
into the appropriate __pycache__ directory, just like the import mechanism 
does? Or is there something else broken here?

Stefan