[Distutils] setuptools fails to install package data from ZIP distribution

[Phillip J. Eby]

At 03:39 PM 9/26/2007 +0200, Philipp von Weitershausen wrote:
>Packages with Zope software typically contain lots of package data
>(templates, resources, translations, configuration files). Last night we
>had somebody with a Windows machine make distributions of some of our
>eggs (using python setup.py sdist) for the first time. It turns out,
>'sdist' on Windows creates a ZIP file while on other platforms it
>creates a gzipped tarball. I find that difference in behaviour a bit
>disturbing.
>
>That behaviour isn't the actual root of the problem we experienced,
>however. These ZIP distributions were then uploaded to PyPI and other
>people happily started installing them. The only problem was that on
>Linux and MacOSX, the installed eggs no longer contained any of the
>package data. The distributions (the ZIP files) do contain it, but
>somehow setuptools fails to install it properly.
>
>Is this a known bug?

It is now.  :)


>Is there a reason for the different default behaviour of 'sdist' on the
>different platforms?

The Windows-created SOURCES.txt contains \r characters, which when 
read on non-Windows platforms causes setuptools to look for files 
with a '\r' on the end, due to this piece of code in the distutils 
"sdist" command (which is reused by setuptools):

     def read_manifest (self):
         """Read the manifest file (named by 'self.manifest') and use it to
         fill in 'self.filelist', the list of files to include in the source
         distribution.
         """
         log.info("reading manifest file '%s'", self.manifest)
         manifest = open(self.manifest)
         while 1:
             line = manifest.readline()
             if line == '':              # end of file
                 break
             if line[-1] == '\n':
                 line = line[0:-1]
             self.filelist.append(line)

This code was apparently a platform-safety issue with the distutils 
all along, but it only affects setuptools because we actually ship 
and use the manifest (as SOURCES.txt) with the source distribution.

I've checked in a fix for both the 0.7 trunk and the 0.6 branch; you 
can update with "easy_install setuptools==dev" or "easy_install 
setuptools==dev06".

The fix is a "belt and braces" one; it ensures that Windows will now 
write SOURCES.txt in a way that unfixed setuptools can read it, AND 
it ensures that non-Windows setuptools can read SOURCES.txt written 
by an unfixed setuptools on Windows.  In other words, as long as 
either the sdist creator or the person installing the sdist have the 
fix, it will work.