setuptools fails to install package data from ZIP distribution

Packages with Zope software typically contain lots of package data (templates, resources, translations, configuration files). Last night we had somebody with a Windows machine make distributions of some of our eggs (using python setup.py sdist) for the first time. It turns out, 'sdist' on Windows creates a ZIP file while on other platforms it creates a gzipped tarball. I find that difference in behaviour a bit disturbing. That behaviour isn't the actual root of the problem we experienced, however. These ZIP distributions were then uploaded to PyPI and other people happily started installing them. The only problem was that on Linux and MacOSX, the installed eggs no longer contained any of the package data. The distributions (the ZIP files) do contain it, but somehow setuptools fails to install it properly. Is this a known bug? Is there a reason for the different default behaviour of 'sdist' on the different platforms? -- http://worldcookery.com -- Professional Zope documentation and training

In my experience, that's usually a problem that happens when you do "python setup.py sdist" using: - a setup.py expects that "find_package_data()" will find package data based on searching for things checked into version control. - a source tree that isn't checked out from version control and does not have a MANIFEST.in file. - C On Sep 26, 2007, at 9:39 AM, Philipp von Weitershausen wrote:
Packages with Zope software typically contain lots of package data (templates, resources, translations, configuration files). Last night we had somebody with a Windows machine make distributions of some of our eggs (using python setup.py sdist) for the first time. It turns out, 'sdist' on Windows creates a ZIP file while on other platforms it creates a gzipped tarball. I find that difference in behaviour a bit disturbing.
That behaviour isn't the actual root of the problem we experienced, however. These ZIP distributions were then uploaded to PyPI and other people happily started installing them. The only problem was that on Linux and MacOSX, the installed eggs no longer contained any of the package data. The distributions (the ZIP files) do contain it, but somehow setuptools fails to install it properly.
Is this a known bug?
Is there a reason for the different default behaviour of 'sdist' on the different platforms?
-- http://worldcookery.com -- Professional Zope documentation and training
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 26 Sep 2007, at 16:07 , Chris McDonough wrote:
In my experience, that's usually a problem that happens when you do "python setup.py sdist" using:
- a setup.py expects that "find_package_data()" will find package data based on searching for things checked into version control.
- a source tree that isn't checked out from version control and does not have a MANIFEST.in file.
I'm fully aware of the problem you're talking about. However, it isn't what I'm describing. In any of the circumstances you mention, the *distribution* won't include the package data. In my case, it's the installed *egg* that doesn't include package data. The distribution (created with sdist) is actually fine, it does include the package data. setuptools just doesn't install it into the egg. For example, download and unzip http://pypi.python.org/packages/ source/z/zope.app.securitypolicy/zope.app.securitypolicy-3.4.0.zip. You will see it includes all of the extra data files. But when you do easy_install zope.app.securitypolicy-3.4.0.zip (best done in a virtual-python or virtualenv), you'll find that the installed egg does not have any of the ZCML files.

At 03:39 PM 9/26/2007 +0200, Philipp von Weitershausen wrote:
Packages with Zope software typically contain lots of package data (templates, resources, translations, configuration files). Last night we had somebody with a Windows machine make distributions of some of our eggs (using python setup.py sdist) for the first time. It turns out, 'sdist' on Windows creates a ZIP file while on other platforms it creates a gzipped tarball. I find that difference in behaviour a bit disturbing.
That behaviour isn't the actual root of the problem we experienced, however. These ZIP distributions were then uploaded to PyPI and other people happily started installing them. The only problem was that on Linux and MacOSX, the installed eggs no longer contained any of the package data. The distributions (the ZIP files) do contain it, but somehow setuptools fails to install it properly.
Is this a known bug?
It is now. :)
Is there a reason for the different default behaviour of 'sdist' on the different platforms?
The Windows-created SOURCES.txt contains \r characters, which when read on non-Windows platforms causes setuptools to look for files with a '\r' on the end, due to this piece of code in the distutils "sdist" command (which is reused by setuptools): def read_manifest (self): """Read the manifest file (named by 'self.manifest') and use it to fill in 'self.filelist', the list of files to include in the source distribution. """ log.info("reading manifest file '%s'", self.manifest) manifest = open(self.manifest) while 1: line = manifest.readline() if line == '': # end of file break if line[-1] == '\n': line = line[0:-1] self.filelist.append(line) This code was apparently a platform-safety issue with the distutils all along, but it only affects setuptools because we actually ship and use the manifest (as SOURCES.txt) with the source distribution. I've checked in a fix for both the 0.7 trunk and the 0.6 branch; you can update with "easy_install setuptools==dev" or "easy_install setuptools==dev06". The fix is a "belt and braces" one; it ensures that Windows will now write SOURCES.txt in a way that unfixed setuptools can read it, AND it ensures that non-Windows setuptools can read SOURCES.txt written by an unfixed setuptools on Windows. In other words, as long as either the sdist creator or the person installing the sdist have the fix, it will work.
participants (3)
-
Chris McDonough
-
Philipp von Weitershausen
-
Phillip J. Eby