setuptools and sdist on Mac OS X, tarfile flukes
Not sure if anyone reported a similar issue here since I'm not subscribed, but here it goes. Some of the new plone.* eggs are being built on OS X by the developers, they usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py sdist upload' Turns out that if you try to easy_install and there's only the sdist package, but no egg, easy_install uses tarfile to unpack the package, build an egg and install the egg. However, for some reason some of those nasty '._<filename>' files that OS X creates end up in the tarball. And tarfile then fails to unpack the tarball. One such package is 'plone.app.controlpanel 1.0a2' (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone....), with the following traceback: C:\DOCUME~1\Sidnei\LOCALS~1\Temp>easy_install -f . plone.app.controlpanel Searching for plone.app.controlpanel Best match: plone.app.controlpanel 1.0a2 Processing plone.app.controlpanel-1.0a2.tar.gz Traceback (most recent call last): File "c:\Python24\Scripts\easy_install-script.py", line 7, in ? sys.exit( File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 1670, in main with_ei_usage(lambda: File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 1659, in with_ei_usage return f() File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 1674, in <lambda> distclass=DistributionWithoutHelpCommands, **kw File "C:\Python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Python24\lib\distutils\dist.py", line 946, in run_commands self.run_command(cmd) File "C:\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 211, in run self.easy_install(spec, not self.no_deps) File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 446, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 471, in install_item dists = self.install_eggs(spec, download, tmpdir) File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\comm and\easy_install.py", line 626, in install_eggs unpack_archive(dist_filename, tmpdir, self.unpack_progress) File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\arch ive_util.py", line 67, in unpack_archive driver(filename, extract_dir, progress_filter) File "c:\python24\lib\site-packages\setuptools-0.6c5-py2.4.egg\setuptools\arch ive_util.py", line 192, in unpack_tarfile tarobj._extract_member(member,dst) # XXX Ugh File "C:\Python24\lib\tarfile.py", line 1440, in _extract_member self.utime(tarinfo, targetpath) File "C:\Python24\lib\tarfile.py", line 1572, in utime raise ExtractError, "could not change modification time" tarfile.ExtractError: could not change modification time I'm left wondering if the issue is: a) Windows b) The nasty OS X file c) The Python 'tarfile' module d) setuptools e) None of the above :) Thoughts? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
Not sure if anyone reported a similar issue here since I'm not subscribed, but here it goes.
Some of the new plone.* eggs are being built on OS X by the developers, they usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py sdist upload'
Turns out that if you try to easy_install and there's only the sdist package, but no egg, easy_install uses tarfile to unpack the package, build an egg and install the egg. However, for some reason some of those nasty '._<filename>' files that OS X creates end up in the tarball. And tarfile then fails to unpack the tarball. One such package is 'plone.app.controlpanel 1.0a2' (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone....), with the following traceback: ... I'm left wondering if the issue is:
a) Windows b) The nasty OS X file c) The Python 'tarfile' module d) setuptools e) None of the above :)
It appears to be a combination of "a" and "c". Windows Python 2.5 does not have an issue with the above file, nor does Cygwin Python 2.3. Windows Python 2.3 and 2.4 do have the problem.
Thoughts?
Dunno. It sounds like the people making these distributions have an over-eager MANIFEST.in or an over-eager source control system. They should probably add a MANIFEST.in rule to exclude the files from the source release.
Hi, Phillip J. Eby wrote:
At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
Not sure if anyone reported a similar issue here since I'm not subscribed, but here it goes.
Some of the new plone.* eggs are being built on OS X by the developers, they usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py sdist upload'
Turns out that if you try to easy_install and there's only the sdist package, but no egg, easy_install uses tarfile to unpack the package, build an egg and install the egg. However, for some reason some of those nasty '._<filename>' files that OS X creates end up in the tarball. And tarfile then fails to unpack the tarball. One such package is 'plone.app.controlpanel 1.0a2' (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone....), with the following traceback: ... I'm left wondering if the issue is:
a) Windows b) The nasty OS X file c) The Python 'tarfile' module d) setuptools e) None of the above :)
It appears to be a combination of "a" and "c". Windows Python 2.5 does not have an issue with the above file, nor does Cygwin Python 2.3. Windows Python 2.3 and 2.4 do have the problem.
Thoughts?
Dunno. It sounds like the people making these distributions have an over-eager MANIFEST.in or an over-eager source control system. They should probably add a MANIFEST.in rule to exclude the files from the source release.
I looked into this a bit more and it seems you cannot fix this issue by using a MANIFEST.in. The problem is that those resource forks aren't found as regular files and so in return you cannot exclude them via a filter. According to http://forums.macosxhints.com/archive/index.php/t-43243.html those resource forks are created by the Mac OS tar program on the fly, which is used internally by distutils.archive_util. There is no command line switch to disable this behavior but some brave soul looked through the source code of the modified tar and found that setting an environment variable called 'COPY_EXTENDED_ATTRIBUTES_DISABLE' to any value should disable this behavior. I just tried to do a 'python setup.py sdist' for plone.app.controlpanel and a 'tar tzvf plone.app.controlpanel-1.0b3dev-r13784.tar.gz' shows me a ._setup.py file in there. If I first do 'export COPY_EXTENDED_ATTRIBUTES_DISABLE=true' and then rerun the sdist command I don't see the ._setup.py file anymore in the tarball. I wonder if this should be fixed on the distutils level? Does including those resource forks make any sense for source tarballs in some situation? Or could the archive_util set this environment variable on Mac OS X prior to calling tar and reset it afterwards? Hanno
On 3/23/07, Hanno Schlichting <plone@hannosch.info> wrote:
Hi,
Phillip J. Eby wrote:
At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
Not sure if anyone reported a similar issue here since I'm not subscribed, but here it goes.
Some of the new plone.* eggs are being built on OS X by the developers, they usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py sdist upload'
Turns out that if you try to easy_install and there's only the sdist package, but no egg, easy_install uses tarfile to unpack the package, build an egg and install the egg. However, for some reason some of those nasty '._<filename>' files that OS X creates end up in the tarball. And tarfile then fails to unpack the tarball. One such package is 'plone.app.controlpanel 1.0a2' (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone....), with the following traceback: ... I'm left wondering if the issue is:
a) Windows b) The nasty OS X file c) The Python 'tarfile' module d) setuptools e) None of the above :)
It appears to be a combination of "a" and "c". Windows Python 2.5 does not have an issue with the above file, nor does Cygwin Python 2.3. Windows Python 2.3 and 2.4 do have the problem.
Thoughts?
Dunno. It sounds like the people making these distributions have an over-eager MANIFEST.in or an over-eager source control system. They should probably add a MANIFEST.in rule to exclude the files from the source release.
I looked into this a bit more and it seems you cannot fix this issue by using a MANIFEST.in. The problem is that those resource forks aren't found as regular files and so in return you cannot exclude them via a filter.
According to http://forums.macosxhints.com/archive/index.php/t-43243.html those resource forks are created by the Mac OS tar program on the fly, which is used internally by distutils.archive_util.
There is no command line switch to disable this behavior but some brave soul looked through the source code of the modified tar and found that setting an environment variable called 'COPY_EXTENDED_ATTRIBUTES_DISABLE' to any value should disable this behavior.
I just tried to do a 'python setup.py sdist' for plone.app.controlpanel and a 'tar tzvf plone.app.controlpanel-1.0b3dev-r13784.tar.gz' shows me a ._setup.py file in there.
If I first do 'export COPY_EXTENDED_ATTRIBUTES_DISABLE=true' and then rerun the sdist command I don't see the ._setup.py file anymore in the tarball.
I wonder if this should be fixed on the distutils level? Does including those resource forks make any sense for source tarballs in some situation? Or could the archive_util set this environment variable on Mac OS X prior to calling tar and reset it afterwards?
I haven't seen a resource fork used for anything of consequence in many years. I don't think anyone using sdist would ever need them included... Settings that env var seems like the right thing to do. -bob
participants (4)
-
Bob Ippolito -
Hanno Schlichting -
Phillip J. Eby -
Sidnei da Silva