[Distutils] setuptools and sdist on Mac OS X, tarfile flukes

[Bob Ippolito]

On 3/23/07, Hanno Schlichting <plone at hannosch.info> wrote:
> Hi,
>
> Phillip J. Eby wrote:
> > At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
> >> Not sure if anyone reported a similar issue here since I'm not subscribed, but
> >> here it goes.
> >>
> >> Some of the new plone.* eggs are being built on OS X by the developers, they
> >> usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py
> >> sdist upload'
> >>
> >> Turns out that if you try to easy_install and there's only the sdist package,
> >> but no egg, easy_install uses tarfile to unpack the package, build an egg and
> >> install the egg. However, for some reason some of those nasty '._<filename>'
> >> files that OS X creates end up in the tarball. And tarfile then fails to
> >> unpack
> >> the tarball. One such package is 'plone.app.controlpanel 1.0a2'
> >> (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone.app.controlpanel-1.0a2.tar.gz#md5=2b2afeaba0d067f6b1a6707c6d69311d),
> >> with the following traceback:
> >> ...
> >> I'm left wondering if the issue is:
> >>
> >>  a) Windows
> >>  b) The nasty OS X file
> >>  c) The Python 'tarfile' module
> >>  d) setuptools
> >>  e) None of the above :)
> >
> > It appears to be a combination of "a" and "c".  Windows Python 2.5 does not
> > have an issue with the above file, nor does Cygwin Python 2.3.  Windows
> > Python 2.3 and 2.4 do have the problem.
> >
> >> Thoughts?
> >
> > Dunno.  It sounds like the people making these distributions have an
> > over-eager MANIFEST.in or an over-eager source control system.  They should
> > probably add a MANIFEST.in rule to exclude the files from the source release.
>
> I looked into this a bit more and it seems you cannot fix this issue by
> using a MANIFEST.in. The problem is that those resource forks aren't
> found as regular files and so in return you cannot exclude them via a
> filter.
>
> According to
> http://forums.macosxhints.com/archive/index.php/t-43243.html those
> resource forks are created by the Mac OS tar program on the fly, which
> is used internally by distutils.archive_util.
>
> There is no command line switch to disable this behavior but some brave
> soul looked through the source code of the modified tar and found that
> setting an environment variable called
> 'COPY_EXTENDED_ATTRIBUTES_DISABLE' to any value should disable this
> behavior.
>
> I just tried to do a 'python setup.py sdist' for plone.app.controlpanel
> and a 'tar tzvf plone.app.controlpanel-1.0b3dev-r13784.tar.gz' shows me
> a ._setup.py file in there.
>
> If I first do 'export COPY_EXTENDED_ATTRIBUTES_DISABLE=true' and then
> rerun the sdist command I don't see the ._setup.py file anymore in the
> tarball.
>
> I wonder if this should be fixed on the distutils level? Does including
> those resource forks make any sense for source tarballs in some
> situation? Or could the archive_util set this environment variable on
> Mac OS X prior to calling tar and reset it afterwards?

I haven't seen a resource fork used for anything of consequence in
many years. I don't think anyone using sdist would ever need them
included... Settings that env var seems like the right thing to do.

-bob