distutils data_files and setuptools.pkg_resources are driving me crazy
Hi,
[using setuptools 0.6b4]
I'm a setuptools user and greatly appreciative of it as well. I'd like
to understand how to use it more appropriately with respect to
bundling miscellaneous data files. "just put them in python packages"
is really not what I want but perhaps its time to refactor my tastes
and accept this is the most appropriate thing to do.
I think the root of my confusion lies in how I perceive the 'pkg_'
prefix of the setuptools pkg_resources api. I think of it in the
general sense of the word package - data I have 'packaged' along with
my library or application. Experience suggests it would be better read
in the python specific sense of 'package': data that is contained in a
file that is a descendant of a python package (the file must be in or
under a directory that contains an __init__.py).
Which of these two views most accurately reflects the usage for which
pkg_resources was designed ?
Is it possible to have a separate 'zip_safe' decision for data files
versus python packages. Ie., a deployed egg with data files and non
zip safe packages would appear in site-packages (or wherever) as both
a zip archive for the zip safe data AND a directory tree containing
the 'eager' resources ?
As context for the rest of this plea for help: A trivial layout that
bundles a .conf file::
setuptools-test
lib/foopackage/__init__.py
lib/foopackage/foo.py
conf/foo.conf
setup.py
versions etc:
Ubuntu(dapper), Python 2.4.3, setuptools-0.6b4. My python is installed
with the prefix: /home/robin/devel/0root. It is a 'proper' install
rather than virtual-python.py setup.
Is there a specific reason why there isn't a find_data_files to
compliment find_packages in setuptools ?
eg., ``data_files=[('/', find_data_files('*.conf'))`` spells:
recursively find all .conf files, starting in the directory containing
my setup.py, and bundle them in my egg root. I would then expect
``unzip -l foopackage-VER-py2.4.egg`` to produce a tree like::
foopackage/*.py, *.pyc
conf/foo.conf
EGG-INFO/ # usual suspects
Does/could setuptools to overload the distutils keyword 'data_files'
and change it's meaning so that it can work with pkg_resources rather
than being --prefix relative ? (package_data, while a useful 2.4
addition, is not what I want here)
In foopackage/foo.py Why are all of::
pkg_resources(Requirement(__name__), '/conf/foo.conf')
pkg_resources(Requirement(__name__), 'conf/foo.conf')
pkg_resources(Requirement('foopackage'), '/conf/foo.conf')
interpreted as relative to the foopackage directory ?
And why does the resource_name '/' not refer to the top of the egg ?
Irrespective of whether I specify a relative or absolute path above
pkg_resources always looks under the top most package directory. Is
this by design ?
Non packaged data files are packaged as siblings of 'foopackage'. What
is the most convenient way package and access these files such that
the references work for egg installs, normal 'setup.py install'
installations and for ``python setup.py develop`` pseudo installs ?
Extending my example with the following changes I explore
pkg_resources.resource_string and friends::
file:setup.cfg
[egg_info]
egg_base=./ # because I guessed (incorrectly) that this would help.
file:setup.py::
from setuptools import setup, find_packages
setup(
name='foopackage',
packages=find_packages('lib'),
package_dir={'','lib'},
data_files=[('conf','conf/foo.conf')],
entry_points=dict(console_scripts=[
'fooconf = foopackage.foo:run']))
file:foopackage/foo.py::
from pkg_resources import resource_string
def run():
print __name__,__file__
try:
foo_config=resource_string(__name__,'/conf/foo.conf')
except IOError, e:
print str(e)
else:
print foo_config
if __name__=='__main__':
run()
running in place::
$python setuptools-test/lib/foopackage/foo.py
__main__ setuptools-test/lib/foopackage/foo.py
[Errno 20] Not no such file or directory:
'setuptools-test/lib/foopackage/conf/foo.conf'
This is (almost) what I'd expect, I have not run setup.py yet so
setuptools/pkg_resources has no way of knowing anything about my
weirdo preferences. Given that setuptools has not had a chance to see
my egg_base setting, I would expect '/' to mean the directory
*containing* the top most package inferable from __file__. So I would
have expected the path in the error to be
'setuptools-test/lib/conf/foo.conf'. But I don't care so much about
the 'pre setup.py' scenario.
Make an egg::
$python setup.py bdist_egg --keep-temp
<snip>
copying conf/foo.conf -> build/bdist.linux-i686/egg/conf
creating 'dist/foopackage-0.0.0-py2.4.egg' and adding
'build/bdist.linux-i686/egg' to it
$ls build/bdist.linux-i686/egg
conf EGG-INFO foopackage
$unzip -l dist/foopackage-0.0.0-py2.4.egg
# paraphrasing the output
foopackage/ *.py *.pyc
conf/foo.conf
EGG-INFO/
Woot! Exactly what I had hoped for.
Install the package using develop mode (note the explicit egg_base
option above)::
First, manually clean up site-packages just to be sure. (rm
easy-install.pth;
rm foopackage*)
$cd setuptools-test
$python setup.py develop
<snip>
Installing fooconf script to /home/robin/devel/0root/bin
$cd ..
$fooconf
Traceback (most recent call last):
<snip>
ImportError: No module named foopackage.foo
$cat $PYSITE/foopackage.egg-link
/home/robin/devel/setuptools-test
$cat $PYSITE/easy-install.pth
import sys; sys.__plen = len(sys.path)
/home/robin/devel/setuptools-test
participants (3)
-
Bob Ippolito
-
Robin Bryce
-
Ronald Oussoren