Hi, [using setuptools 0.6b4] I'm a setuptools user and greatly appreciative of it as well. I'd like to understand how to use it more appropriately with respect to bundling miscellaneous data files. "just put them in python packages" is really not what I want but perhaps its time to refactor my tastes and accept this is the most appropriate thing to do. I think the root of my confusion lies in how I perceive the 'pkg_' prefix of the setuptools pkg_resources api. I think of it in the general sense of the word package - data I have 'packaged' along with my library or application. Experience suggests it would be better read in the python specific sense of 'package': data that is contained in a file that is a descendant of a python package (the file must be in or under a directory that contains an __init__.py). Which of these two views most accurately reflects the usage for which pkg_resources was designed ? Is it possible to have a separate 'zip_safe' decision for data files versus python packages. Ie., a deployed egg with data files and non zip safe packages would appear in site-packages (or wherever) as both a zip archive for the zip safe data AND a directory tree containing the 'eager' resources ? As context for the rest of this plea for help: A trivial layout that bundles a .conf file:: setuptools-test lib/foopackage/__init__.py lib/foopackage/foo.py conf/foo.conf setup.py versions etc: Ubuntu(dapper), Python 2.4.3, setuptools-0.6b4. My python is installed with the prefix: /home/robin/devel/0root. It is a 'proper' install rather than virtual-python.py setup. Is there a specific reason why there isn't a find_data_files to compliment find_packages in setuptools ? eg., ``data_files=[('/', find_data_files('*.conf'))`` spells: recursively find all .conf files, starting in the directory containing my setup.py, and bundle them in my egg root. I would then expect ``unzip -l foopackage-VER-py2.4.egg`` to produce a tree like:: foopackage/*.py, *.pyc conf/foo.conf EGG-INFO/ # usual suspects Does/could setuptools to overload the distutils keyword 'data_files' and change it's meaning so that it can work with pkg_resources rather than being --prefix relative ? (package_data, while a useful 2.4 addition, is not what I want here) In foopackage/foo.py Why are all of:: pkg_resources(Requirement(__name__), '/conf/foo.conf') pkg_resources(Requirement(__name__), 'conf/foo.conf') pkg_resources(Requirement('foopackage'), '/conf/foo.conf') interpreted as relative to the foopackage directory ? And why does the resource_name '/' not refer to the top of the egg ? Irrespective of whether I specify a relative or absolute path above pkg_resources always looks under the top most package directory. Is this by design ? Non packaged data files are packaged as siblings of 'foopackage'. What is the most convenient way package and access these files such that the references work for egg installs, normal 'setup.py install' installations and for ``python setup.py develop`` pseudo installs ? Extending my example with the following changes I explore pkg_resources.resource_string and friends:: file:setup.cfg [egg_info] egg_base=./ # because I guessed (incorrectly) that this would help. file:setup.py:: from setuptools import setup, find_packages setup( name='foopackage', packages=find_packages('lib'), package_dir={'','lib'}, data_files=[('conf','conf/foo.conf')], entry_points=dict(console_scripts=[ 'fooconf = foopackage.foo:run'])) file:foopackage/foo.py:: from pkg_resources import resource_string def run(): print __name__,__file__ try: foo_config=resource_string(__name__,'/conf/foo.conf') except IOError, e: print str(e) else: print foo_config if __name__=='__main__': run() running in place:: $python setuptools-test/lib/foopackage/foo.py __main__ setuptools-test/lib/foopackage/foo.py [Errno 20] Not no such file or directory: 'setuptools-test/lib/foopackage/conf/foo.conf' This is (almost) what I'd expect, I have not run setup.py yet so setuptools/pkg_resources has no way of knowing anything about my weirdo preferences. Given that setuptools has not had a chance to see my egg_base setting, I would expect '/' to mean the directory *containing* the top most package inferable from __file__. So I would have expected the path in the error to be 'setuptools-test/lib/conf/foo.conf'. But I don't care so much about the 'pre setup.py' scenario. Make an egg:: $python setup.py bdist_egg --keep-temp <snip> copying conf/foo.conf -> build/bdist.linux-i686/egg/conf creating 'dist/foopackage-0.0.0-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it $ls build/bdist.linux-i686/egg conf EGG-INFO foopackage $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Woot! Exactly what I had hoped for. Install the package using develop mode (note the explicit egg_base option above):: First, manually clean up site-packages just to be sure. (rm easy-install.pth; rm foopackage*) $cd setuptools-test $python setup.py develop <snip> Installing fooconf script to /home/robin/devel/0root/bin $cd .. $fooconf Traceback (most recent call last): <snip> ImportError: No module named foopackage.foo $cat $PYSITE/foopackage.egg-link /home/robin/devel/setuptools-test $cat $PYSITE/easy-install.pth import sys; sys.__plen = len(sys.path) /home/robin/devel/setuptools-test Rats. It seems like egg_base is taken both as the place to put my .egg-info directory AND as the means of deciding what should be placed on sys.path in order for my package to be importable. remembering the package_dir option I reach for the distutils docs. But a variety of package=[], package_dir=[] combinations have no effect on the easy-install.pth. Double rats. Is there a way to have easy-install.pth, in the develop case, to get entries with the form: '/path/to/source-root/my/python/packages/live/here' ^--------------------------^ this is the bit we already have | assuming egg_base=./ (which it does not by default) AND an independent way of directing pkg_resources to where my data files are rooted ?? I look at the egg install case:: delete dist & build & do python setup.py bdist_egg, delete easy-install.pth and *.link files because I've been fiddling. $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg Copying foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 Erm what happened to my script ? (using setuptools 0.6b4). Quoting the easy_install docs: "Whenever you install, upgrade, or change versions of a package, EasyInstall automatically installs the scripts for the selected package version, unless you tell it not to" I fire up python:: $pwd /home/robin/devel/setuptools-test $cd .. $which python $/home/robin/devel/0root/bin/python $python >>>from foopackage.foo import run >>>run() foopackage.foo /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: '/home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Ctrl-D $ls 0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/conf/ foo.conf (extreme gnashing of teeth, followed by a visit to the refrigerator so i can throw some eggs at the wall) I give up on egg_base, delete my setup.cfg, manually clean up my site-packages directory and delete my dist & build trees. I create a new egg, this time without the egg_base option:: $cd setuptools-test $python setup.py bdist_egg $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Again, exactly what I want and shows that egg_base has *no* effect on the internal layout of the egg. Lets install it:: $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg creating /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Extracting foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installing fooconf script to /home/robin/devel/0root/bin Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 What the frak, this time I get my script. What on earth does egg_base have to do with script generation ? I try it out:: $fooconf foopackage.foo PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: 'PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Curses. <gripe>The fact that distutils does not include files specified in MANIFIEST.in for anything other than the sdist command (source distributions) is really tedious (and horribly confusing when first encountered).</gripe> (minor nit) the include_package_data option is well named but poorly described: "Accept all data files and directories matched by MANIFEST.in or found in source control". This is a lie. Only those files that are desendants of a python package directory (directory that has __init__.py) are considered. "Accept all python package data files " would reduce confusion and pointless optimism. At least for those that know just enough to get themselves into trouble (like me). I very much would prefer that the machinery for including data files in a package to be orthogonal to the source building/packaging machinery. I think there is a compelling argument that says complex data should be explicitly packaged separately. Ie if foopackage had non trivial data then I, as the package author, should create and distribute foopackage.egg and foopackage.data.egg as separate things. foopackage.egg would require foopackage.data and would, as an additionally benefit, be free to use existant setuptools machinery to separate data versions from package versions. In fact, to argue completely against the thrust of this mail I'm now thinking along the lines of: - *never* package data in the same egg as the application or library - *always* create a separate foopackage-data package, even if it has no python source in it beyond setup.py and even if the data is trivial. - use the optional dependencies mechanism to pull data in as needed. Anyhow. Thats got that lot off my chest. I have no intention of giving up on setuptools, it is *far* to useful for that. I do want to hear from distutils folks that could help straighten me out :-) Cheers, Robin