Mailman 3 July 2006 - Distutils-SIG

distutils data_files and setuptools.pkg_resources are driving me crazy
by Robin Bryce 14 Jul '06

14 Jul '06

Hi, [using setuptools 0.6b4] I'm a setuptools user and greatly appreciative of it as well. I'd like to understand how to use it more appropriately with respect to bundling miscellaneous data files. "just put them in python packages" is really not what I want but perhaps its time to refactor my tastes and accept this is the most appropriate thing to do. I think the root of my confusion lies in how I perceive the 'pkg_' prefix of the setuptools pkg_resources api. I think of it in the general sense of the word package - data I have 'packaged' along with my library or application. Experience suggests it would be better read in the python specific sense of 'package': data that is contained in a file that is a descendant of a python package (the file must be in or under a directory that contains an __init__.py). Which of these two views most accurately reflects the usage for which pkg_resources was designed ? Is it possible to have a separate 'zip_safe' decision for data files versus python packages. Ie., a deployed egg with data files and non zip safe packages would appear in site-packages (or wherever) as both a zip archive for the zip safe data AND a directory tree containing the 'eager' resources ? As context for the rest of this plea for help: A trivial layout that bundles a .conf file:: setuptools-test lib/foopackage/__init__.py lib/foopackage/foo.py conf/foo.conf setup.py versions etc: Ubuntu(dapper), Python 2.4.3, setuptools-0.6b4. My python is installed with the prefix: /home/robin/devel/0root. It is a 'proper' install rather than virtual-python.py setup. Is there a specific reason why there isn't a find_data_files to compliment find_packages in setuptools ? eg., ``data_files=[('/', find_data_files('*.conf'))`` spells: recursively find all .conf files, starting in the directory containing my setup.py, and bundle them in my egg root. I would then expect ``unzip -l foopackage-VER-py2.4.egg`` to produce a tree like:: foopackage/*.py, *.pyc conf/foo.conf EGG-INFO/ # usual suspects Does/could setuptools to overload the distutils keyword 'data_files' and change it's meaning so that it can work with pkg_resources rather than being --prefix relative ? (package_data, while a useful 2.4 addition, is not what I want here) In foopackage/foo.py Why are all of:: pkg_resources(Requirement(__name__), '/conf/foo.conf') pkg_resources(Requirement(__name__), 'conf/foo.conf') pkg_resources(Requirement('foopackage'), '/conf/foo.conf') interpreted as relative to the foopackage directory ? And why does the resource_name '/' not refer to the top of the egg ? Irrespective of whether I specify a relative or absolute path above pkg_resources always looks under the top most package directory. Is this by design ? Non packaged data files are packaged as siblings of 'foopackage'. What is the most convenient way package and access these files such that the references work for egg installs, normal 'setup.py install' installations and for ``python setup.py develop`` pseudo installs ? Extending my example with the following changes I explore pkg_resources.resource_string and friends:: file:setup.cfg [egg_info] egg_base=./ # because I guessed (incorrectly) that this would help. file:setup.py:: from setuptools import setup, find_packages setup( name='foopackage', packages=find_packages('lib'), package_dir={'','lib'}, data_files=[('conf','conf/foo.conf')], entry_points=dict(console_scripts=[ 'fooconf = foopackage.foo:run'])) file:foopackage/foo.py:: from pkg_resources import resource_string def run(): print __name__,__file__ try: foo_config=resource_string(__name__,'/conf/foo.conf') except IOError, e: print str(e) else: print foo_config if __name__=='__main__': run() running in place:: $python setuptools-test/lib/foopackage/foo.py __main__ setuptools-test/lib/foopackage/foo.py [Errno 20] Not no such file or directory: 'setuptools-test/lib/foopackage/conf/foo.conf' This is (almost) what I'd expect, I have not run setup.py yet so setuptools/pkg_resources has no way of knowing anything about my weirdo preferences. Given that setuptools has not had a chance to see my egg_base setting, I would expect '/' to mean the directory *containing* the top most package inferable from __file__. So I would have expected the path in the error to be 'setuptools-test/lib/conf/foo.conf'. But I don't care so much about the 'pre setup.py' scenario. Make an egg:: $python setup.py bdist_egg --keep-temp <snip> copying conf/foo.conf -> build/bdist.linux-i686/egg/conf creating 'dist/foopackage-0.0.0-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it $ls build/bdist.linux-i686/egg conf EGG-INFO foopackage $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Woot! Exactly what I had hoped for. Install the package using develop mode (note the explicit egg_base option above):: First, manually clean up site-packages just to be sure. (rm easy-install.pth; rm foopackage*) $cd setuptools-test $python setup.py develop <snip> Installing fooconf script to /home/robin/devel/0root/bin $cd .. $fooconf Traceback (most recent call last): <snip> ImportError: No module named foopackage.foo $cat $PYSITE/foopackage.egg-link /home/robin/devel/setuptools-test $cat $PYSITE/easy-install.pth import sys; sys.__plen = len(sys.path) /home/robin/devel/setuptools-test <snip - its a fresh easy-install.pth file> Rats. It seems like egg_base is taken both as the place to put my .egg-info directory AND as the means of deciding what should be placed on sys.path in order for my package to be importable. remembering the package_dir option I reach for the distutils docs. But a variety of package=[], package_dir=[] combinations have no effect on the easy-install.pth. Double rats. Is there a way to have easy-install.pth, in the develop case, to get entries with the form: '/path/to/source-root/my/python/packages/live/here' ^--------------------------^ this is the bit we already have | assuming egg_base=./ (which it does not by default) AND an independent way of directing pkg_resources to where my data files are rooted ?? I look at the egg install case:: delete dist & build & do python setup.py bdist_egg, delete easy-install.pth and *.link files because I've been fiddling. $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg Copying foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 Erm what happened to my script ? (using setuptools 0.6b4). Quoting the easy_install docs: "Whenever you install, upgrade, or change versions of a package, EasyInstall automatically installs the scripts for the selected package version, unless you tell it not to" I fire up python:: $pwd /home/robin/devel/setuptools-test $cd .. $which python $/home/robin/devel/0root/bin/python $python >>>from foopackage.foo import run >>>run() foopackage.foo /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: '/home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Ctrl-D $ls 0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/conf/ foo.conf (extreme gnashing of teeth, followed by a visit to the refrigerator so i can throw some eggs at the wall) I give up on egg_base, delete my setup.cfg, manually clean up my site-packages directory and delete my dist & build trees. I create a new egg, this time without the egg_base option:: $cd setuptools-test $python setup.py bdist_egg $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Again, exactly what I want and shows that egg_base has *no* effect on the internal layout of the egg. Lets install it:: $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg creating /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Extracting foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installing fooconf script to /home/robin/devel/0root/bin Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 What the frak, this time I get my script. What on earth does egg_base have to do with script generation ? I try it out:: $fooconf foopackage.foo PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: 'PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Curses. <gripe>The fact that distutils does not include files specified in MANIFIEST.in for anything other than the sdist command (source distributions) is really tedious (and horribly confusing when first encountered).</gripe> (minor nit) the include_package_data option is well named but poorly described: "Accept all data files and directories matched by MANIFEST.in or found in source control". This is a lie. Only those files that are desendants of a python package directory (directory that has __init__.py) are considered. "Accept all python package data files " would reduce confusion and pointless optimism. At least for those that know just enough to get themselves into trouble (like me). I very much would prefer that the machinery for including data files in a package to be orthogonal to the source building/packaging machinery. I think there is a compelling argument that says complex data should be explicitly packaged separately. Ie if foopackage had non trivial data then I, as the package author, should create and distribute foopackage.egg and foopackage.data.egg as separate things. foopackage.egg would require foopackage.data and would, as an additionally benefit, be free to use existant setuptools machinery to separate data versions from package versions. In fact, to argue completely against the thrust of this mail I'm now thinking along the lines of: - *never* package data in the same egg as the application or library - *always* create a separate foopackage-data package, even if it has no python source in it beyond setup.py and even if the data is trivial. - use the optional dependencies mechanism to pull data in as needed. Anyhow. Thats got that lot off my chest. I have no intention of giving up on setuptools, it is *far* to useful for that. I do want to hear from distutils folks that could help straighten me out :-) Cheers, Robin

3 6

Re: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy
by Phillip J. Eby 14 Jul '06

14 Jul '06

At 12:56 AM 7/14/2006 +0100, Robin Bryce wrote: >But all of that is less than ideal. AFAICT, if pkg_resources supported >egg root relative resource_names, I would be able to do anything I >want, up to an including writing egg format extensions that do more >specialist things with data packaged in *eggs*. I don't have time right now to respond to this thread in more detail, but I just want to pop in and point out the *reason* that pkg_resources doesn't support egg root relative resource names is because it won't work right with system packaging tools like RPM, Debian, etc. You've pretty much got to use the "metadata" API if you want stuff that belongs to the project instead of to a package in the project. The metadata API has some limitations, however (i.e., no file access), but these might be able to be lifted in a future release in the 0.7 or later series.

1 0

a patch for distutils + gcc + linux, that was never apply, yet will solve many problems.
by Jorge Vargas 11 Jul '06

11 Jul '06

Hello everyone I did a package for gentoo (ebuild) a while ago and it has a C/C++ extension, to compile agains a gecko-compliant mozilla product. Long story short our solution (build agains mozilla) seem to had nothing to do with the real bug which someone posted today. It seems there is a bug in distutils itself, because it is passing the wrong arguments to g++, the most incredible part is that the bug is so simple to fix yet there is no reply, maybe this got lost in the main python repository. here is the most important part of his post ---- /usr/lib/python2.4/distutils/unixccompiler.py to build. ** /usr/lib/python2.4/distutils/unixccompiler.py fills the 'CC' executable variable by invoking /usr/lib/python2.4/distutils/sysconfig.py ** /usr/lib/python2.4/distutils/unixccompiler.py contains a condition that says: elif compiler[:3] == "gcc" or compiler[:3] == "g++": return "-Wl,-R" + dir ** /usr/lib/python2.4/distutils/sysconfig.py returns the 'CC' executable variable by reading from /usr/lib/python2.4/config/Makefile ** If your system is like many others, it will return something like 'CC=i686-pc-linux-gnu-gcc -pthread', will never == "gcc" and so the correct linker options will not be used. ---- if anyone could please verify it and maybe find out why it wasn't apply, honestly I have very little knowlegde of C and it's integration with python to be able to verify it. his full post is at http://bugs.gentoo.org/show_bug.cgi?id=131527#c36 the original disutil gentoo bug: http://bugs.gentoo.org/show_bug.cgi?id=138647 the patch at sourceforce: http://sourceforge.net/tracker/index.php?func=detail&aid=1254718&group_id=5… thanks.

2 2

"Package Index API" draft
by Phillip J. Eby 11 Jul '06

11 Jul '06

This draft reflects the in-development versions of setuptools 0.7a1 and 0.6b4; it does not describe older setuptools versions except as noted under the "Backward Compatibility" section. The items described under "Backward Compatibility" need to be kept in PyPI until everyone in the field has upgraded to setuptools 0.6b4 or better. (Note that 0.6b4 is not released yet!) Package Index "API" ------------------- Custom package indexes (and PyPI) must follow the following rules for EasyInstall to be able to look up and download packages: 1. Except where stated otherwise, "pages" are HTML or XHTML, and "links" refer to ``href`` attributes. 2. Individual project version pages' URLs must be of the form ``base/projectname/version``, where ``base`` is the package index's base URL. 3. Omitting the ``/version`` part of a project page's URL (but keeping the trailing ``/``) should result in a page that is either: a) The single active version of that project, as though the version had been explicitly included, OR b) A page with links to all of the active version pages for that project. 4. Individual version pages should contain direct links to downloadable distributions where possible. It is explicitly permitted for a project's "long_description" to include URLs, and these should be formatted as HTML links by the package index, as EasyInstall does no special processing to identify what parts of a page are index-specific and which are part of the project's supplied description. 5. Where available, MD5 information should be added to download URLs by appending a fragment identifier of the form ``#md5=...``, where ``...`` is the 32-character hex MD5 digest. EasyInstall will verify that the downloaded file's MD5 digest matches the given value. 6. Individual project version pages should identify any "homepage" or "download" URLs using ``rel="homepage"`` and ``rel="download"`` attributes on the HTML elements linking to those URLs. Use of these attributes will cause EasyInstall to always follow the provided links, unless it can be determined by inspection that they are downloadable distributions. If the links are not to downloadable distributions, they are retrieved, and if they are HTML, they are scanned for download links. They are *not* scanned for additional "homepage" or "download" links, as these are only processed for pages that are part of a package index site. 7. The root URL of the index, if retrieved with a trailing ``/``, must result in a page containing links to *all* projects' active version pages. (Note: This requirement is a workaround for the absence of case- insensitive ``safe_name()`` matching of project names in URL paths. If project names are matched in this fashion (e.g. via the PyPI server, mod_rewrite, or a similar mechanism), then it is not necessary to include this all-packages listing page.) 8. If a package index is accessed via a ``file://`` URL, then EasyInstall will automatically use ``index.html`` files, if present, when trying to read a directory with a trailing ``/`` on the URL. Backward Compatibility ~~~~~~~~~~~~~~~~~~~~~~ Package indexes that wish to support setuptools versions prior to 0.6b4 should also follow these rules: * Homepage and download links must be preceded with ``"<th>Home Page"`` or ``"<th>Download URL"``, in addition to (or instead of) the ``rel=""`` attributes on the actual links. These marker strings do not need to be visible, or uncommented, however! For example, the following is a valid homepage link that will work with any version of setuptools:: <li> <strong>Home Page:</strong>  <a rel="homepage" href="http://sqlobject.org">http://sqlobject.org</a> </li> Even though the marker string is in an HTML comment, older versions of EasyInstall will still "see" it and know that the link that follows is the project's home page URL. * The pages described by paragraph 3(b) of the preceding section *must* contain the string ``"Index of Packages</title>"`` somewhere in their text. This can be inside of an HTML comment, if desired, and it can be anywhere in the page. (Note: this string MUST NOT appear on normal project pages, as described in paragraphs 2 and 3(a)!) In addition, for compatibility with PyPI versions that do not use ``#md5=`` fragment IDs, EasyInstall uses the following regular expression to match PyPI's displayed MD5 info (broken onto two lines for readability):: <a href="([^"#]+)">([^<]+)</a>\n\s+$<a href="[^?]+\?:action=show_md5 &digest=([0-9a-f]{32})">md5</a>$

1 0

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by Phillip J. Eby 08 Jul '06

08 Jul '06

At 02:04 PM 7/7/2006 -0400, Fred Drake wrote: >On 7/7/06, Jim Fulton <jim(a)zope.com> wrote: > > > +1 on static pages. I don't, however, see a reason to require > > > valid XML. Or rather, I don't expect to implement XML parsing in > > > easy_install; if the spec is too complex to implement with regular > > > expression matching, it's probably too complex for people to throw > > > together an index with what's at hand. In particular, I'd like it > > > to be practical to put together a simple index just using Apache's > > > built-in directory indexes, as long as they use the right URL > > > hierarchy. That means that class or rel attributes should only be > > > required for links that are requesting non-index pages to be spidered. > > > > I would find parsing much easier with an XML parser than with > > regular expressions. > > I think it would be much more robust too. > >XHTML would be best, though I agree we shouldn't care about validity >so much as just well-formedness (which is required). I think it >should be possible to do it with valid XHTML, though, since whether >that's desired or not is a python.org policy concern. (Not that I >suspect we'll ever really care about that.) > >Of course, it should be possible to parse with htmllib and HTMLParser as well. I still think requiring even HTML validity or well-formedness is YAGNI; one could indeed just pull all well-formed URLs from the page. EasyInstall uses this case-insensitive regular expression to find only href'd urls: href\s*=\s*['"]?([^'"> ]+) In the absence of a requirement for more information than this (perhaps coupled with a "rel" attribute in the same element), I'm wary of starting out by requiring even well-formedness, because it's way overkill for the requirements as I understand them. One of the advantage of defining the URL layout as part of the API is that it gives you enough contextual information to decide what links should be followed, and which ones are purely informational. Indeed, the only reason to look at anything *but* hrefs is to indicate that an *external* (i.e. non-index) link should be followed, to spider for other download links. So if following external links is out of scope for the API we want to define, then *any* information other than the URLs in an API page are YAGNI. Now, all of this is based on my assumption that the use case here is somebody wants to throw together a rough-and-ready package index that tools should be able to use to find *downloadable distributions*. If you and Jim have much more elaborate use cases in mind, then of course some well-formedness might be useful. On the other hand, if such rigor is required, then it seems like we should just be using machine-readable data in the first place, rather than using a dual-purpose format like HTML or XHTML. Just go with a specialized XML dialect or some kind of text format (ZConfig? ;) ) and be done with it.

2 5

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by Phillip J. Eby 07 Jul '06

07 Jul '06

At 02:31 PM 7/7/2006 -0400, Phillip J. Eby wrote: >On the other hand, if such rigor is required, then it seems like we should >just be using machine-readable data in the first place, rather than using a >dual-purpose format like HTML or XHTML. Just go with a specialized XML >dialect or some kind of text format (ZConfig? ;) ) and be done with it. FWIW, I just discovered DOAP, and it seems like a good basis for this sort of thing. PyPI isn't generating 'file-release' info in its DOAP output, but probably should, if this is going to be used for that sort of thing. Likewise, I think that the reST-to-HTML translation should be carried over into the long-description content. (This is all assuming, of course, that you have use cases for which the richness of DOAP is relevant, anyway, as opposed to just trying to find download links for projects with known names.)

1 0

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by richardjones＠optusnet.com.au 07 Jul '06

07 Jul '06

> Phillip J. Eby <pje(a)telecommunity.com> wrote: > Why not? ;) That was actually what I was afraid the reasoning was ;) I guess I just go all wobbly in the knees at the thought of having to maintain a "screen scraping" interface. Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about this very issue last night. I proposed that we detect the user-agent of the setuptools client, and in response send back really minimalist HTML (no surrounding page template). Probably overkill, but this may have been after we'd had beer :) Could you provide a clear list of all the specific changes you wish for us to make at the Sprint? > Nonetheless, there are various aspects of easy_install's behavior and > performance that could be significantly improved by using XML-RPC, so I > definitely want it to do that in 0.7. I'm just wary of removing the > existing behavior until it's clear that it's unnecessary for it to. Oh - another thing that occurred to me -- does setuptools auto update itself? Richard

4 5

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by richardjones＠optusnet.com.au 07 Jul '06

07 Jul '06

> Phillip J. Eby <pje(a)telecommunity.com> wrote: > It looks like easy_install is still actually working at the moment; I > guess > that means you kept the list working if the user-agent is > easy_install. Thank you, and sorry for jumping to conclusions. I took > "This has been removed" to mean that it had been, well, removed. :) No, I'm not sure why it didn't break - the "/" URL behaviour *had* changed to return the home page. I have reinstated the old behaviour but now we're running in mod_python we require an apache restart -- which requires a roto-rooter, so responsiveness to changes like this has just taken a significant nose-dive. > I will mention again, though, that it *can* be removed entirely, without > needing any in-the-field upgrades, if project name matches in URLs can > be made case-insensitive and canonical via safe_name: And I will mention again that I'm not willing to just whack this code into the package index without some thought to to ramifications given that distutils has no such name mangling or limitation. We will have potential (however remote) of name collision and we'll have to deal with that somehow. Richard

2 1

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by richardjones＠optusnet.com.au 07 Jul '06

07 Jul '06

> Phillip J. Eby <pje(a)telecommunity.com> wrote: > If you needed it to go away, giving me some notice would have been > nice. Not just to me, but to all the people who use PyPI via > easy_install. This gratuitous breakage is not nice. OK, OK! I'll put it back, I just won't link to it anywhere. Richard

1 0

Re: [Distutils] [Catalog-sig] Specification for package indexes?
by Phillip J. Eby 07 Jul '06

07 Jul '06

At 11:52 AM 7/7/2006 -0400, Phillip J. Eby wrote: >If you needed it to go away, giving me some notice would have been >nice. Not just to me, but to all the people who use PyPI via >easy_install. This gratuitous breakage is not nice. It looks like easy_install is still actually working at the moment; I guess that means you kept the list working if the user-agent is easy_install. Thank you, and sorry for jumping to conclusions. I took "This has been removed" to mean that it had been, well, removed. :) I will mention again, though, that it *can* be removed entirely, without needing any in-the-field upgrades, if project name matches in URLs can be made case-insensitive and canonical via safe_name: def safe_name(name): """Convert an arbitrary string to a standard distribution name Any runs of non-alphanumeric/. characters are replaced with a single '-'. """ return re.sub('[^A-Za-z0-9.]+', '-', name) If you can fall back to matching safe_name(pkg_name).lower() against search_name.lower() if the URL is not exact, then easy_install will not need the full package listing *or* the XML-RPC interface. It will work as-is in the field today, with no patching required. The only thing the full package listing is used for is to do this search. (Likewise, it would've been the main thing the XML-RPC API would've been used for.)

1 0