Mailman 3 June 2006 - Distutils-SIG

conditional external packages dependancies with setuptools
by Jorge Vargas June 24, 2006

June 24, 2006

hi all I'm new to creating my own setuptools projects, although I have work with other people projects before, so I'm not that lost here. but What I want to accomplish is having dependancies on some packages. for example my project will have diferent database backends so I want to depend on pysqlite or mysql-python depending on which the user wants as his/her backend. I'm going to have lots of diferent backends, even some that are not databases, so this will be a crutial thing for me or I'll end up with very big dist files per release, which will mostly be waste if the user never uses them. is something like this possible with setuptools now? is it plan to be possible in the future? if so any estimates on when will this be?

1 0

Specification for package indexes?
by Jim Fulton June 23, 2006

June 23, 2006

What form must an index take to be usable with setuptools? Is there anything documented how such a beast should be organized? What should it's pages should look like? Is there any special pattern setuptools (easy_install?) looks for to find pages with download links? Or does it search any link given? Also, I've noticed that if you gibe a location with find-links that has links to distributions for the thing you're looking for, it you can specify pretty much anything as an index. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 2

Is there any harm in clearing sys.path_importer_cache from time to time?
by Jim Fulton June 23, 2006

June 23, 2006

I have a script based on setuptools that invokes easy install to check for and download newer distributions. I allow users to specify whether or not they want unzipped downloads. As a result, easy_install will sometimes replace a zip file with a directory or the other way around. This leads to problems in the invoking program because the sys.path_importer_cache is stale. Does anyone know if it is safe to clear the cache from time to time? If so, I'd be inclined to clear it any time I invoke easy_install. If it isn't safe, I'll probably just forego the feature of letting users force downloads to be unzipped, which I may do anyway to be safe. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 2

setuptools : easy_install error on berlios
by AGiss June 23, 2006

June 23, 2006

berlios, sourceforge like, now use prdownloads system, like sourceforge. setuptools fail to find eggs, because it tries to download on "prdownload" which serves only html page with mirrors. package_index.py need a fix_berlios_url (or perhaps a more generic fix_forges_url with a list of translations) I'll have a look at the code, but I don't have much time thoses days. You can test with : easy_install pydirstat

2 5

Requirent specifiers specified? :)
by Jim Fulton June 23, 2006

June 23, 2006

When specifying a dependencies and when asking for a package with easy_install, you can specify one or more specifiers. It's unclear what the rules are for combining specifiers. Imagine that I have a collection of eggs like: /home/jim/tmp/dist: used 92 available 41345796 -rw-rw-r-- 1 jim jim 671 Jun 19 17:43 demoneeded-1.0-py2.4.egg -rw-rw-r-- 1 jim jim 672 Jun 19 17:46 demoneeded-1.1-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.2-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.3-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.4-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.5-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.6-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.7-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.8-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.9-py2.4.egg Now, here are some examples of using the pkg_resources api to fetch a required distribution: >>> import pkg_resources >>> e = pkg_resources.Environment(['tmp/dist']) >>> ws = pkg_resources.WorkingSet() >>> ws.resolve([pkg_resources.Requirement.parse('demoneeded ==1.1, ==1.4')], e) [demoneeded 1.4 (/home/jim/tmp/dist/demoneeded-1.4-py2.4.egg)] Here, the specifiers were or-ed. OK. >>> ws.resolve([pkg_resources.Requirement.parse('demoneeded >1.1, <1.6')], e) [demoneeded 1.5 (/home/jim/tmp/dist/demoneeded-1.5-py2.4.egg)] Here they were and-ed. This makes sense, from a dwimy point of view. :) If they were or-ed, I'd expect to get 1.9. >>> ws.resolve([pkg_resources.Requirement.parse('demoneeded >1.1, <1.6, ==1.8')], e) [demoneeded 1.8 (/home/jim/tmp/dist/demoneeded-1.8-py2.4.egg)] Hm, here the ==1.8 seems to have been or-ed with the result of anding >1.1 and <1.6. >>> ws.resolve([pkg_resources.Requirement.parse('demoneeded <1.1, >1.6')], e) [demoneeded 1.9 (/home/jim/tmp/dist/demoneeded-1.9-py2.4.egg)] >>> ws.resolve([pkg_resources.Requirement.parse('demoneeded !=1.8, ==1.8')], e) [demoneeded 1.9 (/home/jim/tmp/dist/demoneeded-1.9-py2.4.egg)] I really don't know what's going on with these. :) Are the rules for combining specifiers specified anywhere? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 23

Optimizing easy-install upgrade
by Jim Fulton June 22, 2006

June 22, 2006

Suppose I have the directory: /home/jim/tmp/dist: used 92 available 41345796 -rw-rw-r-- 1 jim jim 671 Jun 19 17:43 demoneeded-1.0-py2.4.egg -rw-rw-r-- 1 jim jim 672 Jun 19 17:46 demoneeded-1.1-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.2-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.3-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.4-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.5-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.6-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.7-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.8-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.9-py2.4.egg and then run easy install telling it to install something I already have: [jim@ds9 ~]$ /usr/local/python/2.4/bin/easy_install -d tmp/dist -mxU demoneeded==1.1 Searching for demoneeded==1.1 Reading http://www.python.org/pypi/demoneeded/ Couldn't find index page for 'demoneeded' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://www.python.org/pypi/ Best match: demoneeded 1.1 Processing demoneeded-1.1-py2.4.egg Using /home/jim/tmp/dist/demoneeded-1.1-py2.4.egg Because this distribution was installed --multi-version, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("demoneeded") # latest installed version pkg_resources.require("demoneeded==1.1") # this exact version pkg_resources.require("demoneeded>=1.1") # this version or higher Note also that the installation directory must be on sys.path at runtime for this to work. (e.g. by being the application's script directory, by being on PYTHONPATH, or by being added to sys.path by your code.) Processing dependencies for demoneeded==1.1 Note that it looked at pypi even though it already had the only acceptable version. I'm not sure why it did this. Is this just a case of a missing optimization? Or is there some other reason for checking the index in a situation like this? I have some scripts that invoke easy_setup and I'd like to try to do some of this logic myself. Given a requirement, I'd like to get the specifiers and decide myself whether to invoke easy_install. I have 2 problems: - I don't want to parse the requirement myself, but, rather, use Requirement.parse. If I use Requirement.parse, I can use the specs attribute to get the specifiers, however, this attribute isn't documented. Should I assume that it is private? Or is it safe to use. - I'm still a bit foggy on how to interpret the specifiers. But we're discussing that in another thread. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 4

Puzzling easy_install behavior
by Jim Fulton June 22, 2006

June 22, 2006

I'd like to share an experiment that I find puzzling and see if anyone could share some insight. Perhaps I'm doing something incorrectly. So, I have an links directory: [jim@ds9 tmp]$ ls links demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg index.html [jim@ds9 tmp]$ cat links/index.html <html><body> <a href="demo-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a> <a href="demoneeded-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a> </body></html> and an empty eggs directory: [jim@ds9 tmp]$ ls eggs Note that demo depends on demoneeded, without any qualifiers. I run easy_setup (0.6b3): [jim@ds9 tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -deggs demo and get the eggs I expect. [jim@ds9 tmp]$ ls eggs demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg I update the links with new versions of demo and demoneeded: [jim@ds9 tmp]$ ls links demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg index.html demo-1.1-py2.4.egg demoneeded-1.1-py2.4.egg [jim@ds9 tmp]$ cat links/index.html <html><body> <a href="demo-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a> <a href="demoneeded-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a> <a href="demo-1.1-py2.4.egg">demoneeded-1.1-py2.4.egg</a> <a href="demoneeded-1.1-py2.4.egg">demoneeded-1.1-py2.4.egg</a> </body></html> I haven't changed the unqualified dependence on demoneeded. Now I rerun easy_install: [jim@ds9 tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -deggs demo and I get a new demo egg even though I didn't elect to upgrade: [jim@ds9 tmp]$ ls eggs demo-1.0-py2.4.egg demo-1.1-py2.4.egg demoneeded-1.0-py2.4.egg I find this a bit mysterious. I thought easy_install wouldn't search the find links and index if existing distributions meet a requirement unless --upgrade was used. Is special consideration given to file URLs? (If so, this makes test writing a bit more challenging.) If I supply the upgrade option, I get: [jim@ds9 tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -U - deggs demo Couldn't find index page for 'demo' (maybe misspelled?) Scanning index of all packages (this may take a while) [jim@ds9 tmp]$ ls eggs demo-1.0-py2.4.egg demo-1.1-py2.4.egg demoneeded-1.0-py2.4.egg So when I said upgrade, the index was checked, but when I installed the distributions initially, it wasn't. Why? Why is upgrade handled differently from an initial install. Also, it's a bit surprising that I didn't get the available upgrade for demoneeded. Is that intended? Are there plans for a recursive upgrade option? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 3

Re: [Distutils] Infinite easy_install loop for mathdom==0.7.1
by Phillip J. Eby June 21, 2006

June 21, 2006

At 12:31 AM 6/22/2006 +0200, =?ISO-8859-2?Q?Micha=B3_Kwiatkowski?= wrote: >Sorry for flooding the list with my questions, but I think I found >another problem with easy_install. Calling it on "mathdom==0.7.1" >invokes an infinite sourceforge download loop. I've given up after >watching "Download failed" messages for about 20 minutes. So I can't >prove this loop is infinite. ;-) The problem is the broken URL for mathdom 0.7.1 provided on the mathdom PyPI page. Sourceforge has no such file. Please let the author know about the problem. In the meantime: easy_install http://cheeseshop.python.org/packages/source/m/mathdom/mathdom-0.7.1.tar.gz should get you the package that is available. (It's not using the 2.4 egg because again, you're on Python 2.3.) >BTW, is there any kind of tracker for setuptools, so I could use that >instead of writing directly to this list? No; the intent is to allow everyone to learn from the answers, and thus better support others. This wouldn't happen as well with a walled-off tracker.

1 0

Re: [Distutils] easy_install download logic?
by Phillip J. Eby June 21, 2006

June 21, 2006

At 11:02 PM 6/21/2006 +0200, =?ISO-8859-2?Q?Micha=B3_Kwiatkowski?= wrote: >Hi, > >I'm wondering how exactly easy_install looks up packages on PyPI. >Right now I'm looking at two specific packages: moofx and SQLObject. >Both have "Home page" link and both have eggs distributions listen on >PyPI. The only difference is that moofx has also a "Download URL", >which seems to be bad. > >When I do "easy_install SQLObject" it looks at PyPI, then link from >"Home page" field and fall backs to egg: > >===== >$ sudo easy_install SQLObject==0.7.0 >Searching for SQLObject==0.7.0 >Reading http://www.python.org/pypi/SQLObject/ >Reading http://sqlobject.org >Best match: SQLObject 0.7.0 >Downloading >http://cheeseshop.python.org/packages/2.3/S/SQLObject/SQLObject-0.7.0-py2.3… >===== > >But this doesn't happen for moofx. Actually, it does. Notice that you're using Python 2.3 -- moofx only has a Python 2.4 egg uploaded, and no source distro or 2.3 egg. >It looks at PyPI, then at "Home >page", then at "Download URL" and fails, ignoring eggs that are there: Because the egg is for 2.4, and you're using 2.3. >Having broken download URL is a bad thing, but shouldn't easy_install >use eggs when they're available? easy_install builds a complete index of the eggs or other downloadable distributions it finds, and then chooses the "best" one. >And why search "Homepage" and >"Download URL" links when eggs are already on PyPI? To ensure that you can request the latest version; sometimes packagers don't update their PyPI pages as often.

1 0

Infinite easy_install loop for mathdom==0.7.1
by Michał Kwiatkowski June 21, 2006

June 21, 2006

Sorry for flooding the list with my questions, but I think I found another problem with easy_install. Calling it on "mathdom==0.7.1" invokes an infinite sourceforge download loop. I've given up after watching "Download failed" messages for about 20 minutes. So I can't prove this loop is infinite. ;-) BTW, is there any kind of tracker for setuptools, so I could use that instead of writing directly to this list? I couldn't found one one the homepage. mk -- . o . >> http://joker.linuxstuff.pl << . . o It's easier to get forgiveness for being wrong o o o than forgiveness for being right.

1 0