Mailman 3 March 2010 - Distutils-SIG

Re: [Distutils] use of '_' in package name causing version parsing issue?
by P.J. Eby 12 Mar '10

12 Mar '10

At 12:38 PM 3/11/2010 +0530, Baiju M wrote: >On Thu, Mar 11, 2010 at 11:05 AM, Baiju M <mbaiju(a)zeomega.com> wrote: > > If "_" is a valid project_name identifier, why it is replaces with "-" ? In order to have a canonicalized name form which can be escaped in filenames for unambiguous identification of an egg's project and version. Egg filenames use '-' as a separator between name, version, python version, and platform. A '-' in any of these components is escaped as '_', so that the '-' remains a viable and unambiguous separator. This means that '_' gets turned back into a '-' when unescaped, so the mapping between '_' and '-' is part of the safe_name canonical form. >There nearly 300 packages in PyPI with "_" in the package name. >For all the packages built using Setuptools, the "Name" field in >the PKG-INFO file is replaced with "-". > >I checked some of the packages built with "distutils.core" [1] >Distutils is not replacing "Name" field in PKG-INFO file >with "-". > >Why Setuptools is behaving different from Distutils ? Because distutils wasn't built in a world where: package names needed to be uniquely and unambiguously machine-parseable from filenames. The code that easy_install has for dealing with distutils-named source distributions has to guess at possible interpretations of those filenames, because distutils filenames don't distinguish between a '-' in a name or version, and a '-' *between* names and versions. Ultimately, the simplest way to deal with this was to treat runs of '_' (or any other non-alphanum, non-dot character), as being identical to a single '-'. >Buildout has a functionality to "pin-down" ("lock down"/"nail down") versions >of eggs (distribution?). There is another functionality to enforce >"pinining-down" >versions of all eggs used in a particular Buildout configuration. If we >use "_" as the package name (distribution name?), this functionality is not >working. Your comparisons should be based on the 'key' attribute of Distribution and Requirement objects, rather than relying on direct string operations of your own. The 'key' attribute contains a form of project name suitable for equality/inequality comparisons. In other words, you should not take unparsed data from your configuration and compare it against pkg_resources attributes. Use constructors like using Requirement.parse() and Distribution.from_filename() to create objects with 'key' attributes, then compare keys, or just use Requirement.__contains__. For example: if someDistribution in Requirement.parse(projname+'=='+exactversion): # someDistribution is exactly version exactversion of projname The pkg_resources API is there precisely so that you don't have to know all the low-level details like syntax rules and escaping.

2 1

Re: [Distutils] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them
by P.J. Eby 11 Mar '10

11 Mar '10

At 10:41 AM 2/25/2010 -0500, Olemis Lang wrote: >PS: BTW, how could I trigger easy_install(ation) at a given point >while implementing a distutils command, and let the command perform >further actions if deps are installed correctly ? Setuptools' "Distribution" object has a method for fetching dependencies. See setuptools' "test" command for an example. (This doesn't install the dependencies globally, just drops eggs into the build directory. But they're there and available to be reused for installation in a later phase, under normal circumstances.)

4 12

Re: [Distutils] use of '_' in package name causing version parsing issue?
by P.J. Eby 11 Mar '10

11 Mar '10

At 09:50 AM 3/10/2010 +0530, Baiju M wrote: >I spend some time with Buildout and setuptools code to identify the issue. >I will try to explain my findings. > >1. Buildout is relying on pkg_resources.Requirement.parse function to > get the "project_name" like this: > > pkg_resources.Requirement.parse('jiva_interface').project_name > > I can see from the code of `Requirement` class that, the `__init__` > method is deprecated and recommend to use `parse` > function. It is undocumented, not deprecated. You are simply not supposed to create instances via that (private) constructor. > Does this mean that we should not use the attributes > of an instance of `Requirement` class? This is very important as > the `parse` function return a list of instances of `Requirement` class. Requirement objects are documented; see: http://peak.telecommunity.com/DevCenter/PkgResources#requirement-objects > So, if it is acceptable to use the "project_name" attribute, then > Buildout can rely on it, right ? Yes. > According to this code, this will be the result: > > pkg_resources.safe_name('jiva_interface') > 'jiva-interface' > > And: > > pkg_resources.Requirement.parse('jiva_interface').project_name > 'jiva-interface' > > Is this behavior correct ? Yes it is. All non-alphanumeric, non-dot characters are replaced with '-' in a project name. This turns project names like e.g. "Foo's Bar Factory" into their canonical form (i.e., "Foo-s-Bar-Factory"). > If you think what setuptools doing is fine, we will make changes > in Buildout code to use the "safe_name" method where ever it directly > get "project_name". Why do you think you need that? (Most likely, you are mistaken, since the only reason the unsafe_name attribute exists is to deal with a limitation in older versions of the PyPI software, which did not support doing "safe_name" redirection.) If you can describe what you're actually trying to do with this information, perhaps there is a safer/more documented way that I can suggest.

2 2

Re: [Distutils] use of '_' in package name causing version parsing issue?
by P.J. Eby 10 Mar '10

10 Mar '10

At 03:03 PM 3/9/2010 -0600, Brad Allen wrote: >Today I was informed of an issue in which buildout (with the latest >setuptools) is not resolving version numbers properly, causing the >wrong package to be selected in some cases. The cause identified was >having '_' in the package name. I suspect there is a miscommunication or misunderstanding somewhere. It is perfectly acceptable to have a '_' in a package name or project name. This: >| >>> a="jiva_interface-2.3.6-py2.6.egg" >| >>> b="jiva_interface-2.3.8-py2.6.egg" >| >>> pkg_resources.parse_version(a) Is the wrong API to use to parse an egg filename, as parse_version() is for parsing a version that's already extracted from a filename. This is the right API for extracting a version from a filename: >>> pkg_resources.Distribution.from_filename(a).version '2.3.6' >>> pkg_resources.Distribution.from_filename(b).version '2.3.8' >>> pkg_resources.Distribution.from_filename(c).version '0.1.1' >>> pkg_resources.Distribution.from_filename(d).version '0.1.2' And here's the correct one for extracting the parsed version from a filename: >>> pkg_resources.Distribution.from_filename(a).parsed_version ('00000002', '00000003', '00000006', '*final') >>> pkg_resources.Distribution.from_filename(b).parsed_version ('00000002', '00000003', '00000008', '*final') >>> pkg_resources.Distribution.from_filename(c).parsed_version ('00000000', '00000001', '00000001', '*final') >>> pkg_resources.Distribution.from_filename(d).parsed_version ('00000000', '00000001', '00000002', '*final') As you can see, these APIs work just fine, so the example given is a red herring, unless Buildout is using the APIs incorrectly (which I really doubt it is). Usually, the situation where people run into trouble with unusual package names or filenames is when they produce a source distribution manually, or by using something other than distutils/setuptools (that has different filename escaping rules), or when they manually rename a file before uploading, and expect it to still work the same. It would be a good idea for you to check which of these things (if any) is taking place, and provide details of the specific problem, with steps to reproduce it, since the example given probably has nothing to do with it. Thanks!

3 4

Re: [Distutils] setuptools error when using --prefix option, complains directory does not exist
by P.J. Eby 10 Mar '10

10 Mar '10

At 02:57 PM 3/9/2010 -0500, hari jayaram wrote: >Hi > >I am installing setuptools using the --prefix option it complains >the directory does not exist even though it does. The install works >however when I use the --instal-dir option as it recommends. > >macbook-pro-17:~ hari$ sudo sh setuptools-0.6c11-py2.6.egg >--prefix=/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages That's because the directory you gave is not a *prefix*, it's an installation directory. A prefix, in distutils terms, is generally the top-level directory under which Python's lib, bin, and other directories are located. So, the correct --prefix would be "/Library/Frameworks/Python.framework/Versions/2.6" in this case. (Notice the absence of lib/python2.6/site-packages.) > [Errno 2] No such file or directory: > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/lib/python2.6/site-packages/test-easy-install-79530.pth' If you look closely at this path, you'll see that when the distutils creates an install directory from your prefix (which is already an installation directory), you end up trying to install to a 'lib/python2.6/site-packages/lib/python2.6/site-packages' directory. That's because when you give a prefix, distutils decides what subdirectories of the prefix to install things in, and it can't tell that you already gave it a subdirectory of the prefix.

2 2

setuptools error when using --prefix option, complains directory does not exist
by hari jayaram 09 Mar '10

09 Mar '10

Hi I am installing setuptools using the --prefix option it complains the directory does not exist even though it does. The install works however when I use the --instal-dir option as it recommends. Thought I would pass this along Hari macbook-pro-17:~ hari$ sudo sh setuptools-0.6c11-py2.6.egg --prefix=/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/lib/python2.6/site-packages/test-easy-install-79530.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/lib/python2.6/site-packages This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option).

2 1

[issue105] include a copy of the licence text
by Zooko O'Whielacronx 09 Mar '10

09 Mar '10

New submission from Zooko O'Whielacronx <zooko(a)zooko.com>: I am trying to get Tahoe-LAFS listed on the FSF/UNESCO directory of Free Software: http://directory.fsf.org/ . One of the sticking points is that Tahoe-LAFS includes a copy of setuptools (http://allmydata.org/trac/tahoe-lafs/browser/misc/dependencies/setuptools-0… ) and, as the FSF representative wrote: """ Next, the following code misc/dependencies/setuptools/EGG-INFO/PKG-INFO calls for either the Zope Public License or the Python Software Foundation License neither of which is present. One of them needs to be added in order to fix the problem. Both can be found through links on our website. """ To fix this, include a copy of the licence text with the distribution. (Or close this ticket as wontfix and I'll work-around it.) ---------- messages: 509 nosy: zooko priority: wish status: unread title: include a copy of the licence text _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue105> _______________________________________________

1 0

Re: [Distutils] Newby Question: install easy_install for python2.5 on RHEL
by P.J. Eby 09 Mar '10

09 Mar '10

At 01:22 PM 3/2/2010 -0800, Walter Coole wrote: >Content-Class: urn:content-classes:message >Content-Type: multipart/alternative; > boundary="----_=_NextPart_001_01CABA4E.86D76612" > >When I try to install with: > > sudo sh setuptools-0.6c11-py2.5.egg > >I get an error: > > error: invalid Python installation: unable to open > /usr/lib64/python2.5/config/Makefile (No such file or directory) > >That I don't really understand. Yes, it's true that there is no >/usr/lib64/python2.5/config directory, but I've gotten along nicely >without one for quite a while. You are probably missing your distribution's "python-dev" or "python-devel" package, which is required to use the distutils at all, let alone setuptools.

2 1

[issue104] bdist_egg ignoring egg_info:writers
by Dan Cotruta 08 Mar '10

08 Mar '10

New submission from Dan Cotruta <dan.cotruta(a)artsalliancemedia.com>: It seems that creating eggs with bdist_egg does not read egg_info writers for extra files to be placed in EGG-INFO. Running setup.py egg_info first fixes the problem, but it seems like this step should not be necessary. Am I mistaken, or is this a real bug? ---------- messages: 507 nosy: dcotruta priority: bug status: unread title: bdist_egg ignoring egg_info:writers _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue104> _______________________________________________

1 0

Re: [Distutils] trouble building setuptools on centos 3.8
by P.J. Eby 06 Mar '10

06 Mar '10

At 02:52 PM 3/2/2010 -0500, Mathieu Jobin wrote: >maybe you can help me. there is something wrong with building this >rpm package ? > >maybe because I am running .... > >Python 2.2.3 (#1, May Â 1 2006, 12:33:49) Setuptools requires Python 2.3, preferably 2.3.5 or higher.

1 0