Mailman 3 March 2010 - Distutils-SIG

use of '_' in package name causing version parsing issue?
by Brad Allen March 15, 2010

March 15, 2010

Today I was informed of an issue in which buildout (with the latest setuptools) is not resolving version numbers properly, causing the wrong package to be selected in some cases. The cause identified was having '_' in the package name. I am still investigating this issue, but thought it would be worth mentioning on the list in case anyone had a quick answer. Here is the note I received which raised the issue: | > The problem was, they used "_" in the package | > name. Buildout and many other systems like setuptools,RPM,DEB etc. | > treat "_" as a special character to distinguish package name and | > version. | > So, it's not a good idea to use "_" in the name of a package. | > The same is the situation with "-" (minus/hyphen) character. | > | > The lesson learned: Don't use "-" and "_" in package name. | | >>> import pkg_resources | >>> a="jiva_interface-2.3.6-py2.6.egg" | >>> b="jiva_interface-2.3.8-py2.6.egg" | >>> pkg_resources.parse_version(a) | ('*jiva', '*_', '*interface', '*final-', '00000002', '00000003', | '00000006', '*final-', '*py', '00000002', '00000006', '*egg', | '*final') | >>> pkg_resources.parse_version(b) | ('*jiva', '*_', '*interface', '*final-', '00000002', '00000003', | '00000008', '*final-', '*py', '00000002', '00000006', '*egg', | '*final') | >>> c="ZeStorage-0.1.1-py2.4.egg" | >>> d="ZeStorage-0.1.2-py2.4.egg" | >>> pkg_resources.parse_version(c) | ('*zestorage', '*final-', '00000000', '00000001', '00000001', | '*final-', '*py', '00000002', '00000004', '*egg', '*final') | >>> pkg_resources.parse_version(d) | ('*zestorage', '*final-', '00000000', '00000001', '00000002', | '*final-', '*py', '00000002', '00000004', '*egg', '*final') |

3 3

setuptools dependencies failure and wildcards
by Nicholas Bower March 14, 2010

March 14, 2010

I have the following simple dependencies using setuptools; install_requires=['shapely==1.0.14','sqlalchemy==0.5.8'], Two questions; 1) Shapely will not work. The available packages are; Shapely-1.0.14.win32.py24.exe (md5). Shapely-1.0.14.win32.py25.exe (md5). Shapely-1.0.14.win32.py26.exe (md5). but python version is not being appended as below. Is shapely thus incompatible with setuptools? If so, what are the name conventions the author should be using and when does setuptools try for the current python version only? Searching for shapely==1.0.14 Reading http://pypi.python.org/simple/shapely/ Reading http://trac.gispython.org/projects/PCL/wiki/Shapely Reading http://trac.gispython.org/projects/PCL/wiki/ShapeLy Best match: Shapely 1.0.14 Downloading http://pypi.python.org/packages/2.5/S/Shapely/Shapely-1.0.14.win32.e xe#md5=92281555b23623b8cf7b4fbc9d01f17a error: Can't download http://pypi.python.org/packages/2.5/S/Shapely/Shapely-1.0.14.win32.exe: 404 Not Found 2) How does one say "the latest release of sqlalchemy 0.5.x". Unfortunately <0.6 picks up 0.6beta and I can see no way to do wildcards in the docs and just say "give me the latest 0.5 series". Thanks, Nick

2 1

Re: [Distutils] Pypy and site.py and makepath
by P.J. Eby March 14, 2010

March 14, 2010

At 06:32 PM 3/13/2010 +0100, Lennart Regebro wrote: >On Sat, Mar 13, 2010 at 18:25, P.J. Eby <pje(a)telecommunity.com> wrote: > >> Actually, I suddenly remembered -v, and ran python -v. and the local > >> site.py is *not* being imported, which explains why no matter what I > >> write in it it gets ignored. So there seems to be at least two > >> differences here. > > > > Perhaps PyPy also has an optimization that makes it use a built-in version > > of site.py, instead of using standard import logic. Finding that > would be a > > bit harder than tracking down the imp problem, though, as it doesn't appear > > that site is a "builtin" in PyPy the way imp is. It does appear to be > > imported by http://codespeak.net/pypy/dist/pypy/bin/py.py however. > > > > (I'm assuming, btw, that by "local site.py" you mean the patching site.py, > > not a site.py located in say, PyPy's site-packages directory.) > >Right, I mean the one in the Distribute root. When starting pypy in >the distribute directory, it will load the site.py, and try to >"imp.load_module('site',stream,path,descr)". As you meantion, that >doesn't actually seem to do anything. > >When starting CPython in the distribute directory, the local site.py >is ignored. Interesting; that seems like an odd point of difference. Perhaps the '' is added to sys.path later in CPython than it is in PyPy. (In any case, that 'site.py in the Distribute root' (aka setuptools' site-patch.py) is only intended to be executed when it's in a PYTHONPATH directory.)

2 1

Re: [Distutils] Pypy and site.py and makepath
by P.J. Eby March 13, 2010

March 13, 2010

At 05:54 PM 3/13/2010 +0100, Lennart Regebro wrote: >On Sat, Mar 13, 2010 at 17:42, Lennart Regebro <regebro(a)gmail.com> wrote: > > On Sat, Mar 13, 2010 at 16:07, P.J. Eby <pje(a)telecommunity.com> wrote: > >> 2. It is finding a valid site.py, but the standard import protocols (i.e., > >> imp.find_module/load_module and > >> path_importer_cache[modulename].load_module()) are not working correctly > >> when called on an existing module. > > > > Yeah, this seems like the problem. It imports it, but neither globals > > or locals change. It's hard to say exactly what the problem is as I > > can't debug what happens in CPython, as no matter what I put into the > > local site.py, nothing seems to happen. So I't hard to me to check > > what the behaviour *should* be, but it seems clear that this is where > > the difference is. > >Actually, I suddenly remembered -v, and ran python -v. and the local >site.py is *not* being imported, which explains why no matter what I >write in it it gets ignored. So there seems to be at least two >differences here. Perhaps PyPy also has an optimization that makes it use a built-in version of site.py, instead of using standard import logic. Finding that would be a bit harder than tracking down the imp problem, though, as it doesn't appear that site is a "builtin" in PyPy the way imp is. It does appear to be imported by http://codespeak.net/pypy/dist/pypy/bin/py.py however. (I'm assuming, btw, that by "local site.py" you mean the patching site.py, not a site.py located in say, PyPy's site-packages directory.)

2 1

Re: [Distutils] Pypy and site.py and makepath
by P.J. Eby March 13, 2010

March 13, 2010

At 05:42 PM 3/13/2010 +0100, Lennart Regebro wrote: >On Sat, Mar 13, 2010 at 16:07, P.J. Eby <pje(a)telecommunity.com> wrote: > > 2. It is finding a valid site.py, but the standard import protocols (i.e., > > imp.find_module/load_module and > > path_importer_cache[modulename].load_module()) are not working correctly > > when called on an existing module. > >Yeah, this seems like the problem. It imports it, but neither globals >or locals change. It's hard to say exactly what the problem is as I >can't debug what happens in CPython, as no matter what I put into the >local site.py, nothing seems to happen. So I't hard to me to check >what the behaviour *should* be, but it seems clear that this is where >the difference is. What the behavior should be is that when you call load_module (either on an importer instance or the imp.load_module function), and the module already exists in sys.modules, that's the module object that should be updated. reload() relies on this, as do various other things such as lazy importing tools. A quick glance at PyPy's source (at http://codespeak.net/pypy/dist/pypy/module/imp/ ) reveals that it does support reusing a module in sys.modules, but for some reason this code is shut off by a keyword argument that defaults to false. This behavior is *definitely* not what CPython's load_module does. I didn't dig into whether this is also true for any importers that PyPy might put into the path importer cache, but it's definitely true for imp.load_module. One could argue, I suppose, whether the imp behavior is official Python spec vs. CPython-only feature, but setuptools isn't the only thing that relies on it, and PEP 302 *does* makes the reuse behavior explicitly required for loader objects. Also, its absence would make it bloody hard (probably impossible, actually) to implement a reload() function (either the base one or an improved version) in terms of the imp API. Simplest fix for this in PyPy would probably be to change load_module() in modules.imp.interp_inp to call importing.load_module() with the 'reuse' argument set to True. A test for this behavior is straightforward: stick a module object into sys.modules with a variable set, then use load_module() to load something that sets another variable, then check that both variables are set in the module. (Also, it should assert that load_module() returns the module object that was *already* in sys.modules, rather than that it just shoved a new module into sys.modules -- which is what the current PyPy code apparently does.)

1 0

RFC: enhanced file list description for Distutils (PEP 376 extension)
by Tarek Ziadé March 13, 2010

March 13, 2010

Hello, We worked during the Pycon sprint on PEP 376. The goal was to finish the section about the RECORD file syntax. This file contains the list of installed files when a project is installed using distutils (and distutils-based tools.) For all the files that are supposed to land in the Python tree on the system, like Python modules, it's pretty clear and simple. The fuzzy part in distutils has always been about resource files (configuration files, help files, etc.) that should land on specific locations on the target system if we want to be FHS-compliant. We ended up working on something bigger for that. We wanted to make it easier for developer to describe what their projects contain, and easier for OS packagers to place resources files of a project in the proper location. For instance, a ini-file in your Foo project, called foo.cfg, should land in /etc/ under Debian. From a developer point of view, the location of the file on the target system doesn't really matter as long as he can reach it somehow in its code. People that want a generic solution put that file in the source tree and use __file__ to find the path. That is not really robust and doesn't work if the source tree is in a zip file for instance. Setuptools provides an api for that, but the resource cannot be relocated on the system by the packagers. We worked on a solution to make life easier for developers and packagers. IOW, let the packager define the paths of the files on the system, and provide an API for the developer to get the file back. Proposal ------------ Here's the proposal : 1. A ``resources`` section in ``setup.cfg`` that maps resource files to their categories (and optionally subdirectory prefixes within those categories) 2. A ``sysconfig.cfg`` file at the system Python level that maps categories to a position on the filesystem 3. A simple ``pkgutil.open()`` API to access resources from code Please read the full detailed proposal here : http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst, and comment ! :) Calendar ------------ I am working on finishing some details at the end of the proposal, but the document is already readable (the unfinished parts are after the API section). If we can manage to finish the discussion on this in the upcoming weeks and get it accepted by Guido, we could include in 2.7 a sysconfig.cfg file that is used by the existing sysconfig.py APIs (right now it is using a python dict for the paths), and add APIs in pkgutil. If we finish this work too late, (which is most likely to be the case because 2.7b1 is due in a few weeks), we will add these APIs in distutils2 I guess, and see if we can add them in the next version of 3.x (or 2.8 ;) ) Regards, Tarek -- Tarek Ziadé | http://ziade.org

1 0

Re: [Distutils] Pypy and site.py and makepath
by P.J. Eby March 13, 2010

March 13, 2010

At 10:49 AM 3/13/2010 +0100, Lennart Regebro wrote: >On Sat, Mar 13, 2010 at 10:13, Lennart Regebro <regebro(a)gmail.com> wrote: > > Well, I don't know if it does, but adding the makepath method from > > Pythons site.py seems to solve that problem anyway, and you get other > > errors. I don't have time to look into this very deeply, but > > Distribute/Setuptools seems to be a good test for cross-platform > > compatibility as it involves much magic. :) > >Well, PyPy distutils.sysconfig does not have a customize_compiler, so >it breaks there anyway. > >But I realized something more: Could it be that in CPython, Pythons >site.py is read in first and that context is retained when running a >local site.py? No. The site-patch code explicitly finds and loads the old module into the same namespace, in order to avoid the compatibility issues I mentioned previously. (i.e., some distros patch the site module to add custom site paths, despite the fact that the whole purpose of .pth files is to make it so you *don't* have to patch site.py to add such paths!) >Because now I again get the error that another method >is missing (addsitedir) which does exist in Pythons site.py. It exists >in PyPy's site.py as well, but not in distributes site.py. Here are some possible ways you could get that behavior: 1. site-patch cannot load PyPy's built-in site module, and is instead finding an *empty* site.py on sys.path after it, OR 2. It is finding a valid site.py, but the standard import protocols (i.e., imp.find_module/load_module and path_importer_cache[modulename].load_module()) are not working correctly when called on an existing module. If you trace the execution of the site-patch to whichever load_module line is getting run, and print the value of the 'item' variable, you'll see what path entry the "original" site is being discovered in, and you'll then be able to tell which of the above possibilities applies. (Offhand, I'd guess that the import bug is perhaps a bit more likely; the reloading behavior of loaders and imp is a little on the obscure side and might not be sufficiently covered by the stdlib or PyPy tests. The likely failure is that some piece of code is returning a *new* module object, rather than updating the existing one as PEP 302 requires.)

2 2

Re: [Distutils] Pypy and site.py and makepath
by P.J. Eby March 13, 2010

March 13, 2010

At 12:45 AM 3/13/2010 +0100, Lennart Regebro wrote: >No, it's the site.py in the Distribute tree I'm talking about. Sorry >for being unclear about this. It may be a broken copy, yes. Since distribute is a fork of the 0.6 branch of setuptools, it's presumably the special site.py loader (called setuptools/site-patch.py in the 0.7 trunk) that's used for making PYTHONPATH directories support .pth files. It's placed in PYTHONPATH directories used with easy_install, and it expects to find another 'site' module loadable on sys.path. It was done this way because it needed to see an unmangled sys.path before the site module gets a hold of it, and because it couldn't simply *replace* the site module without interfering with Linux distro-specific patches to the site module to add in additional site-package directories in various nonstandard ways that would be better done in .pth files to begin with. ;-) Anyway, if PyPy includes a precompiled or "builtin" site module of some kind, it's unlikely that the patch file would work correctly, even if PyPy allows executing a custom site.py.

2 2

Pypy and site.py and makepath
by Lennart Regebro March 12, 2010

March 12, 2010

What exactly is site.py for? I tried installing Distribute with PyPy, and it fails, because if you start it, it runs site.py, and site.py uses makepath, but doesn't import or define it, so it fails. Running the tests with python2.6 and it seems site.py is never run, but removing it makes the tests fail. Any insight into this? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

3 3

Re: [Distutils] use of '_' in package name causing version parsing issue?
by P.J. Eby March 12, 2010

March 12, 2010

At 11:42 AM 3/12/2010 +0530, Baiju M wrote: >On Thu, Mar 11, 2010 at 9:13 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 12:38 PM 3/11/2010 +0530, Baiju M wrote: > >> > >> On Thu, Mar 11, 2010 at 11:05 AM, Baiju M <mbaiju(a)zeomega.com> wrote: > >> > If "_" is a valid project_name identifier, why it is replaces with "-" ? > > > > In order to have a canonicalized name form which can be escaped > in filenames > > for unambiguous identification of an egg's project and version. > > > > Egg filenames use '-' as a separator between name, version, python version, > > and platform. A '-' in any of these components is escaped as '_', so that > > the '-' remains a viable and unambiguous separator. This means that '_' > > gets turned back into a '-' when unescaped, so the mapping between '_' and > > '-' is part of the safe_name canonical form. > >Sorry, I confused. > >Then, why "_" was added as a valid project name identifier ? It's a valid project name character; what it's not is a *canonical* project name character.

1 0