Today I was informed of an issue in which buildout (with the latest
setuptools) is not resolving version numbers properly, causing the
wrong package to be selected in some cases. The cause identified was
having '_' in the package name. I am still investigating this issue,
but thought it would be worth mentioning on the list in case anyone
had a quick answer.
Here is the note I received which raised the issue:
| > The problem was, they used "_" in the package
| > name. Buildout and many other systems like setuptools,RPM,DEB etc.
| > treat "_" as a special character to distinguish package name and
| > version.
| > So, it's not a good idea to use "_" in the name of a package.
| > The same is the situation with "-" (minus/hyphen) character.
| > The lesson learned: Don't use "-" and "_" in package name.
| >>> import pkg_resources
| >>> a="jiva_interface-2.3.6-py2.6.egg"
| >>> b="jiva_interface-2.3.8-py2.6.egg"
| >>> pkg_resources.parse_version(a)
| ('*jiva', '*_', '*interface', '*final-', '00000002', '00000003',
| '00000006', '*final-', '*py', '00000002', '00000006', '*egg',
| >>> pkg_resources.parse_version(b)
| ('*jiva', '*_', '*interface', '*final-', '00000002', '00000003',
| '00000008', '*final-', '*py', '00000002', '00000006', '*egg',
| >>> c="ZeStorage-0.1.1-py2.4.egg"
| >>> d="ZeStorage-0.1.2-py2.4.egg"
| >>> pkg_resources.parse_version(c)
| ('*zestorage', '*final-', '00000000', '00000001', '00000001',
| '*final-', '*py', '00000002', '00000004', '*egg', '*final')
| >>> pkg_resources.parse_version(d)
| ('*zestorage', '*final-', '00000000', '00000001', '00000002',
| '*final-', '*py', '00000002', '00000004', '*egg', '*final')
At 06:32 PM 3/13/2010 +0100, Lennart Regebro wrote:
>On Sat, Mar 13, 2010 at 18:25, P.J. Eby <pje(a)telecommunity.com> wrote:
> >> Actually, I suddenly remembered -v, and ran python -v. and the local
> >> site.py is *not* being imported, which explains why no matter what I
> >> write in it it gets ignored. So there seems to be at least two
> >> differences here.
> > Perhaps PyPy also has an optimization that makes it use a built-in version
> > of site.py, instead of using standard import logic. Finding that
> would be a
> > bit harder than tracking down the imp problem, though, as it doesn't appear
> > that site is a "builtin" in PyPy the way imp is. It does appear to be
> > imported by http://codespeak.net/pypy/dist/pypy/bin/py.py however.
> > (I'm assuming, btw, that by "local site.py" you mean the patching site.py,
> > not a site.py located in say, PyPy's site-packages directory.)
>Right, I mean the one in the Distribute root. When starting pypy in
>the distribute directory, it will load the site.py, and try to
>"imp.load_module('site',stream,path,descr)". As you meantion, that
>doesn't actually seem to do anything.
>When starting CPython in the distribute directory, the local site.py
Interesting; that seems like an odd point of difference. Perhaps the
'' is added to sys.path later in CPython than it is in PyPy.
(In any case, that 'site.py in the Distribute root' (aka setuptools'
site-patch.py) is only intended to be executed when it's in a
At 05:54 PM 3/13/2010 +0100, Lennart Regebro wrote:
>On Sat, Mar 13, 2010 at 17:42, Lennart Regebro <regebro(a)gmail.com> wrote:
> > On Sat, Mar 13, 2010 at 16:07, P.J. Eby <pje(a)telecommunity.com> wrote:
> >> 2. It is finding a valid site.py, but the standard import protocols (i.e.,
> >> imp.find_module/load_module and
> >> path_importer_cache[modulename].load_module()) are not working correctly
> >> when called on an existing module.
> > Yeah, this seems like the problem. It imports it, but neither globals
> > or locals change. It's hard to say exactly what the problem is as I
> > can't debug what happens in CPython, as no matter what I put into the
> > local site.py, nothing seems to happen. So I't hard to me to check
> > what the behaviour *should* be, but it seems clear that this is where
> > the difference is.
>Actually, I suddenly remembered -v, and ran python -v. and the local
>site.py is *not* being imported, which explains why no matter what I
>write in it it gets ignored. So there seems to be at least two
Perhaps PyPy also has an optimization that makes it use a built-in
version of site.py, instead of using standard import logic. Finding
that would be a bit harder than tracking down the imp problem,
though, as it doesn't appear that site is a "builtin" in PyPy the way
imp is. It does appear to be imported by
(I'm assuming, btw, that by "local site.py" you mean the patching
site.py, not a site.py located in say, PyPy's site-packages directory.)
At 05:42 PM 3/13/2010 +0100, Lennart Regebro wrote:
>On Sat, Mar 13, 2010 at 16:07, P.J. Eby <pje(a)telecommunity.com> wrote:
> > 2. It is finding a valid site.py, but the standard import protocols (i.e.,
> > imp.find_module/load_module and
> > path_importer_cache[modulename].load_module()) are not working correctly
> > when called on an existing module.
>Yeah, this seems like the problem. It imports it, but neither globals
>or locals change. It's hard to say exactly what the problem is as I
>can't debug what happens in CPython, as no matter what I put into the
>local site.py, nothing seems to happen. So I't hard to me to check
>what the behaviour *should* be, but it seems clear that this is where
>the difference is.
What the behavior should be is that when you call load_module (either
on an importer instance or the imp.load_module function), and the
module already exists in sys.modules, that's the module object that
should be updated. reload() relies on this, as do various other
things such as lazy importing tools.
A quick glance at PyPy's source (at
http://codespeak.net/pypy/dist/pypy/module/imp/ ) reveals that it
does support reusing a module in sys.modules, but for some reason
this code is shut off by a keyword argument that defaults to
false. This behavior is *definitely* not what CPython's load_module does.
I didn't dig into whether this is also true for any importers that
PyPy might put into the path importer cache, but it's definitely true
One could argue, I suppose, whether the imp behavior is official
Python spec vs. CPython-only feature, but setuptools isn't the only
thing that relies on it, and PEP 302 *does* makes the reuse behavior
explicitly required for loader objects. Also, its absence would make
it bloody hard (probably impossible, actually) to implement a
reload() function (either the base one or an improved version) in
terms of the imp API.
Simplest fix for this in PyPy would probably be to change
load_module() in modules.imp.interp_inp to call
importing.load_module() with the 'reuse' argument set to True. A
test for this behavior is straightforward: stick a module object into
sys.modules with a variable set, then use load_module() to load
something that sets another variable, then check that both variables
are set in the module.
(Also, it should assert that load_module() returns the module object
that was *already* in sys.modules, rather than that it just shoved a
new module into sys.modules -- which is what the current PyPy code
We worked during the Pycon sprint on PEP 376. The goal was to finish
the section about the RECORD file syntax. This file contains the list
of installed files when a project is installed using distutils (and
For all the files that are supposed to land in the Python tree on the
system, like Python modules, it's pretty clear and simple. The fuzzy
part in distutils has always been about resource files (configuration
files, help files, etc.) that should land on specific locations on the
target system if we want to be FHS-compliant.
We ended up working on something bigger for that. We wanted to make it
easier for developer to describe what their projects contain, and
easier for OS packagers to place resources files of a project in the
For instance, a ini-file in your Foo project, called foo.cfg, should
land in /etc/ under Debian. From a developer point of view, the
location of the file on the target system doesn't really matter as
long as he can reach it somehow in its code. People that want a
generic solution put that file in the source tree and use __file__ to
find the path. That is not really robust and doesn't work if the
source tree is in a zip file for instance. Setuptools provides an api
for that, but the resource cannot be relocated on the system by the
We worked on a solution to make life easier for developers and
packagers. IOW, let the packager define the paths of the files on the
system, and provide an API for the developer to get the file back.
Here's the proposal :
1. A ``resources`` section in ``setup.cfg`` that maps resource files
to their categories (and optionally subdirectory prefixes within those
2. A ``sysconfig.cfg`` file at the system Python level that maps
categories to a position on the filesystem
3. A simple ``pkgutil.open()`` API to access resources from code
Please read the full detailed proposal here :
comment ! :)
I am working on finishing some details at the end of the proposal, but
the document is already readable (the unfinished parts are after the
If we can manage to finish the discussion on this in the upcoming
weeks and get it accepted by Guido, we could include in 2.7 a
sysconfig.cfg file that is used by the existing sysconfig.py APIs
(right now it is using a python dict for the paths), and add APIs in
If we finish this work too late, (which is most likely to be the case
because 2.7b1 is due in a few weeks), we will add these APIs in
distutils2 I guess, and see if we can add them in the next version of
3.x (or 2.8 ;) )
Tarek Ziadé | http://ziade.org
At 10:49 AM 3/13/2010 +0100, Lennart Regebro wrote:
>On Sat, Mar 13, 2010 at 10:13, Lennart Regebro <regebro(a)gmail.com> wrote:
> > Well, I don't know if it does, but adding the makepath method from
> > Pythons site.py seems to solve that problem anyway, and you get other
> > errors. I don't have time to look into this very deeply, but
> > Distribute/Setuptools seems to be a good test for cross-platform
> > compatibility as it involves much magic. :)
>Well, PyPy distutils.sysconfig does not have a customize_compiler, so
>it breaks there anyway.
>But I realized something more: Could it be that in CPython, Pythons
>site.py is read in first and that context is retained when running a
No. The site-patch code explicitly finds and loads the old module
into the same namespace, in order to avoid the compatibility issues I
mentioned previously. (i.e., some distros patch the site module to
add custom site paths, despite the fact that the whole purpose of
.pth files is to make it so you *don't* have to patch site.py to add
>Because now I again get the error that another method
>is missing (addsitedir) which does exist in Pythons site.py. It exists
>in PyPy's site.py as well, but not in distributes site.py.
Here are some possible ways you could get that behavior:
1. site-patch cannot load PyPy's built-in site module, and is instead
finding an *empty* site.py on sys.path after it, OR
2. It is finding a valid site.py, but the standard import protocols
(i.e., imp.find_module/load_module and
path_importer_cache[modulename].load_module()) are not working
correctly when called on an existing module.
If you trace the execution of the site-patch to whichever load_module
line is getting run, and print the value of the 'item' variable,
you'll see what path entry the "original" site is being discovered
in, and you'll then be able to tell which of the above possibilities applies.
(Offhand, I'd guess that the import bug is perhaps a bit more likely;
the reloading behavior of loaders and imp is a little on the obscure
side and might not be sufficiently covered by the stdlib or PyPy
tests. The likely failure is that some piece of code is returning a
*new* module object, rather than updating the existing one as PEP 302
At 12:45 AM 3/13/2010 +0100, Lennart Regebro wrote:
>No, it's the site.py in the Distribute tree I'm talking about. Sorry
>for being unclear about this. It may be a broken copy, yes.
Since distribute is a fork of the 0.6 branch of setuptools, it's
presumably the special site.py loader (called
setuptools/site-patch.py in the 0.7 trunk) that's used for making
PYTHONPATH directories support .pth files. It's placed in PYTHONPATH
directories used with easy_install, and it expects to find another
'site' module loadable on sys.path.
It was done this way because it needed to see an unmangled sys.path
before the site module gets a hold of it, and because it couldn't
simply *replace* the site module without interfering with Linux
distro-specific patches to the site module to add in additional
site-package directories in various nonstandard ways that would be
better done in .pth files to begin with. ;-)
Anyway, if PyPy includes a precompiled or "builtin" site module of
some kind, it's unlikely that the patch file would work correctly,
even if PyPy allows executing a custom site.py.
What exactly is site.py for? I tried installing Distribute with PyPy,
and it fails, because if you start it, it runs site.py, and site.py
uses makepath, but doesn't import or define it, so it fails.
Running the tests with python2.6 and it seems site.py is never run,
but removing it makes the tests fail.
Any insight into this?
Lennart Regebro: Python, Zope, Plone, Grok
+33 661 58 14 64
At 11:42 AM 3/12/2010 +0530, Baiju M wrote:
>On Thu, Mar 11, 2010 at 9:13 PM, P.J. Eby <pje(a)telecommunity.com> wrote:
> > At 12:38 PM 3/11/2010 +0530, Baiju M wrote:
> >> On Thu, Mar 11, 2010 at 11:05 AM, Baiju M <mbaiju(a)zeomega.com> wrote:
> >> > If "_" is a valid project_name identifier, why it is replaces with "-" ?
> > In order to have a canonicalized name form which can be escaped
> in filenames
> > for unambiguous identification of an egg's project and version.
> > Egg filenames use '-' as a separator between name, version, python version,
> > and platform. A '-' in any of these components is escaped as '_', so that
> > the '-' remains a viable and unambiguous separator. This means that '_'
> > gets turned back into a '-' when unescaped, so the mapping between '_' and
> > '-' is part of the safe_name canonical form.
>Sorry, I confused.
>Then, why "_" was added as a valid project name identifier ?
It's a valid project name character; what it's not is a *canonical*
project name character.