I've updated distlib as follows:
1. I've added support for the equivalent of "pkg_resources.resource_filename"
for returning a true filesystem path for a resource. I've added info about
this in the "Distlib's design" document, though the reference API has not
been fully updated - I thought I'd see if there was any feedback about the
approach I've used. Code and tests have been added, though.
2. I've added the PyPI client code from distutils2 in distlib.pypi. I only
tweaked it a little to run under 2.x and 3.x from a single code base,
ensuring that all the tests passed. I hope to update it with better support
for finding archives not hosted on PyPI, using hints.
I'm also considering the following changes, which I'd like some feedback on:
* Merging the distlib.depgraph package into the distlib.database package, as
they are fairly interlinked (e.g. adding requires() and required_by() methods
* Removing depgraph.main() and its test, because I don't think it belongs in the
library layer, but is better implemented as a utility script.
Please let me have your comments.
I thought it would be interesting to get an idea of the relative
popularity of the various pkg_resources calls in the interest of
emulating a subset. So I looked it up on Nullege. The absolute
popularity is certainly different from the numbers below.
I think of require() and load_entry_point() used in console scripts as
implementing a dynamic linker, so for example if some code crashes
when the KDEWallet bindings are installed, you would still be able to
run it by omitting those from sys.path. Gem certainly works this way.
I'm not 100% sure pkg_resources works that way. Those functions
traverse the dependency graph at runtime which is fast because it
doesn't include the "search pypi" stuff.
resource_filename is very popular. I would have thought
resource_stream would be more popular. Unless your package is zipped
resource_filename is trivial to implement.
If you are writing an installer, you can unpack a single distribution
to a folder, run find_distributions() on that folder, and get a
Distribution() object with the dependencies as a dict.
-- Daniel H
pkg_resources APIs by number of Nullege Call() counts:
'declare_namespace', 643; obsoleted by Python 3.3 or pkgutil
'working_set', 55 (all samples); not a function
'find_distributions', 25; needed by installers
'Distribution', 22; needed by installers
'ResolutionError', 6 samples
'VersionConflict', 13 samples
'DistributionNotFound', 41 samples
'UnknownExtra', 8 samples
'safe_name', trivial regexp replace
'yield_lines', 15; trivial
'split_sections', 9; .ini-style [section] parser
'safe_extra', 0; another regexp replacement
'to_filename', 9; another text replacement
# Constants that control which kinds of dists are preferred
'NullProvider', subclassed a few times for unit test mocks
'DefaultProvider', 21 mentions
'ZipProvider', 1 subclass
'register_loader_type', 6; one user is a code signing library
I'm trying to setup a modular buildout configuration, but I'm hitting my
head on an invisible wall...
Basically, I'd like to have a top-level buildout.cfg that "collects"
settings from various "subprojects"; in particular, each subproject may
"augment" a set of eggs, some of which may be defined as "develop" (in
buildout parlance). Finally, with a single "zc.recipe.egg" I wanna
install a custom interpreter with all those eggs inside.
So, I have the top-level configuration that uses "extends" to include a
extends = base.cfg projects.cfg
that in turn brings in the various subprojects snippets thru "extends":
extends = src/this-project.cfg src/that-project.cfg
these snippets typically contain a section like
eggs += this-project-egg some-other-dependency-egg
develop += src/this-project
parts += this-project-specific-part
The problem I'm having is that for incomprensible (to me, of course :)
reason some of those settings do not "emerge" at the top level (others
do!), as if a later "extends" is clobbering a previous one.
I built a minimalistic example  that shows the unexpected behaviour;
doing a "buildout annotate" I get something like the following:
If you look at the [projects] section, you can see that while the "eggs"
setting carries the three expected values (one for each subproject plus
one extra), the "develop" setting contains just a single value, the one
injected by proj2, and the one that should be coming from proj1 is
missing. If I omit "src/proj2" from the extends option, then the
"develop" setting correctly contains just "src/proj1"...
Of course I tried hard to understand what's going, even walking into
buildout 1.5.2 code with pdb. I tried also with the latest development
version (the upcoming 1.6) as well as the github branch (IIUC it
derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is
slightly different, and at first it seemed doing the right thing, but at
a closer look it really moved the problem from the "develop" setting
As far as I can tell, there is something strange happening in the
_update_section() function when the _open() function handles the
"extends" setting: at some point it "loses" the cognition that a certain
assignment is an expansion (tecnically, that the key is actually
"develop +", instead of "develop"), so the final "s1.update(s2)"
overrides it, instead of augmenting it...
Of course I may be doing the wrong thing or missing the obvious, and if
so I'll be happy if you can shed some light on me.
Thanks in advance for any help and sorry for the long post,
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele(a)metapensiero.it | -- Fortunato Depero, 1929.
> Of course it's not really a mystery; a tremendous number of Apis only accept
> a file name. That's why it's there. And only something well connected to
> the importer system knows how and whether the extraction is needed.
Why is that the case? If the bytes are available, then whoever needs can
write them to a temp file, pass that name, and clean up afterwards. Is
managing a cache of arbitrary files extracted from zips is a mainstream use
case which distlib needs to support?
Yes, and I've added its package.yaml to the original Gist at
> From: Daniel Holth <dholth(a)gmail.com>
>To: Vinay Sajip <vinay_sajip(a)yahoo.co.uk>
>Sent: Monday, 1 October 2012, 15:27
>Subject: Re: [Distutils] [Python-Dev] distlib updated with resources API
>Have you read the IPython setup.py?
I've been comparing how PEP386/distutils2.version treats versions
and how pkg_resources from setuptools treats versions and it
confirmed a worry of me with the way that dev is treated in PEP386.
In PEP386 you have several kinds of "top level" releases, these
are alpha, beta, candidate, and final and they are sorted as you
would expect. On top of that it has pre and post releases for
any of those top level releases. The pre releases are tagged
with a ".dev" and post releases are tagged with a ".post". They
are sorted immediately before/after the "main" release that they
are pre/post for.
In pkg_resources dev is treated as it's own "main" release and
sorts it in front of an alpha.
This means that given the versions:
["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"]
PEP386 will sort them as:
["0.1a1", "0.1b1", "0.1c1", "0.1.dev1", "0.1"]
and pkg_resources will sort them as:
["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"]
To further complicate things the most common usage I've personally seen
in the wild is that of "0.1dev" or "0.1dev1" which the author expects to sort
before an alpha (In this case distutils2.version throws an error, but the suggest
function is able to turn it into 0.1.dev1).
I think this difference is going to cause confusion, especially during the the
transition period when you're going to have people using both pkg_resources
and the new PEP386 functions.
Since PEP386 is only in the "Accepted" stage, and isn't part of the official
implementation yet, is it at all possible to revise it? Ideally I think to follow
with the prior art, and people's expectations "dev" should be moved to a "main"
release type sorted before an alpha, and to take it's place as a pre release modifier
perhaps something like "pre" can be used instead (e.g. "0.1.pre1").
I'm pleased to announce wheel version 0.10.1.
In this version of wheel, Paul Moore has done a lot of work on wheel's
reference installer, a simple installer that can only install from
local wheel files. The wheel installer can now accept pip-style
requirements files but it doesn't automatically install dependencies,
and it does not uninstall conflicting packages i.e. previous versions
of the same package. If you need a mature full-featured installer you
should use pip.
A notable feature of this version of wheel is that you can use it even
when your virtualenv does not contain setuptools or distribute. For
/tmp/venv/bin/python wheel-0.10.1-py2.py3-none-any.whl/wheel install
executes wheel's wheel to install itself. This feature should be handy
for bootstrapping. Wheel's own scripts still use pkg_resources, so you
should invoke the tool with "python -m wheel" if you do not want to
install distribute or setuptools.
On my machine, it takes about 0.7 seconds to install IPython from a wheel.