interesting point and worth a discussion about different roles (developer, system
admin, final user etc.) having different needs.
I believe distutils is used as tool primarily (setup.py bdist_rpm/msi to create
installable objects, setup.py bdist_sdist to manage the source code etc.): this
complicates the landscape.
Some developer has expectation (wrong IMHO) to replace a whole lot of tools with
it (downloaders/installers/package managers/etc.): this is an uphill struggle
against already widely deployed systems (dpkg/yum/even windows has something!).
I think Tarek did some work in the past and the result is visible in the "The
Hitchhiker’s Guide to Packaging" but I've no idea where it went in the end ...
only is left is a punch-in-the-eye page (http://www.python.org/doc) :D
These days I'm stuck with the old KISS approach and I write a setup.py to create
"packages" and a makefile to create doc, run tests etc. I'm fairly happy with that.
On Wed 14/11/12 08:32, "Ronald Oussoren" ronaldoussoren(a)mac.com wrote:
> I'm not convinced that distutils is untestable. A major problem with
> distutils is that its API is barely documented, which effectly makes
> all of it public API (simular to asyncore). IIRC that's the main
why distutils is frozen right now: with a lot of changes to distutils
started to complain that this could break there there distutils
> The lack of a specification also makes testing harder, as it is
what should be tested.
> Distutils-SIG maillist - Distutils-SIG(a)python.org
Distutils is good enough: it could be better but for what is required
(essentially copying files and creating packages installers) is fine. The only
corner case is an absolute pain in the neck is in the cross compile scenario.
Currently I don't have *any* need for "auto" tools (setuptools and descendants).
As an example I always advice to avoid like a plague anything that depends on
setuptools... with all the due respect I think is the poor's developer attempt to
play the sys admin game.
Even virtual env is a poor work around on the python interpreter not being
relocatable (as in a portable app fashion).
On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren(a)mac.com wrote:
> On 13 Nov, 2012, at 16:00, Daniel Holth <dholth@gmail
> .com> wrote:
> > I want to remove distutils from the standard
> Why? Distutils may not be perfect, but is usable for basic packages. It
> could even be enhanced to support these peps and be even more useable,
> although patches for that would run into the self-imposed freeze of
> distutils development.
> Distutils-SIG maillist - Distutils-SIG(a)python.org
I think Metadata 1.3 is done. Who would like to czar?
On Oct 22, 2012 12:53 PM, "Daniel Holth" <dholth(a)gmail.com> wrote:
> Based on the other "required" field's absence in the wild, only
> "Metadata-Version", "Name", "Version", and "Summary" are required.
> Hopefully a clearer explanation of 0, 1, or many occurrences of each
Working on distlib, I noticed that PyPI contains projects with circular
dependencies. Here's an extract from setup.py in the latest Zope2 (2.13.19):
'Products.OFSP >= 2.13.2',
] + additional_install_requires,
whereas the the setup.py in the latest Products.OFSP (2.13.2) has:
'Zope2 >= 2.13.0a1',
So, according to the declarations, each package depends on the other. Can
setuptools / distribute deal with this sort of situation? If so, how does
I am performing a "fun" experiment with Plone in which I plan to
include a variety of namespace packages in a single distribution. This
works great with non-namespaced packages. Or rather I should say: if
you combine a series of name spaced packages into a flat namespace, it
works. E.g. the Products and plone packages:
However, this completely falls down (at the distutils level, I guess)
when you attempt to do the same thing with namespace packages e.g.:
In distutils parlance, it looks like this (this works):
What I'd love to be able to do is to continue to add items to package_dir e.g.:
Because this would allow me to use git submodules. But, I'm guessing,
since distutils never supported namespace packages (it's a setuptools
feature, right?) then this will never work as I'd like it to. Aside
from actually combining all the namespace packages, is there any other
Thanks for any thoughts,
Alex Clark · https://www.gittip.com/aclark4life/
Following some changes to distlib, there have been improvements in how it can
resolve dependencies. I ran a test on all source archives reachable through
PyPI, and initial results seem to show that out of over 25,000 projects and
112,000 source archives, we can extract dependency metadata from all but 2,540
archives relating to 916 projects.
The distlib locator code has been updated to access this metadata, and this
makes it possible to resolve dependency graphs reasonably quickly, without
downloading any archives.
The dependency finder command-line script is at
The script, finddeps.py, when given a distribution name, will attempt to locate
it and its dependencies, identify requirements which couldn't be satisfied,
show a topological sort (advisory, given the existence of circular dependencies
on many PyPI projects) and a download ordering.
Example results are given in the same Gist for:
Flask (3 dists in all, resolved in ~1 second)
apycotbot (22 dists in all, resolved in ~5 seconds)
collective.megaphone (242 dists in all, resolved in ~55 seconds)
You can also specify constraints, e.g.
$ python finddeps.py "pyramid (> 1.0, < 1.3)"
I also included a separate script for testing the case described in
as mentioned by Carl. In this case, we get the expected result, with the
dependency tree looking like this:
B 1.0 [B]
D 0.9 [D (<= 0.9)]
C 1.0 [C]
D 0.9 [D (<= 1.1)]
D 0.9 [D (<= 1.1)]
D 0.9 [D (<= 0.9)]
If anyone has the time to try out distlib, I'd appreciate some feedback on the
dependency finder (or anything else for that matter) and any unexpected results
you get in your trials.
You should be able to run the dependency finder on most projects on PyPI, but
not ones which don't have a reachable download (for example, "Goose". But
there, "pip install Goose" fails, too).
Thanks and regards,
I'll give you that number(?) but ...
mercurial, docutils, jinjia2 pygments, sphinx, lxml, nose, cherrypy, django, pyqt ...
all they don't need/use setuptools: that 25% left is quite an interesting field
to play in.
If setuptools was "significant packaging innovation" do you think people wouldn't
have embraced already?
Allow me to call *that* a nonsense.
ps. my experience is on the field, so please give me the credit of many years of
experience if I'm say I'm not that keen on "auto" tools: in this kiss rules.
On Tue 13/11/12 17:35, "Daniel Holth" dholth(a)gmail.com wrote:
> Setuptools! You would avoid 75% of pypi. It is nonsense to pretend
> that setuptools is not a significant packaging innovation. Its main
> flaw is that it is based on distutils, a non-extensible design.
> distutils2 is a lot of setuptools and distutils code with the
> plug-ability taken out.
> Perhaps I should say that I would like distutils to become as
> relevant to packaging as the cgi module is to web development. It is
> not a short-term goal.
Although I think the ~ is a very ugly -, it could be useful to change the
separator to something less commonly used than the -.
It would be useful to be able to use the hyphen - in the version of a
package (for semver) and elsewhere. Using it as the separator could make
parsing the file name a bit trickier than is healthy.
This change would affect PEP 376 which reads:
This distinct directory is named as follows::
name + '-' + version + '.dist-info'
with today's hyphen/underscore folding: re.sub('[^A-Za-z0-9.]+', '-',
version), could become
It would also affect pip, setuptools, and the wheel peps. If we do this, I
would like to allow Unicode package names at the same time. safe_name(),
the pkg_resources function that escapes package names for file names, would
re.sub(u"[^\w.]+", "_", u"package-name", flags=re.U)
In other words, the rule for package names would be that they can contain
any Unicode alphanumeric or _ or dot. Right now package names cannot
practically contain non-ASCII because the setuptools installation will fold
it all to _ and installation metadata will collide on the disk.
safe_version(), presently the same as safe_name() would also need to allow
+ for semver.
Does anyone have the energy to actually implement a proof-of-concept?
I'd like to submit the Wheel PEPs 425 (filename metadata), 426
(Metadata 1.3), and 427 (wheel itself) for acceptance. The format has
been stable since May and we are preparing a patch to support it in
pip, but we need to earn consensus before including it in the most
widely used installer.
Wheel is a binary packaging format that is mostly based on the
phenomenal work done by 'packaging' as PEP 376 et al. The resulting
packages solve packaging problems by being installable without
setup.py or any variation of distutils, lxml can be installed in 0.7
seconds, and a simple installer is just "unzip" inside site-packages.
Wheel installers know about the major version number of the spec
itself, and will refuse to install version 2.0 wheels if they do not
understand them, so the format can be changed down the line.
Let me know what I need to do to get it accepted, if anything needs to
be added or revised, or, preferably, that it is awesome and you want
to use it ASAP.
Dear Distutils-sig members,
I'm having some trouble packaging a python project. I have read several guides but I still seem to do something wrong. Could somebody take a look to see what I am doing wrong?
Packaging details project:http://stackoverflow.com/questions/13307408/python-packaging-data-f…
I think the content of my MANIFEST.in is correct and I am probably doing something wrong in my setup.py file. You can find a list in that link of the configurations of setup.py I have already tried but lead to no success.