[Distutils] A new, experimental packaging tool: distil
Vinay Sajip
vinay_sajip at yahoo.co.uk
Tue Mar 26 09:54:20 CET 2013
I've created a new tool called distil which I'm using to experiment with
packaging functionality.
Overview
--------
It's based on distlib and has IMO some interesting features. With it, one can:
* Install projects from PyPI and wheels (see PEP 427). Distil does not invoke
setup.py, so projects that do significant computation in setup.py may not be
installable by distil. However, a large number of projects on PyPI *can* be
installed, and dependencies are detected, downloaded and installed. For those
distributions that absolutely *have* to run setup.py, distil can create
wheels using pip as a helper, and then install from those wheels.
* Optionally upgrade installed distributions, whether installed by distil or
installed by pip.
* Uninstall distributions installed by distil or pip.
* Build source distributions in .tar.gz, .tar.bz2, and .zip formats.
* Build binary distributions in wheel format. These can be pure-Python, or have
C libraries and extensions. Support for Cython and Fortran (using f2py) is
possible, though currently distil cannot install Cython or Numpy directly
because of how they use setup.py.
* Run tests on built distributions.
* Register projects on PyPI.
* Upload distributions to PyPI.
* Upload documentation to http://pythonhosted.org/.
* Display dependencies of a distribution - either as a list of what would be
downloaded (and a suggested download order), or in Graphviz format suitable
for conversion to an image.
Getting started is simple (documentation is at [2]):
* Very simple deployment - just copy distil.py[1] to a location on your path,
optionally naming it to distil on POSIX platforms. There's no need to install
distlib - it's all included.
* Uses either a system Python or one in a virtual environment, but by default
installs to the user site rather than system Python library locations.
* Offers tab-completion and abbreviation of commands and parameters on
Bash-compatible shells.
Logically, packaging activities can be divided into a number of categories or
roles:
* Archiver - builds source distributions from a source tree
* Builder - builds binary distributions from source
* Installer - installs source or binary distributions
This version of distil incorporates (for convenience) all of the above roles.
There is a school of thought which says that that these roles should be
fulfilled by separate programs, and that's fine for production quality tools -
it's just more convenient for now to have everything in one package for an
experimental tool like distil.
Actual Improvements
-------------------
Despite the fact that distil is in an alpha stage of development and has
received no real-world exposure like the existing go-to packaging tools, it
does offer some improvements over them:
* Dependency resolution can be performed without downloading any distributions.
Unlike e.g. pip, you are told which additional dependencies will be
downloaded and installed, before any download occurs.
* Better information is stored for uninstallation. This allows better feedback
to be given to users during uninstallation.
* Dependency checking is done during uninstallation. Say you've installed a
distribution A, which pulled in dependencies B and C. If you request an
uninstallation of B (or C), distil will complain that you can't do this
because A needs it. When you uninstall A, you are offered the option to
uninstall B and C as well (assuming you didn't install something else that
depends on B or C, after installing A).
* By default, installation is to the user site and not to the system Python, so
you shouldn't need to invoke sudo to install distributions for personal use
which are not for specific projects/virtual environments.
* There's no need to "install" distil - the exact same script will run with any
system Python or any venv (subject to Python version constraints of 2.6, 2.7,
3.2 or greater).
Bootstrapping pip
-----------------
I've used distil to bootstrap pip, then used that pip to install other stuff.
I created a fresh PEP 405 venv with nothing in it, used distil to install a
wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used
distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy
(using pip) from PyPI. See [4] for the transcript.
I would welcome any feedback you could give regarding distil/distlib. There is
of course a lot more testing to do, but I consider these initial findings to be
promising, and worth sharing. If you find any problems, you can raise issues
at [5].
Regards,
Vinay Sajip
[1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py
[2] https://pythonhosted.org/distil/
[3] https://bitbucket.org/vinay.sajip/distribute3/downloads/
[4] https://gist.github.com/vsajip/5243936
[5] https://bitbucket.org/vinay.sajip/distlib/issues/new
More information about the Distutils-SIG
mailing list