[Distutils] A new, experimental packaging tool: distil

[Vinay Sajip]

I've created a new tool called distil which I'm using to experiment with
packaging functionality.

Overview
--------

It's based on distlib and has IMO some interesting features. With it, one can:

* Install projects from PyPI and wheels (see PEP 427). Distil does not invoke
  setup.py, so projects that do significant computation in setup.py may not be
  installable by distil. However, a large number of projects on PyPI *can* be
  installed, and dependencies are detected, downloaded and installed. For those
  distributions that absolutely *have* to run setup.py, distil can create
  wheels using pip as a helper, and then install from those wheels.
* Optionally upgrade installed distributions, whether installed by distil or
  installed by pip.
* Uninstall distributions installed by distil or pip.
* Build source distributions in .tar.gz, .tar.bz2, and .zip formats.
* Build binary distributions in wheel format. These can be pure-Python, or have
  C libraries and extensions. Support for Cython and Fortran (using f2py) is
  possible, though currently distil cannot install Cython or Numpy directly
  because of how they use setup.py.
* Run tests on built distributions.
* Register projects on PyPI.
* Upload distributions to PyPI.
* Upload documentation to http://pythonhosted.org/.
* Display dependencies of a distribution - either as a list of what would be
  downloaded (and a suggested download order), or in Graphviz format suitable
  for conversion to an image.
  
Getting started is simple (documentation is at [2]):

* Very simple deployment - just copy distil.py[1] to a location on your path,
  optionally naming it to distil on POSIX platforms. There's no need to install
  distlib - it's all included.
* Uses either a system Python or one in a virtual environment, but by default
  installs to the user site rather than system Python library locations.
* Offers tab-completion and abbreviation of commands and parameters on
  Bash-compatible shells.

Logically, packaging activities can be divided into a number of categories or
roles:

* Archiver - builds source distributions from a source tree
* Builder - builds binary distributions from source
* Installer - installs source or binary distributions

This version of distil incorporates (for convenience) all of the above roles.
There is a school of thought which says that that these roles should be
fulfilled by separate programs, and that's fine for production quality tools -
it's just more convenient for now to have everything in one package for an
experimental tool like distil.

Actual Improvements
-------------------

Despite the fact that distil is in an alpha stage of development and has
received no real-world exposure like the existing go-to packaging tools, it
does offer some improvements over them:

* Dependency resolution can be performed without downloading any distributions.
  Unlike e.g. pip, you are told which additional dependencies will be
  downloaded and installed, before any download occurs.
* Better information is stored for uninstallation. This allows better feedback
  to be given to users during uninstallation.
* Dependency checking is done during uninstallation. Say you've installed a
  distribution A, which pulled in dependencies B and C. If you request an
  uninstallation of B (or C), distil will complain that you can't do this
  because A needs it. When you uninstall A, you are offered the option to
  uninstall B and C as well (assuming you didn't install something else that
  depends on B or C, after installing A).
* By default, installation is to the user site and not to the system Python, so
  you shouldn't need to invoke sudo to install distributions for personal use
  which are not for specific projects/virtual environments.
* There's no need to "install" distil - the exact same script will run with any
  system Python or any venv (subject to Python version constraints of 2.6, 2.7,
  3.2 or greater).

Bootstrapping pip
-----------------

I've used distil to bootstrap pip, then used that pip to install other stuff.
I created a fresh PEP 405 venv with nothing in it, used distil to install a
wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used
distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy
(using pip) from PyPI. See [4] for the transcript.

I would welcome any feedback you could give regarding distil/distlib. There is
of course a lot more testing to do, but I consider these initial findings to be
promising, and worth sharing. If you find any problems, you can raise issues
at [5].

Regards,

Vinay Sajip

[1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py
[2] https://pythonhosted.org/distil/
[3] https://bitbucket.org/vinay.sajip/distribute3/downloads/
[4] https://gist.github.com/vsajip/5243936
[5] https://bitbucket.org/vinay.sajip/distlib/issues/new