[Distutils] "Python Package Management Sucks"

Matthias Klose doko at cs.tu-berlin.de
Wed Sep 24 18:48:33 CEST 2008


Chris Withers writes:
> (I'll be CC'ing the distutils sig in to these replies as this discussion 
> probably belongs there...)
> 
> Nicolas Chauvat wrote:
> >> The slides for my two talks can be found here:
> >>
> >> http://www.simplistix.co.uk/presentations
> > 
> > "Python Package Management Sucks"
> > 
> > Install debian and get back to productive tasks.
> 
> This is an almost troll-like answer.
> See page 35 of the presentation.

I disagree. You could think of "Packages are Pythons Plugins" (taken
from page 35) as a troll-like statement as well.

Apparently this user is happy about Debian and Ubuntu how they
distribute python modules and it's worth evaluating why.

> If you're too lazy, here's a re-hash:
> 
> - there are lots of different operating systems
> 
> - even with Linux, there are many different package management
>   systems

and a subset of all these operating systems (linux or other) do have
the need to distribute python and a set of python modules and
extensions. they cannot rely on a plugin system outside the (os)
distribution.

> - all the package management systems behave differently and expect 
> packages to be set up differently for them

correct, but again they share common requirements.

> - expecting package developers to shoulder this burden is unfair and

> results in various bitrotting repackaged versions of python packages

> rather than just one up-to-date version maintained by the entity 
> originating the python package

some people prefer to name this "stable releases" instead of
"bitrot".  usually you can be assured that the set of python modules
delivered with an (os) distribution is tested and works. I doubt you
do achieve the same with an untested set of up-to-date versions.

Speaking of extensions "maintained by the entity originating the
python package": this much too often is a way of bitrot. is the
shipped library up to date? does it have security fixes? how many
duplicates are shipped in different extensions? does the library need
to be shipped at all (because some os does ship it)?

> - Adobe Photoshop Plugins, Firefox Add-ons, etc do not delegate their 
> management to an OS package manager.

this is known trouble for os distributors, and your statement is
generally wrong. firefox plugins are packaged in distributions and the
plugin system is able to cope with packaged plugins.

> Packages are Python's "plugins" and so should get the same type of
> consistent, cross-platform package management targetted at the
> application in question, which is Python in this case.

No, as explained above. Considering an extension interfacing a library
shipped with the os, you do want to use this library, not add another
copy. and you do want to interface exactly this version. An upstream
extension maintainer cannot provide this unless he builds this
extension for every (os) distribution and maintains it during the os'
lifecycle.

Your view is very common (and legitimate) in the Python world
(distributing an batteries-all-included application), but not
everybody does need this (like os distributors, using the
reliable-power-plug-where-available), and in some areas it starts to
hurt.

Your paper gives a nice overview of the shortcomings of each of the
build/distribution systems. From an (os) distributors point of view I
would add som things (also posted in [1]).

 - os distributors usually try to minimize the versions they include,
   trying to just ship one version.  This single version is installed
   with --single-version-externally-managed, so that it can be
   imported without any pkg_resouces magic and fiddling with pth
   files. Unfortunately then it is not possible to use/import another
   version using pkg_resources. As discussed at PyCon such a setup is
   very common for os distributors. None of the tools does support
   this.

 - setuptools has the narrow minded view of a python package being
   contained in a single directory, which doesn't fit well when you
   do have common locations for include or doc files. A linux
   distribution is supposed to follow the File Hierarchy Standard
   (FHS). None of the tools does support this.

 - setuptools (sharing with distutils) doesn't allow splitting of
   a python package into several distribution binary packages, e.g.
   have a separate package with large docs, allowing installation
   on a server without having to install X, and so on. But maybe this
   is something an (os) distributor only cares about.

You will see that most packages included in (os) distributions still
use plain distutils, even if the setup.py does support a setuptools
based install. That's simply because distutils doesn't get into the
way packaging the python module with rpm or dpkg. E.g. namespace
packages are a consequence how setuptools distributes and installs
things. Why force this on everybody?

A big win could be a modularized setuptools where you are able to only
use the things you do want to use, e.g.

 - version specifications (not just the heuristics shipped with
   setuptools).

 - specification of dependencies.

 - resource management

 - a module system independent from any distribution specific stuff.

 - any other distribution specific stuff.

The "Wouldn't it be nice if?" pacman (page 55) sounds like a nice
idea, if it could just handle multiple repositories and one of them
being the archive of the os distributor (iirc the java jsr277 proposed
the use of multiple repositories, even in different formats).

  Matthias

[1] http://mail.python.org/pipermail/distutils-sig/2008-September/010045.html


More information about the Distutils-SIG mailing list