[Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns

[Jeff Rush]

I was in a Packaging BoF yesterday and, although not very relevant to the 
packager bootstrap thread, Guido has asked me to post some of the concerns.

The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu 
and such.  Everyone had strong expressions of frustration with the status quo 
and most had tried to resolve their issues but had their patches rejected.  I 
am not taking either side and whether those rejections were justified I cannot 
say, but the general feeling of their concerns intentionally not being 
addressed isn't healthy.  Several had abandoned setuptools, deeming it a 
failed solution and others called for a fork.

To start, I am not a leader of the group nor do I claim I accurately captured 
and expressed all their concerns.  I apologize to those in the BoF for any 
misrepresentations.

1. Many felt the existing dependency resolver was not correct.  They wanted a
    full tree traversal resulting in an intersection of all restrictions,
    instead of a first-acceptable-solution approach taking now, which can
    result in top-level dependencies not being enforced upon lower levels.  The
    latter is faster however.  One solution would be to make the resolver
    pluggable.

2. People want a solution for the handling of documentation.  The distutils
    module has had commented out sections related to this for several years.

3. A more flexible internal handing of the different types of files is needed.
    Currently the code, data, lib, etc. files are aggregated at build time and
    people would like them to be kept separate until install/packaging time.

    They also want greater flexibility in the kinds of files identified for
    packaging.  There is currently a single plugin entrypoint for file_finding,
    so people have resorted to abusing the setuptools function find_packages()
    again and again with different include/exclude args.  A solution is to
    expand the set of entrypoints into finer grained categories.  They also
    want a way to expand the set of categories rather than a fixed set, which
    can be easily done with entrypoint groups and names.

    People also want a greater variety of file_finders to be included with
    setuptools.  Instead of just CVS and SVN, they want it to comprehend
    Mercurial, Bazaar, Git and so forth.

4. They want an uninstall setuptools command.  Adding one to remove a specific
    egg isn't difficult but correctly removing those dependencies that came in
    with that egg, without breaking later installs can be tricky.

    This is complicated because there isn't a single global package namespace
    to manage, when you factor in virtualenv and buildout sandboxes and
    per-user package areas.  This differs from how RPMs and .debs are viewed.

5. There was concern over the .pth mechanism used by setuptools re activation.
    First, there is a (perceived) performance issue with increasingly adding
    every ZIP file explicitly onto sys.path.  This may or may not be a red
    herring.

    The other is the use of a single .pth file to control the list of activated
    packages.  Those who produce distributions would prefer a magic directory
    into which links to distributions could be dropped, similar to the current
    best practices for Linux, with /etc/conf.d/, /etc/profile.d/,
    /etc/xinetd.d/ and so forth.

6. There is a need for more extensibility hooks.  People want places to plug
    in special handling.  For example:

    a) setuptools has a --record option to capture the list of files installed
       for use by subsequent packaging tools.  Some want that list to be
       available to a setuptools plugin.

    b) some want hooks for post-build/post-install actions, instead of the
       current approach of writing a custom build class that handles it all.

7. Many wanted to ability to install files anywhere in the install tree and
    not just under the Python package.  Under distutils this was possible but
    it was removed in setuptools for security reasons.  Custom code can still
    be written to do this explicitly but this is not popular.  Neither
    setuptools nor distutils has the ability to rename files at install time.

A fair question is whether it is the job of setuptools (or any Python 
packaging solution) to cover all these bases.  The risk of not doing the job 
is that some of those in attendance were rolling their own solutions which do 
not play well with packages installed using other means, not seeing them. 
Distutils has intentionally tried to -not- be a general replacement packaging 
solution, with its support of the "bdist" command for various 
platform-specific distribution formats.  We should continue not trying to 
replace platform-specific packaging technologies but perhaps improve our 
control of their creation.

As mentioned, some of these concerns can be resolved by adding 
customization-pressure-release entrypoints to setuptools, and some can be 
handled with much better documentation of use cases and what to do.  And some 
of it is confusion over packaging libraries versus applications, where 
setuptools focuses on the former and zc.buildout focuses on the latter.  But 
buildout is very young, maintains isolation from the system Python and was not 
known to many of the packaging BoF attendees.

Some of this may seem down on eggs, but I think they are really cool and would 
like to see them adopted as the standard for packaging Python software.  There 
are rough spots on setuptools and buildout that would benefit from opening up 
the process and bringing in more developers, and communicating what they are 
and more importantly, what they are not.  I believe the lack of a coherent 
packaging and deployment story for Python is hurting its uptake in many 
sectors and would like to work with others to strengthen this area of Python.

-Jeff