[Python-Dev] My summit notes (packaging)

[Tarek Ziadé]

Here's a wrapup of my notes for the Distutils part (with Jim's help).
The PyPI part will come later.

I have presented the various problems developers have with packaging
today, wether they are using Distutils, Setuptools or any other third
party tools out there.

Here's the short-list:

- Distutils code is still undertested, with over 100 issues to
  take care of in the tracker.

- Setuptools is the de-facto extension of Distutils. It brings
  a lot of good features, but some features are controversial
  amongst the community.

- We have now two tools that install a package and look for its
  dependencies at PyPI and install them: pip and easy_install.
  The latter is included in setuptools. They both aim at the
  same goal besides a few differences, and they both rely
  on a new metadata introduced by setuptools, wich is.
  "install_requires". This new metadata extends the metadata.
  described in PEP 314 but is slightly different from.
  what is descibred in the Draft PEP 345  ("Requires").
..
  PEP 345 introduces "Requires" and "Provides" wich are
  are implemented in Distutils and PyP, but are not
  widely used. 40 out of +4000 if I remember correctly. Martin will
  correct me here if I am wrong.

- pip, easy_install and distutils use different formats for installed
  metadata. For instance, Distutils installs a PKG-INFO file whereas
  setuptools creates a directory with the PKG-INFO file in it.

- There's no way to uninstall a package. The only way to do it
  is to remove manually the directory located in your site-packages
  and remove the line located in the .pth file that might point to it.

- There's no standard way to use resource files, for instance
  to be able to read or write a resource file that might be installed
  in a specific place on the target system by the system packager.
  Most of the time people are just working with files located.
  in the package directory.

Ian Bicking and Jim Fulton gave short presentations of virtualenv
and zc.buildout. These tools will let you build an isolated environment,
where you can install some packages of certain versions, that play well
together to run an application, without having to put anything into your
site-packages. Many agreed that adding packages into the system's
site-packages for an application to run is not a good idea.

Also, many agreed that these third party tools are doing their.
job and that we shouldn't try to include some of the features they.
provide in any way in the standard library. Such as supporting multiple.
versions for the same package for example.

Instead, the idea is to try to reduce the scope of Distutils and to
provide some better API and conventions for people to access to the package
metadata (wether it's installed or not) because people have different
needs and purposes using these data.

*What's next ?*

Guido provided a high level plan for the work to be done in packaging.
Let me quote him:

    - standardize more metadata, especially including dependencies,.
    and APIs for accessing the metadata and dependencies.

    - there should be an API to get metadata for a package without
    actually executing any of the package's installation script.

    - this will go into the stdlib for Python 2.7 and 3.1

    - it will be provided as separately downloadable backports for prior
    versions (maybe using a bootstrap not unlike ez_install)

    - keep distutils, but start deprecating certain higher-level
    functionality in it (e.g. bdist_rpm)

    - don't try to provide higher-level functionality in the stdlib, but
    instead let third party tools built on top of these core APIs compete

So we are currently working this week to flesh out this plan, and we
will send a mail to Distutils-SIG with the ouput asap.

Cheers
Tarek

-- 
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/