[Distutils] formencode as .egg in Debian ??

Phillip J. Eby pje at telecommunity.com
Thu Nov 24 21:17:23 CET 2005

At 11:36 AM 11/24/2005 -0800, Robert Kern wrote:
>Phillip J. Eby wrote:
> > Note, by the way, that those two things are the only essentials here, as
> > best I can tell, and I've already stated my willingness to change *how*
> > those two things get accomplished.  For clarity, I will repeat yet again,
> > in yet another way:
> >
> > 1. Egg-based projects need to install their published metadata, in a
> > well-known location relative to the installation location of their 
> code, so
> > that it can be found by searching sys.path, so that it and other projects
> > can locate the metadata for currently-importable projects, *without*
> > needing to first import the project's code.
> >
> > 2. Egg-based projects need to be able to identify whether another Python
> > project package is installed and what version it is, without requiring
> > modification to that other project's code or needing to import it.  (And
> > this is independent of whether the depended-on project was packaged as an
> > egg by its author.)
> >
> > As far as I'm aware, those are the irreducible technical minimum
> > requirements for making an egg-based project work.  *How* these
> > requirements are met is quite flexible, as there are already three working
> > layouts that achieve this.  As I said before, I'm quite willing to
> > implement a fourth.  But nobody has been proposing anything that meets
> > these requirements, because they're too busy trying to prove the
> > requirements don't exist or are somehow not real.
>[Note: I am a happy Debian user, though not a DD. I am also one of the
>developers of a Debian-packaged Python package, and we're considering
>using pkg_resources to implement certain new features. I swear, this is
>like watching two parents fight. Anyways...]
>I think one of the sticking points with the Debian developers has been
>that the .egg-info metadata is being put into /usr/lib/... when
>according to Debian policy and general UNIX lore, such should be placed
>somewhere in /usr/share/.... Would it be possible to treat
>/usr/share/pythonX.Y-egginfo/ as a proxy for
>/usr/lib/pythonX.Y/site-packages/ when searching for .egg-info directories?

The sticking point is the relative relationship to a sys.path directory 
(line 2 of requirement 1, above).  Since a program such as Trac can 
legitimately manipulate sys.path to e.g. add plugin directories, we need to 
be able to do some reasonable transformation to *every* sys.path entry, 
preferably without introducing some kind of distro-specific configuration 
to match special directories and redirect them.  We really want to treat 
every sys.path entry as a valid place to find a project's metadata, since 
each is a valid place for the actual packages to live.

There is of course also a performance issue; currently we need only scan 
each sys.path directory at most once in order to locate all packages and 
their metadata.  (.egg directories don't need to be scanned unless looking 
for a dependency that's not already on sys.path.)  The scan of non-.egg 
directories identifies any .egg-info directories there.  So, for a cost of 
zero or one listdir() for each sys.path entry, we can find the locations of 
all the discoverable metadata.

However, if the issue here is where the data files actually live, couldn't 
Debian just put the egg info wherever it wants, and symlink to it from 
site-packages?  That would already work with the current runtime, although 
it seems to me like it's just making more work for the Debian packagers, 
for no benefit that I comprehend.  (Especially since in the case of 
non-setuptools packages, it might be just an empty file anyway.)

Note that Python packages have always been able to include data files 
inside their package directories, and in fact the *single most popular* 
distutils modification made by published Python packages is to *not* 
install data under /usr/share and instead put those files in the packages' 
directories, where the package can find it without needing to know the 
platform-specific or distro-local place for data.  This is why setuptools 
pioneered an easier way to add package data almost 2 years ago, and that 
way was officially backported into the distutils in Python 2.4.  See:


Anyway, symlinks would work, but it seems to me rather inconsistent to push 
project-level data to /usr/share while leaving module-level data in 
/usr/lib.  And pushing either one of the two to /usr/share seems like 
wasted effort on a non-problem to me.  In neither case is the data going to 
be changed; it can be just as read-only as the code is.

(But just because I don't think it's useful, I'm not going to act like it's 
not a real problem for the Debian team, or insist that putting files in 
/usr/share is undesirable.  If it's desirable to them, it's desirable to 
them.  The only relevant question is how can we both get our desires met.)

More information about the Distutils-SIG mailing list