[Distutils] formencode as .egg in Debian ??

Phillip J. Eby pje at telecommunity.com
Thu Nov 24 22:03:10 CET 2005

At 09:10 PM 11/24/2005 +0100, Josselin Mouette wrote:
>A sane way, first of all, means a consistent way. Having two sorts of
>Debian python packages is a no-go. Therefore, if we want to switch to a
>new way of distributing packages, there has to be some serious grounds
>for it. Currently, the picture I have of eggs doesn't show any
>advantages over the plain old distutils we are currently using.

What advantage does the internet have?  It lets you be connected to the 
internet.  Eggs are a network, packaging them lets your packages be part of 
that network.  That's the only *possible* benefit to eggs from a system 
packager's point of view, which is why it has never been my policy to go to 
system packagers and ask them to include eggs.  Instead, I leave it to 
their users to ask, because it's really none of my business whether a 
distributor packages eggs.  That's between you and your users.  I only get 
involved if someone asks for help in packaging the eggs, and I help anybody 
who asks.

Which is why I find this discussion so frustrating; somehow or other, 
people keep spinning this back to some kind of advocacy thing.  If you want 
to provide eggs, great.  You don't want to provide them, great.  It really 
makes no difference to me; it's an issue between you and your users.  I'm 
only here to help if you *do* want to provide eggs and have issues with 
*how* to do it in a way that works for you.

[snip items already discussed that have technical solutions already arrived at]

>A sane way means compliance with standards, especially the filesystem
>hierarchy standard. When some people are trying to separate .py
>and .pyc/.pyo files to respect it, you're asking to put them in a single

PEP 304 has been languishing for years, apparently due to lack of 
interest.  In any case, its motivation was never FHS conformance, as far as 
I can tell.

More immediately relevant, I'm 1) not asking anybody to do anything, and 2) 
an archive is only one of three already-valid ways to lay out the 
installation of an egg.

> > 1. Egg-based projects need to install their published metadata, in a
> > well-known location relative to the installation location of their 
> code, so
> > that it can be found by searching sys.path, so that it and other projects
> > can locate the metadata for currently-importable projects, *without*
> > needing to first import the project's code.
> >
> > 2. Egg-based projects need to be able to identify whether another Python
> > project package is installed and what version it is, without requiring
> > modification to that other project's code or needing to import it.  (And
> > this is independent of whether the depended-on project was packaged as an
> > egg by its author.)
> >
> > As far as I'm aware, those are the irreducible technical minimum
> > requirements for making an egg-based project work.  *How* these
> > requirements are met is quite flexible, as there are already three working
> > layouts that achieve this.  As I said before, I'm quite willing to
> > implement a fourth.  But nobody has been proposing anything that meets
> > these requirements, because they're too busy trying to prove the
> > requirements don't exist or are somehow not real.
>Alright, let me try to propose something. How about defining a place in
>the python distribution, say, /usr/share/python2.x/eggs (but of course
>configurable when installing python), where this information should go?
>An installed package would look like a directory structure of .py files
>in /usr/lib/python2.x/site-packages, and a metadata file
>in /usr/share/python2.x/eggs.

See line 2 of item 1 above; we need a "well-known location located relative 
to the installation location of their code".  Python programs have limited 
access to knowledge about platform directory structure, except as expressed 
in sys.path.  Also, they need to be able to support development 
environments, application plugin directories, and other sys.path 
manipulation.  So, ideally the approach needs to be able to be applied 
uniformly to each sys.path entry.

One of the reasons for this is that it allows the runtime to reason about 
the contents of sys.path.  If the metadata is in a particular directory, it 
knows that by adding or removing that directory from sys.path, what 
packages will be added or removed from importability, and it also knows 
which packages may be shadowing other packages.  In the case of namespace 
packages, it knows what subdirectories of sys.path directories need to be 
added to the package's in-memory __path__ in order to merge 
separately-distributed components into a common virtual package.

These goals seem unachievable with an absolute location that's divorced 
from any direct connection to a sys.path entry, although there are a couple 
of workarounds possible.  See below.

>In fact, this approach is quite similar to the .egg-info approach, but
>it would be better to put the files at the right place from the very
>beginning, and, more importantly, to be able to deal with other packages
>not having such .egg-info files.

It doesn't matter where the files actually live; it meets the technical 
requirements just fine if you symlink the metadata into site-packages, and 
would work with the existing runtime.  (Or does the FHS disallow symlinking 
out of site-packages, too?  I'm actually surprisd the FHS would be 
dictating anything about Python package internals anyway, but oh well.)

Alternately, you could just put your proposed /usr/share directory on 
sys.path (just after site-packages) which would solve the issue of trying 
to have some new way of configuring the location, which setuptools would 
then need to be able to read.  Sure, that would increase the length of 
sys.path, but only by one, no matter how many eggs are installed by Debian.

> > Since I'm neither a Debian nor Mozilla developer, I have no way to know
> > what these problems you speak of are, nor any way to tell whether the
> > alleged flaws of the Mozilla packaging relate in any meaningful way to the
> > alleged flaws of eggs.  It's you who's in the position of giving advocacy
> > without providing any real information about the issues.
>This is true. In short, my two main concerns about mozilla-related
>packaging are total absence of library versioning, and the shipping of
>extensions in .xpi files, which are tarballs containing information and
>meta-information about the package. You can surely understand that my
>concerns about eggs approach those of .xpi files.

Yes - but I fear that you may be making an inaccurate comparison based on a 
superficial similarity.  Distribution of binary eggs of anything but 
pure-Python libraries (i.e., ones with no native code) is understood to be 
impractical for anything but well-identified slow-moving platforms like 
Windows.  If you observe e.g. TurboGears' distribution practices, you'll 
see that it relies on Python source packages for "impure" projects on 
anything but OS X and Windows.  As a practical matter, eggs containing 
native code for Linux get built from source on the target machine, unless 
some system packager provides them.

It sounds like your Mozilla complaints are directed at Mozilla-provided 
pre-packaged binaries that include native code, which is not something that 
most Python developers *want* to get involved in.  There will undoubtedly 
be projects (especially commercial ones) that provide plugin binaries for 
various platforms, but even they will prefer to rely on system-packaged 
libraries for anything that's not bleeding edge.  And the plugins 
themselves most likely will live in those apps' plugin directories, not 
system directories.

(I'm of course making several assumptions here about what your concerns are 
and what kind of issues you've encountered, as you still haven't provided 
anything other than a description of similarity in form, and some vague 
hints (e.g. "library versioning") about what might actually be at issue.)

More information about the Distutils-SIG mailing list