[Distutils] formencode as .egg in Debian ??

[Phillip J. Eby]

At 12:54 PM 11/25/2005 +1100, David Arnold wrote:
>So, if a system package, shipped by the upstream developer as an egg, is
>"unpacked" into a directory structure, and its metadata is maintained
>in a .egg-info file somewhere in sys.path, non-system eggs will have all
>they need to operate correctly?

Yes, with a few clarifications.  The internal structure of an egg, let's 
say foobar-1.2-py23.egg, would look something like:

     foobar/
            __init__.py
            baz.py
            # plus .pyc files, etc.

     EGG-INFO/
              PKG-INFO  # distutils metadata like description/version
              requires.txt   # optional and required dependencies
              # plus other metadata files, either setuptools-defined or
              # project specific

If you unpack this as-is, but rename EGG-INFO to foobar.egg-info (today) or 
foobar-1.2.egg-info (when I release 0.6a9 of setuptools), and the whole 
tree above is in a directory on sys.path, this egg is good to go.

I would like to clarify the phrase "shipped as an egg", though.  To me, 
that would mean that the developer is distributing a binary .egg file, and 
I'm assuming that Debian is primarily interested in *source* packages, 
being a Free Software distribution.  (A binary .egg doesn't have to contain 
source code at all; you can specifically build it with the source stripped 
if you desire.)  The plan for setuptools 0.6a9 is to provide an option to 
"setup.py install" that will basically install the layout described above, 
with the correctly named .egg-info directory automatically 
created.  (Normally, the whole tree above is instead nested in an .egg file 
or directory.)

I think I should also clarify that whether the upstream developer sets out 
to package their project as an egg or not, it's possible to create an 
.egg-info directory and PKG-INFO file to identify that distribution, using 
setuptools' "easy_install" program and the source distribution.  So if the 
developer of 'foobar' did not choose to create an egg or use setuptools, 
this doesn't stop a developer who wants to *use* foobar from simply running 
easy_install to create an .egg file for it.  So, this is what I mean when I 
say there's no such thing as a non-egg package for an egg 
developer.  Someone who depends on a package can simply say they depend on 
it, and when they build their package, they'll get eggs for their 
dependencies as a side effect.


>So that's another goal of eggs?  To provide information to a package
>maintainer to assist in determining if it's the user's PYTHONPATH or
>.pth files that are causing a bug?

More specifically, what versions of what packages they're *actually* using, 
as opposed to what they think they installed or have on their 
system.  PYTHONPATH and .pth files can of course be a factor in that, but 
also just people thinking they installed something, or not knowing that a 
bug is fixed in a particular version.  Part of it too is finding out 
whether they're reporting a regression or whether they're just still using 
a version that has a bug that's been fixed.  In the case of the TurboGears 
mailing list, it's often been the case that TurboGears users flush out a 
bug in a dependency, which then gets fixed, but then a new TurboGears user 
maybe reports the same problem, and then it's obvious from their error 
message whether or not they upgraded.

I realize this is stuff you guys probably do all day for system packages, 
but eggs make the support job easier upstream too.


>I can see that this is *nice*; I'd debate "need".  But I'm happy to
>accept that for egg-based stuff, this is a nice feature.

Well, need is relative.  A project like TurboGears "needs" this, because 
otherwise it would be uneconomical to provide the current level of support 
on as many platforms.  So, one project's "nice to have" may be another 
project's lifeblood, depending on available resources.  They've also made 
it easier for the authors of TurboGears' dependencies to assist in support 
as well.  For me, I'm glad that these features have helped to make 
something like TurboGears possible and practical.


>I'm not going to try to assert "Unix values" here.  My observation is
>that historically, Unix has installed things into one of a couple of
>directory hierarchies (/usr, /usr/local, /opt).  Within those
>hierarchies, there has been scope for only one version of any given
>thing.

Um, sure.  Not sure what this has to do with the present discussion.  As a 
practical matter, only *one* version of an egg can be *active* (i.e. 
importable) on sys.path within a given process anyway.  It's also clearly 
not going to be the case on a Debian system that somebody would have 
multiple versions of something living in /usr/lib, although they might do 
it for /usr/local or in a user-private directory.

So, I think maybe I lost the train of thought on this point here.  I was 
under the impression that the consensus of the Debian-Python folks so far 
was that of any egg format, the "single version externally managed" one 
using .egg-info directories was preferred, since it is basically the same 
as your current layout.  (It's also convenient for me to implement, because 
it's basically the same as the format already used by the "setup.py 
develop" command for temporarily adding a project's source checkout to 
sys.path.)


>   Phillip> And we'd like all this to cleanly work with any
>   Phillip> locally-installed non-Debian eggs that might be in the mix,
>   Phillip> since we need to do development, beta testing, etc.
>
>   >>  And non-egg packages as well, right?
>
>   Phillip> There isn't any such thing, from an egg developer's
>   Phillip> perspective.
>
>Really?  So if I use one egg, everything has to be an egg?

I'm not sure I follow you.  If I'm an egg developer, and I want to use 
other Python packages in my project, I add their project names and versions 
to my setup.py, and then I get them installed for free.  If an .egg-info on 
sys.path indicates that the project I want is already on my system, then 
the tools don't go hunting on PyPI and the runtime doesn't gripe about 
missing dependencies.

Note again that the dependencies *don't* need to be distributed as 
eggs.  They can be distributed as source, eggs, .exe installers (Windows 
only), or Subversion URLs, as long as either PyPI has a usable link, or if 
I supply one in my project configuration.  These dependencies' authors 
don't even need to have heard of the concept of eggs, they just need a 
reasonably-standard Python distutils package with a setup.py.

Thus, if I'm developing an egg, yes, all my dependencies have to be eggs, 
but this doesn't imply that I'm pushing eggification upstream, it just 
means that I can install their package as an egg locally, which essentially 
amounts to adding the PKG-INFO file in either an EGG-INFO or .egg-info 
directory.  (The distutils normally generate this PKG-INFO file as part of 
creating a source distribution, so it's not even an egg-specific file format.)

So, projects using setuptools get to take advantage of most any project 
using distutils, and the upstream projects are modified only by adding the 
egg-info, in order to allow the tools and runtime to know when a dependency 
has already been satisfied.

While I don't advocate changing all Debian Python packages to add this 
metadata, I do suggest it's a practical way to deal with certain dependency 
issues.  For example, TurboGears depends on ElementTree, which is not 
packaged as an egg by its author.  (I think that Kid, which is also an 
egg-packaged TurboGears dependency, may depend on ElementTree as 
well.)  Anyway, the quickest way to get all this stuff working without a 
lot of hacks to the dependency metadata would be to install an .egg-info 
marker with the ElementTree package, so that the egg tools and runtime on 
any user's machine will simply know what version of ElementTree is present, 
and be happy.

I know - you can think of other ways to deal with this.  However, most of 
the ways that have been suggested to date fail in the use case where a user 
has been using the Debian package, and Kevin moves to requiring a new 
version of ElementTree or some other dependency, perhaps a new SVN revision 
that hasn't been released -- foobar-1.3.dev-r4262, let's say.  (Setuptools 
users can have their builds tagged with a repository revision 
number.)  This release of foobar isn't going to be in Debian unless you're 
tracking subversion revisions of experimental projects daily - and maybe 
you are, I don't know.  The point is that when the Debian package no longer 
satisfies the dependency, the egg tools move smoothly to downloading and 
installing wherever the user has configured their development environment 
to install it, say their ~/pydev directory.  So now we've segued smoothly 
into "multiple versions" being installed, but the "system version" is still 
intact.

A month later, a stable package is released and I upgrade my Debian 
install.  This is a later version than the development version I have in 
~/pydev, so the egg tools switch back to that as the preferred version 
unless I have a .pth file specifically requesting activation of the ~/pydev 
version as the active version for the other work I'm doing.  (And even then 
it'll still prefer the Debian version if I don't have a ~/pydev version 
that satisfies something's dependency.)

These transitions can only be so seamless if the Debian-installed version 
of foobar includes the egg-info marker so that the tools know what version 
is sitting in /usr/lib, as opposed to the version(s) I have hanging in my 
~/pydev.


>   Phillip> Any distutils package can be made into an egg, because all of
>   Phillip> the metadata needed is supplied by the standard distutils
>   Phillip> setup script.  So, if you have the source, you can make it an
>   Phillip> egg.
>
>What if I don't have the source (or setup.py) ?

What do you have instead?  There really aren't many formats for shipping 
binary Python packages.  The only ones provided by the distutils are 
bdist_dumb, bdist_wininst, and bdist_rpm.  It seems to me that all of these 
formats except bdist_dumb include enough metadata to be able to get the 
project name and version, which is all you need to create enough metadata 
to make a usable egg.  The "easy_install" tool actually supports turning 
bdist_wininst packages into eggs directly.  I'm not sure if you could do it 
with a bdist_dumb.  A bdist_rpm probably has most of what you need just in 
the filename alone, at least if you're doing it manually.  (Distutils-built 
distributions' filenames are too ambiguously formatted for automated 
parsing, alas, even though a human reader can usually tell what they mean.)

Anyway, all you need to make a non-egg package into an egg is its project 
name and version number.  If you have those two things, you can make a 
PKG-INFO file, and that's all you need for today's egg runtime.  For 0.6a9, 
you won't even need to put the data in a file, just the filename.


>Accepting that there will be parallel (I hesitate to say "competing")
>systems, and that keeping them in sync is both hard and necessary seems
>to be the open issue.

I think this may actually be an illusion, perhaps brought about by 
preconceptions based on experiences with other packaging systems.  All we 
need is that:

1. For Debian packages of setuptools-using packages (i.e., projects like 
FormEncode that explicitly set themselves up to be eggs), all the included 
metadata is installed in an .egg-info directory alongside the 
package.  This is nothing more than including all the package's required 
contents, so there's no "parallel" anything going on here.

2. For Debian packages of non-setuptools packages, that are a dependency of 
a setuptools-using package, add an empty .egg-info file named for the 
dependency's project name and version number, as specified in its setup.py 
name/version options.  This is just a simple addition to the packaging, and 
again doesn't seem to create any "parallel" anything.  You do not need to 
go back and repackage every single Debian-Python package unless you feel 
that that's a more efficient way to handle it.  You can simply add the 
.egg-info on an as-needed basis, when you package setuptools-using projects.

Now, there is the separate issue of whether you want to create a separate 
pyegg or python-pypi namespace for these packages, so that you can keep a 
closer match between package names and PyPI project names.  That's for you 
guys to decide, as that's a matter of policy and process.  But I don't see 
anything forcing you to make such a split, so again I don't get the 
"parallel" part.