[Distutils] formencode as .egg in Debian ??

[Phillip J. Eby]

At 10:29 AM 11/25/2005 +0000, Donovan Baarda wrote:
>On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote:
>[... long informative explanation of egg...]
>
>So it sounds like egg duplicates much of the functionality of the Debian
>Package manager, but also extends it slightly in python-specific ways.
>
>I can under stand the Debian packagers asking themselves; do we just
>wrap an egg in a Debian package and accept the duplicated functionality,

It's not duplicated functionality in normal operation.  In the case where a 
user decides to use easy_install to install something instead of using a 
Debian package (e.g. because it's not available as one yet), it simply 
allows them to *also* use Debian packages to satisfy dependencies.  If the 
user wants to pull an up-to-the-minute TurboGears, which uses a mix of 
other bleeding-edge stuff and stable stuff, it'd be nice for them to be 
able to use their system packages to satisfy those more stable dependencies.

In the case where the user is *not* using easy_install, then all 
dependencies will be met by system packages, and the only "duplication" is 
that the paranoid setuptools integrity-check wrappers will want to be 
assured that yes, everything is still there.


>The biggest risk of just wrapping the duplicated functionality is; are
>there incompatibilities or policy violations in the egg way of doing
>things?
>
>In particular, will an egg wrapped inside a Debian package magically
>install other bits of software not from Debian packages? Will it install
>them in the correct places?

Not unless the application was specifically written to do this.  For 
example, I believe Trac has a web interface that allows you to upload 
plugin eggs to the Trac application, which will go in that specific Trac 
installation's plugin directory.

However, there is *no* automatic download-and-install for eggs *except* by 
running setup.py commands or by using easy_install.  Any 
download-and-install features provided by a particular application or 
library are that author's responsibility.  And I deliberately didn't make 
it *too* easy to implement that functionality, because I believe people 
should think carefully about the ramifications of download-and-install 
without the user explicitly running setup.py or easy_install.  In 
particular, most applications needing such functionality should be 
downloading into an application-specific plugin directory (which they then 
add to their sys.path at runtime), and definitely not to system locations.


>If TurboGears depends on an "egg'ed" ElementTree, what happens if a
>system has ElementTree installed as a non-egg package? Does installing
>TurboGears as an egg inside a Debian package require also installing
>another ElementTree as an egg inside a Debian package? Or worse, will it
>automatically download and install an egg'ed ElementTree not from a
>Debian package? Will you end up with two ElementTree's installed, one
>egg'ed and one not?

If you are running easy_install or setup.py to install TurboGears, and an 
egged ElementTree is not available, one is going to get installed for you 
-- because in that scenario you are asking for installation and are 
presumably doing it to /usr/local or to ~/somewhere.  If you are merely 
*using* TurboGears and an egged version of a dependency is not available, 
you will get a DistributionNotFound exception at startup or during runtime, 
depending on how the author defined the dependency and whether it was an 
optional one.  (Optional dependencies allow the package author to define 
optional features for which additional dependencies are required.  Usually 
this is in the form of self-registered entry points so that the 
dependencies are checked just before trying to import and invoke the code 
that implements the feature.  The application can then trap any 
DistributionNotFound or VersionConflict errors and continue without the 
feature.)

For more info about optional features, see:

http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies


>Debian has versioned dependencies. If a package depends on a particular
>version of another package that doesn't exist yet, you can't install it;
>full stop. If you want to package something that depends on a SVN
>revision of some dependency, then you need to package that SVN version.

Right; the point here is that if somebody is going *outside* of Debian 
packages and explicitly using setup.py or easy_install to install it, then 
it would be nice for them to *still* be able to use system packages to meet 
some of the dependencies, rather than having to re-download, re-build, and 
re-install new copies of the stuff they already have.  This isn't about 
changing Debian to support bleeding edge things, it's about allowing people 
doing bleeding edge things to not need to duplicate what Debian already 
provides.


>I think they a both important. If I was using eggs, I'm not sure I'd
>want them to be packaged as anything other than an egg until I was ready
>to release, and then I'd want it **not** packaged as an egg, as a "proof
>of it's release readiness".

Well, that would only work if you weren't developing something using entry 
points to tie an extensible application or framework together.  And that 
didn't use egg metadata for anything else.  But if all you used it for was 
dependencies, then that might make some sense...  except that it would also 
mean that nobody else could *use* your stable version to do *their* 
development, which is one of the goals here.


>It's perhaps unfortunate that egg's bundle package management with
>generic meta-data management.

It's not really bundled - integrity checking and inter-package linking do 
not a package manager make.  EasyInstall is the (rather primitive) package 
manager, and it only comes into play if you are manually installing 
things.  For the normal apt-get scenario, EasyInstall isn't involved, with 
the possible exception of applications like Trac installing plugins into 
their private plugins directories.


>  I know that package management uses
>meta-data, but if it was implemented as one wrapped around the other,
>you could "peel the package management layer off", leaving the metadata
>management there, and you could produce clean deb's using only Debian
>package management, but with the egg meta-data management still in
>place.

There's nothing to peel off, unless you believe that it's wrong to have 
runtime integrity checking.  That's all the egg runtime does, unless an 
application developer codes up something special for their application by 
wrapping easy_install.  When an application says to the runtime, "I need 
foobar", the runtime just checks if foobar is there.  This is useful for 
optional feature dependencies, for example, and safe integrity checking is 
definitely needed for apps with plugins (which is what eggs were originally 
invented for: extensible Python applications like Zope, Trac, Chandler, etc.).

Vincenzo's idea about having easy_install be able to wrap apt-get is 
interesting, in that it would allow apps that want plugin installation to 
satisfy dependencies via system packages that *aren't* already installed, 
but in practice I think such applications are going to be restricted to 
local or specific-user installation by the very nature of their intended 
uses, so being able to have easy_install get automatically hooked to 
apt-get doesn't seem necessary to me.  In the Debian world, easy_install 
should probably only be used for user-private and/or system-local 
non-Debian stuff, including apps' private plugins.  (And if you did want 
easy_install to be able to use apt-get to satisfy dependencies, it probably 
means you should just be using something like easy_deb instead, to build 
debs of the stuff you want.)