[Distutils] Adding egg support (Resource API) to existing packages

[Phillip J. Eby]

At 10:26 AM 7/14/2005 +0100, Paul Moore wrote:
>What I'm saying, is that I see a need for *another* transitional
>process - enabling users and/or package authors to make their code
>"resource safe", *without* having to buy into the egg/setuptools
>transition.

It still seems to me that you're missing something.  If I don't care about 
eggs, why would I care about resource safety?  Its only benefit is to allow 
my code to be put in an egg!  :)


>I see the resource API as a major benefit in isolation -
>it's a general tool which can be used with zipimport, or other import
>hooks, and is not egg-specific.

In principle that's true.  In practice, the only other tool I know of where 
it'd actually be *relevant* today would be py2exe -- and you don't get all 
of the features there that you get with eggs.  For example, 
resource_filename() works with eggs and unmanaged packages, but not with 
py2exe.  Indeed, the majority of pkg_resources' code deals with things that 
are only of relevance to eggs; simple resource safety is hardly any code at 
all; something like:

     if hasattr(module,'__loader__'):
         if hasattr(module.__loader__,'get_data'):
             # code to call get_data w/appropriate value
         else:
             # error
     else:
         # use __file__

This doesn't give you resource_filename() and it doesn't let you put C 
extensions in a zipfile, but it's adequate to implement basic zip-safety.


>  There's very little downside (IMHO) to
>using the resource API.

You have to bundle pkg_resources, and then deal with potential version 
conflicts to it.  The bundling isn't a big deal, but the version conflict 
issue is very big because it means you could break someone's whole system 
by installing an old version of pkg_resources that can't deal with some new 
feature used by other packages they've installed.


> > You're also missing the other half of the problem, where pkg_resources
> > needs setuptools so it can be installed in a way that's safe from being
> > overwritten by random packages reinstalling it!  This is also why other
> > packages can't bundle it, without effectively bundling setuptools too.  So,
> > the simplest solution is to Just Drink the Kool-Aid and use setuptools
> > already.  It's not like there aren't numerous other benefits to using it
> > for your packaging, anyway.
>
>Hmm, I've read this a number of times and it still seems to me to be
>saying that all of setuptools' package management goodies break
>horribly if an unmanaged copy of a package is installed

No, it's saying that if an unmanaged copy of *pkg_resources* is installed, 
you could break somebody's whole installation, because somebody who just 
bundles it could be distributing a version that doesn't work with stuff 
somebody already has installed.  This is one of the many reasons that 
"bundling sucks", but it's downright dangerous for something like 
pkg_resources.

Let me repeat: bundling pkg_resources without setuptools is *dangerous*, 
because you could screw up somebody's system just by not keeping your 
pkg_resources version up-to-date.  You could say, "well, I'll have my setup 
check the version", but now you're modifying setup.py a bunch, and by the 
time you make it safe, you could've spent less time just slapping 
ez_setup.py in there and been done with it.


>  - making it a
>very "all or nothing" solution. While I accept that something like
>require("setuptools=0.5a7,=0.5a11,=0.5a12") is probably not going to
>work well if there's an unmanaged copy of setuptools on sys.path,
>can't you make some attempt in simpler cases (maybe introspect for
>__version__, and if not present assume the unmanaged version is
>"wrong", and put a good version at the *front* of sys.path - or
>something). Some form of graceful degradation should be available,
>surely?

So come up with a proposal for *how*, that doesn't involve duplicating 
setuptools' code in pkg_resources or vice versa.  Unless you're suggesting 
I put ez_setup inside of pkg_resources as some kind of trojan?  :)

More seriously, I suppose I could write some kind of bootstrap code in 
pkg_resources that scans sys.path for a setuptools egg and then reloads 
pkg_resources from it if it's not running in an egg.  That would make it 
safe to bundle pkg_resources as long as it was a version with that 
feature.  It would be hideous and stupid, though, because it'd be doubling 
the import time for people who do have setuptools.  I want installing 
setuptools to be a benefit, not a penalty.

My perspective on this is that I don't really care if packages use 
pkg_resources now; if nobody does it until pkg_resources is in the stdlib, 
that's okay by me.  Eggs work fine in unpacked form, they just slow down 
top-level imports a little.


>  What if I install a new package, which I know depends on a
>number of major packages I already have installed - and the new one is
>an egg, but the old ones have been there for ages and are used by a
>number of existing scripts. Surely I don't have to switch everything
>to managed eggs in one "big bang" just to get the one new package
>installed?

This isn't a problem for most eggs; it's strictly a problem with 
overwriting the egg runtime.