[Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[Paul Moore]

2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> P.J. Eby wrote:
>> At 08:43 PM 7/5/2009 +0200, Tarek Ziadé wrote:
>>> But if it's based on PEP 302 protocols and if the pkgutil code works
>>> with the sys.meta_path hook,
>>> setuptools could then provide its loader, based on its EggFormats and
>>> act as a provider without being broken.
>>
>> You misunderstand me.  The whole point of putting .egg-info in distutils
>> in the first place was to enable setuptools to detect the presence of
>> disutils-installed packages.  That's what's broken by changing the name.

This is getting confusing. Is Phillip saying that setuptools will cope
with the file changing to a directory without modification, but it
can't handle a change in the name?

My site-packages has a confusing mix of egginfo directories and files.
Note that I NEVER use setuptools other than where an existing
package's setup.py requires it. In that case, I still only do python
setup.py bdist_wininst and install the generated installer.

So is PEP 376 going to be able to cope with the stuff I have installed
at the moment? If not, what's the point???

Example:

I have the "coverage" package installed. It does not use setuptools at
runtime - I could happily install it from a bdist_wininst on a machine
without setuptools installed, and it would run fine. (In fact, that's
what I did!) It has created a coverage-3.0-py2.6.egg-info *directory*
in my site-packages, with no RECORD file - presumably, this is in a
setuptools format.

If setuptools is not going to change to conform to PEP 376, then any
tools built using PEP 376 will fail to recognise my coverage install.
Which, as far as I am concerned, is a completely standard
bdist_wininst installer.

In fact, the above strongly suggests to me that PEP 376 must NOT
follow setuptools conventions - otherwise, it will end up clashing
with the files setuptools is putting on my system.

> How much information does setuptools actually need in order to tell that
> a distribution is already present? Presumably the existing .egginfo
> files generated by distutils are sufficient for that task?

It appears so, but setuptools doesn't use (or create!!!) those files
in its own installer formats.

> If so, I'd still prefer to keep the new metadata safely out of reach of
> the legacy package management systems that don't understand it, while
> having distutils retain the ability to generate a legacy ".egginfo" file
> to warn off the existing package management tools from installing the
> same distribution again.

So for a minimal case of a single .py file packaged up as a
distribution, we now have the .py file, a legacy .egginfo file, a new
.pydist directory containing RECORD and PKG-INFO files?

That's getting silly.

>
> So instead of ".egginfo becomes a directory" as in the current version
> of PEP 376, the existing distutils .egginfo file generation would
> instead remain unchanged with a ".pydist" directory being added
> alongside to hold all the new goodies. That way, the legacy formats
> (EGG-INFO, .egg-info file, .egg-info directory) continue to play as well
> together as they do now, while the new distutils system will understand
> .pydist directories natively and allow the injection of hook
> implementations by the third party package management tools in order to
> understand the old formats.
>
> That seems like it will give us the best of both worlds - a clean base
> to work from in the future, but without breaking the existing
> interoperability mechanisms (at least in the short term - maybe by the
> time 3.3 rolls around we will be able to start thinking about
> deprecating the legacy .egginfo formats, but even considering that
> question is a long way off).

And yet, given that PEP 376 is explicitly being designed with a goal
being to act as the common standard that *all* package management
formats can use, is it not the whole point that once it's defined and
we have achieved consensus, existing package managers will switch to
using it rather than retaining a range of custom formats?

I honestly fail to see why we are catering to this "setuptools cannot
change" view. Usually, it's the standard library that cannot change,
because of the longer release cycle of the core. If a standard is
agreed, and setuptools won't conform to it, maybe it's time for
someone to fork setuptools (unilaterally, if Phillip can't agree to an
amicable transfer). Otherwise setuptools will remain a roadblock for
any development of distutils.

Frustrated-ly y'rs
Paul.