[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

[Paul Moore]

2009/10/8 Tarek Ziadé <ziade.tarek at gmail.com>:
>> The egg format
>>
>> One thing missing from your roadmap (unless I missed it) is the fate
>> of the egg (zipfile) format. If it's to remain a valid option
>> (bdist_egg and the like) then I'd assume that Distribute needs to be
>> the place to develop it.
>> One thing it will need is an equivalent to
>> PEP 376 introspection and uninstall features. And *that* is where I
>> believe a standard protocol belongs - developed based on the real use
>> requirements for egg files, with a suitable file-based layer which can
>> be added into distutils (somewhat like importlib does for PEP 302)
>> once the protocol is agreed and standardised.
>
> Distribute is planning to continue supporting the egg format, but will
> probably drop the zipped version.
> This is still under discussions though so don't take this as the final word.
>
> In any case, distribute.resources will probably add the missing bits
> on the top of a PEP 376 -compliant Distutils
> to provide support for any file querying Distutils will not provide
> (for installation and uninstallation)

[Note: For the purposes of below I use "egg format" to only refer to
zipped eggs. I see no value AT ALL in having an unzipped "egg format"
which is almost the same as, but subtly different from, the
distutils-supported standard filesystem format. If the distutils
standard doesn't provide for all the requirements, fix that, don't add
another variation!]

I'm not sure I follow this. I think it's important to provide for
non-filesystem based packages (that's the whole point of things like
PEP 302 and importlib!). The fact that core distutils doesn't support
installing, querying or removing such packages is fine. They can be
supplied by 3rd party systems like Distribute.

However, it is *not* fine (in my view) for each independent format to
have to implement its own set of interfaces. PEP 302 is precisely the
solution for this in terms of import - every importer has a
standardised set of protocols to follow, and if it does so, then it
will (should :-)) work seamlessly with any Python code. PEP 302 is not
perfect, in part because it aims to be minimal rather than
comprehensive, hence the fact that not all packages are "zip safe".
But that's a failing in PEP 302 rather than in the concept.

Multiple package formats should be supported in the same way - with a
set of well-defined (via a PEP) interfaces that all package formats
must provide, which enable core and user code to be written in a
format-independent way. Once such a PEP is written, distutils' PEP 376
support should be rewritten to use those interfaces, and to implement
the necessary filesystem-format support layer. (In other words,
whereas I support reducing PEP 376's scope to filesystem only for now,
I still strongly advocate that the longer-term solution should be
based on a user extensible format-independent approach).

In this context, eggs are "merely" the first (and most important)
example of a format extension, and so should drive the development of
a standard.

To summarise:

I believe that we need a statement of direction on the (zipped) egg
format. I see a number of options:

1. Retain it as-is in Distribute, but deprecated and no further development.
2. Deprecate egg support in Distribute, and ultimately drop it.
3. Develop into a reference example of an extensible architecture for
adding to PEP 376 support.
4. Extend PEP 376 support to eggs by means of egg-specific approaches
hooking into distutils.

For the record, I dislike (1), I see (2) as a lost opportunity, I see
(3) as the right way to go, but appreciate that it's the most work,
and I strongly oppose (4) (not least because it effectively makes a
latrge part of PEP 302 and importlib wasted effort).

It looks like you're proposing (2), but aren't entirely clear because
you're still allowing for non-zipped "egg" formats (which I see no
value in, as I noted above).

I'm willing to help work on option (3). Once it's decoupled from
questions around OS package managers (which are more the province of
PEP 376) then I'd hope that my lack of knowledge about package
managers is less of a problem.

Paul.