[Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[M.-A. Lemburg]

Paul Moore wrote:
> 2009/7/7 M.-A. Lemburg <mal at egenix.com>:
>> The PEP should take the chance to define not only the
>> directory, but also the complete set of files in there and
>> their format.
>>
>> A typical setuptools dir looks like this:
>>
>> dependency_links.txt
>> namespace_packages.txt
>> not-zip-safe
>> PKG-INFO
>> requires.txt
>> SOURCES.txt
>> top_level.txt
>>
>> Ie. a complete mess of upper-case/lower-case names, names with
>> underscores or hyphens as word separator, files with extensions,
>> files without them.
>>
>> This needs some serious cleanup.
>>
>> A new dir name will allow to do this, so +1 on a new dir name.
> 
> I agree the current situation is a mess. I believe that everything you
> mention is related to setuptools - so essentially, we have the
> situation:
> 
> - the existing setuptools format is a mess (at least in the opinion of
> MAL and me :-))
> - PEP 376 is an opportunity to consider cleanup
> - there are some strong supporters of keeping things setuptools-compatible
> - nobody has come forward with any other real-world use case than setuptools
> - consequently, those of us arguing for cleanup are talking theoretically :-(
> - Do you (MAL) have any real use for this data? Any non-setuptools use
> case, no matter how small, would really help here.

No, we currently don't have a package installer that would
implement what setuptools does on the user side, but without
all the black magic stuff.

What we do have is a somewhat different approach to packaging
binary packages: we call them pre-built archives.

The idea is that the developer runs a complete build command on
the target platform, the bdist_prebuilt command then packages up
everything (including the current state of distutils) and the
user then runs the usual

    python setup.py install

on her machine to complete the setup.

The major advantage here is that the user actually has full
control over where things are installed and how.

So whatever distutils implements as part of the install command,
this format will also support. We also provide an uninstall command,
that basically runs an installation in reverse.

Something else we've done recently is write a bdist_egg command
that creates .egg files without relying on setuptools. It's
fairly easy to do and if there's interest, I can add that to
core distutils. Changing the layout of the EGG-INFO dir embedded
in those .egg files is pretty easy to do, so I don't really see
much of a problem with changing it or renaming the dir.

> I suspect practicality will beat purity here. Which would be fine, if
> the "practical" cases weren't just setuptools.
> 
> I understood that there was a lot of interest in a "distutils cleanup"
> from the packaging community. And yet so far in this discussion, the
> main participants have seemed to be (apologies to anyone if I've
> misunderstood their position):
> 
> - Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed
> - Me, advocating Windows issues and PEP 302 integration (to cover eggs
> and general flexibility)
> - Phillip (and occasional others), representing setuptools POV
> - Interested python-dev participants
> 
> Nobody representing any other 3rd party packaging solution than setuptools.

Well I've added another one above, the pre-built approach :-)

Writing an uninstall command really isn't all that hard, provided
you stick with the standard distutils "python setup.py install"
dance. The whole packaging approach only complicates things, IMHO.

> Much as I'd like to, I can't really argue the case for breaking
> setuptools compatibility without practical reasons. And if we're not
> going to do that, PEP 376 reduces to a further stage of the
> incremental move of setuptools into distutils core (by removing the
> partial solution of a .egg-info file, and replacing it with a full
> setuptools .egg-info directory, plus a few introspection APIs as a
> sweetener).

Uhm, we are talking about a Python standard here and since there
is no prior implementation in Python stdlib, we're not breaking
anything and are free to design whatever we like.

Of course, it does make a lot of sense to build upon things that
have been used in the past, so looking at setuptools is certainly
a useful strategy. However, let's learn from their mistakes
and take the change to clean up things a bit.

There are good reasons to do so:

 * the naming scheme should be concise

 * the meta-data directory should get a name that doesn't
   imply any specific distribution form...

Calling the directory .egg-info is natural if you're only dealing
with eggs. It is not when running standard "python setup.py install"
or having a mixed setup using different packaging mechanisms.

> 2009/7/7 Ronald Oussoren <ronaldoussoren at mac.com>:
>> But why break existing code without having any other benifits?  If I read
>> the discussion correctly the name would be changed without any changes to
>> the contents of the metadata directory.  I would be more inclined to be in
>> favour if the name change had a sound technical reason, such as a change of
>> the contents of the directory which would make setuptools "egg-info"
>> directories incompatible with the PEP376 ones.
> 
> See MAL's comments above, and my response. If (and only if!) his
> proposal is accepted, then a name change starts to be worth discussing
> further. As things stand at the moment, the structure appears to be
> setuptools-compatibile (other than the new RECORD file, which Phillip
> has committed to adding) so a name change isn't worth it.
> 
> I can't comment myself on setuptools compatibility, so I'm assuming
> here that Phillip will speak up if the proposed format is *not*
> compatible...

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/