[Python-Dev] PEP 376

[Tarek Ziadé]

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> "The problem is that many people use easy_install or pip to install
> their packages..."
>
> As already noted by others, it's not clear why this is a problem. But
> worse is the fact that the paragraph reads to me as saying that
> easy_install/pip/etc don't follow the *current* standard structure.
> Given this fact (which *is* a problem!) I fail to see why I should
> expect them to follow any *new* standard - so how does this PEP
> actually address the issue? Rather, it seems to say to me that the
> existing tools have a history of ignoring the standard approach, so
> this PEP is going to be useless :-(
>
> [I expect that isn't what you're trying to say, so you may just need
> to clarify your meaning. But I do think it's important to address the
> question of how this PEP is going to ensure that existing tools adapt.
> In particular, setuptools seems to have completely stagnated, so I see
> very little likelihood that setuptools is going to change to conform
> to the PEP - how does that affect things?]

Right it's unclear, I'll work on this part.

To resume :

- Phase 1 : introduction of the egg-info file in distutils

Philipp introduced the creation of a file named xxx.egg-info file in 2006
(see http://bugs.python.org/issue1459476)  alongside
distutils-installed package, that contains
the metadata of the distribution.

- Phase 2:  two new formats in the setuptools project

Then he created two new formats in the setuptools project:

1. an .egg-info directory containing a PKG-INFO file, which is similar
to the previous .egg-info file. This happened
   because other files were added in that directory.

2. an .egg directory, possibly zipped, that is a self-contained
version of the distribution, containing the metadata and all other
files.

Setuptools uses 2. by default. There's an option when you use
setuptools to force 1. (–single-version-externally-managed *or*
--root).

- Phase 3: adopting a unified standard.

It occurs that having a .egg-info directory (1.) is way better than a
single file because it's a place-holder for other files.
It is also adopted by the community when it comes to install
setuptools-based package.:

- This is what "pip" uses to install packages in a more flat manner.
- It's also the policy under debian
(http://wiki.debian.org/DebianPython/NewPolicy)
- and under Fedora
(http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools)

The .egg directory (2.) is more controversial because it a
self-contained directory that doesn't install packages
so it makes two different ways to work with distributions for packagers.

So I have proposed in the PEP to adopt the standalone .egg-info
directory as the new distutils unified standard.

This means that all the third-party tools out there already conform to
that standard, and that packages installed
in other formats will not benefit from the new APIs. which means that
people that want to work with distributions installed as .egg
directories will have to use setuptools APIs.
Which makes sense.

>
> ".egg-info becomes a directory"
>
> Don't refer to the setuptools documentation (EggFormats)! It is
> obscure and user-hostile, as well as not actually being the same as
> the PEP's proposal. Rather, just document the structure of .egg-info.
> If you want, you can then add a cross-reference note, saying something
> like "The setuptools structure, as proposed in the EggFormats
> documentation for that package [ref], is a subset of this standard. In
> order to conform to this PEP, setuptools will have to be amended to
> only install .egg-info directories in the format defined by this PEP".

I'll work that way.

>
> "However, it will impact the setuptools and pip projects, but given
> the fact that..."
>
> Confusing. Will these tools need to change (I believe so) or not? If
> they will need to change, that hardly counts as "no deep consequences"
> - there's the whole backward compatibility issue for them to handle.

I'll add this in a backward compatibility section, as suggested
earlier by someone.

>
> "Adding a RECORD file"
>
> "The RECORD format"
>
> "Adding an INSTALLER file in the .egg-info directory"

These section will have more details about the way they interact with bdist_*

for the rest of the mail, I'll clarify the rest of the PEP using your feedback;

> "New APIs in pkgutil"
>
> You say "the best place to put these APIS seems to be pkgutil". You
> should be more definite - "these APIs will be added to the pkgutil
> module".

ok

> General rule - don't document (and commit yourself to) any internal
> details that the user can't access from the public API. It just makes
> backward compatibility harder to maintain.

The purpose is to provide this documentation to third-party projects that want
to implement a packaging system on the top of these classes.

Maybe this should be removed from the PEP and but into another
document targeted to developers ?

>
> "Adding an uninstall function"
>
> One point - what happens if the uninstall method can't remove a file
> or directory? Will it leave things in a broken/partially installed
> state? This needs to be documented (although huge robustness and
> recovery measures don't need to be taken in a simple reference
> implementation). This is likely to be a somewhat common issue on
> Windows in particular, where you can't delete open files.

Good question. I didn't go that far yet because I didn't implement
that part in the
prototype. Maybe the files could be moved to a temporary place before they are
all deleted, allowing to cancel the uninstallation in case a file
couldn't be moved.

> "Installer marker"
>
> There are times when I would imagine it would be useful to force
> uninstallation of the basic package, even if that leaves stuff behind
> related to a particular installer. A force=True argument would be the
> easiest, but if you'd rather not add that, it would be useful to
> document the workaround:
>
>    inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read()
>    uninstall('docutils', installer=inst)

Ok,

>
> If nothing else, it shows a simple example of how to use the APIs. A
> few more such examples in the PEP might be nice.
>
> But as I said, overall, I'm much happier with the PEP now that I've
> read it through and taken the time to digest it.

Great thanks for the feedback.

Tarek