If no one objects, I'd like to push PEP 376 in the "accepted" status and go ahead with its implementation, with continuous feedback at Distutils-SIG as we did to build it.

I think this isn't quite the process. In the past, every PEP required BDFL pronouncement, which you should now seek. Regards, Martin

On Tue, Jun 30, 2009 at 2:32 PM, Nick Coghlan wrote:

Martin v. Löwis wrote:

...

...

If no one objects, I'd like to push PEP 376 in the "accepted" status and go ahead with its implementation, with continuous feedback at Distutils-SIG as we did to build it.

I think this isn't quite the process. In the past, every PEP required BDFL pronouncement, which you should now seek.

Agreed. While Guido is highly likely to just accept the distutils-sig consensus on something like this, that doesn't eliminate the need for him to actually *say* that he is approving the PEP on that basis.

Sure, I didn't want to bypass the process, I was not really sure about the right move on this PEP since it was based on the summit decisions, I'll wait for Guido decision, thanks. Cheers Tarek -- Tarek Ziadé | http://ziade.org

2009/6/30 Guido van Rossum :

And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)

My impression (as someone who does read the distutils SIG, but in all honesty has very little practical interest in distutils internals): - It's very focused on distutils internals. If it has an impact on end users (as opposed to packagers), it's very hard to discern. The only hint of such a thing is the mention of an uninstall function. But it's only an API. So still no end user impact [1] :-( - The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters... - It's quite dense for the casual reader not familiar with the terminology. I've never managed to read the whole thing through, personally. I'd suggest two things: - Add a section to the PEP describing the purely end user impact of the changes - Post that to python-list, with a note pointing to the PEP for people who care about distutils details If that gets no feedback, you've done as much as you can. Paul. [1] I'd actually like it if the PEP defined an uninstall command - something like "python -m distutils.uninstall packagename". It can be as minimalist as you like, but I'd like to see it present.

2009/6/30 Guido van Rossum :

On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan wrote:

...

Martin v. Löwis wrote:

...

...

If no one objects, I'd like to push PEP 376 in the "accepted" status and go ahead with its implementation, with continuous feedback at Distutils-SIG as we did to build it.

I think this isn't quite the process. In the past, every PEP required BDFL pronouncement, which you should now seek.

Agreed. While Guido is highly likely to just accept the distutils-sig consensus on something like this, that doesn't eliminate the need for him to actually *say* that he is approving the PEP on that basis.

Sure. :-)

So what *is* the distutils-sig consensus?

The consensus is to have one and only one way to install distributions in Python, and a way to retrieve information on installed distributions, and their files.

And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)

I am not sure who you are thinking about, I am blogging a lot on the topic and I am trying to get key players involved in this, like os packagers etc. We've built this PEP in respect with existing tools like setuptools, etc, and I am sending mails at python-dev to make sure evereyone involved in python development has a chance to provide some feedback,

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

-- Tarek Ziadé | http://ziade.org

Tarek Ziadé wrote:

On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote:

...

- The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters...

setuptools implemented *needed* features, like a way for developers to browse installed packages.

That doesn't imply that setuptools itself is needed. I can browse installed packages with dpkg -l, or "add-and-remove programs".

Now if the fact that we want to introduce the good ideas of setuptools into distutils, (problems Phillip resolved) will make people push it back *even* they are good idead, needed features, is something we need to fight against.

Assuming "we" have consensus before we start fighting against others. FWIW, I abstain from commenting on PEP 376. I don't need it, but it doesn't seem to hurt having it, especially since "egg-info" already managed to sneak in.

I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit'

Just be patient. Both Python 2.7 and 3.2 are still many months ahead, so there is no urgency (AFAICT). Regards, Martin

2009/6/30 Tarek Ziadé :

On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote:

...

- The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters...

setuptools implemented *needed* features, like a way for developers to browse installed packages.

No problem with any of that. Please understand that on the whole, I'm in favour of the setuptools features. (While I dislike intensely some of the setuptools "ecosystem", such as easy_install. In fact, one of my goals in wanting this PEP accepted is to ensure that I can get the "good bits" of setuptools, without having to buy into the easy_install, dependency management, automated download infrastructure that currently goes with it).

We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so)

"We" in this context denotes the people at the summit. Please remember that people who weren't there still have an opinion - and it may well differ. I'm not saying that it *does* differ, just that the view of the summit should not be viewed as conclusive (unless the way Python is developed is very different from how I understand it).

So I worked in PEP 376 to introduce them.

And as with any PEP, that's a difficult and thankless task (I got a taste myself with PEP 302, and that was far easier), so just for the record, you have my thanks for doing this.

Now if the fact that we want to introduce the good ideas of setuptools into distutils, (problems Phillip resolved) will make people push it back *even* they are good idead, needed features,  is something we need to fight against.

But there *are* issues with setuptools. Some users have mentioned them on a number of occasions. I've raised a few myself. By all means introduce the good ideas into the core - but make sure people agree that the ideas *are* good. And remember - all I said was that people may react against the setuptools-style terminology. Not that they dislike the *idea*, just that they might dislike the way it's presented. After all, one common complaint with setuptools (and one that I agree with) is that its documentation is obscure to the point of being hostile.

...

- It's quite dense for the casual reader not familiar with the terminology. I've never managed to read the whole thing through, personally.

I'll try to add a definitions section,

Thank you. I'll try to make the time to go through the PEP and comment more fully.

...

[1] I'd actually like it if the PEP defined an uninstall command - something like "python -m distutils.uninstall packagename". It can be as minimalist as you like, but I'd like to see it present.

it's already there:

http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function

No, that defines an API, which as stated in the PEP, "allows a third-party application to use the uninstall function and make sure it's the only program that can remove a distribution it has previously installed". It does NOT define a standard uninstall command, to complement the standard install command. Paul.

On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano wrote:

On Wed, 1 Jul 2009 05:19:07 am Tarek Ziadé wrote:

...

2009/6/30 Guido van Rossum : ...

...

So what *is* the distutils-sig consensus?

The consensus is to have one and only one way to install distributions in Python,

"One and only one way"? Shouldn't that be "one obvious way"?

There is a big difference -- the first implies that Python should actively prohibit (somehow!) or at least inhibit any other install mechanisms, while the second implies that there should be a preferred, obvious, but not compulsory, mechanism.

If you say "one and only one", and I take you at your word, then I can only assume that you want to actively prevent me from manually dropping .py files into my PYTHONPATH and having them work. I would guess that's probably not what you mean, but that's what it sounds like you're proposing. I don't like guessing -- would you please clarify what you mean?

one and only one way to install a distribution using its setup.py script, among the ways provided by easy_install, distutils and pip. e.g. avoiding having different format/location for the generated .egg.info;

Some questions and comments regarding the PEP:

Rationale: http://www.python.org/dev/peps/pep-0376/#id13

"There are too many ways to do it."

Why is having multiple ways of adding distributions a problem to be solved? On my Linux system, I can add packages with rpm/yum, or I can compile them from source and manage them myself. I possibly even could (but never have!) install apt-get and use it to manage packages.

If you choose not to manually manage them, the packaging system you use is installing files in standard places where you expect to find them, and provides tools to know what is installed. Right now we don't have this standard in Python stdlib to interact with the distributions in a Python installation. And since packaging systems that are used out there could use the same standard, since we would like to make distutils a reference package for this kind of need, it seems like a good idea to define this standard in here. then each packaging system is free to implement its features on the top of it, ala wsgiref.

Another rationale which should be added:

There is no standard way of uninstalling distributions.

How distributions are installed: http://www.python.org/dev/peps/pep-0376/#id14

"The problem is that many people use easy_install (from the setuptools project [4]) or pip [5] to install their packages, and these third-party tools do not install packages in the same way that Distutils does"

Why is that a problem to be solved?

Being able to retrieve the metadata of an installed distribution, or the list of its installed files, no matter which tool installed it.

Uninstall information: http://www.python.org/dev/peps/pep-0376/#id15

"Under some circumstances, you might not be able to know for sure that you have removed everything, or that you didn't break another distribution by removing a file that is shared among several distributions."

I don't see how this proposal will help in the second case. If you install distribution Spam, containing file spam.py, and then install distribution Ham, which requires spam.py, what is to prevent you from removing Spam and breaking Ham?

If you don't propose a solution for the dependency problem, you should say so.

This problem is solved as described later in the PEP, with the API that allows you to get the list of the distributions that use a given file. (thanks to the RECORD files) If Spam and Ham use smap.py, and if you uninstall Spam, this file will not be removed because the API will tell you its used in both distributions.

The RECORD format: http://www.python.org/dev/peps/pep-0376/#id19

"Each record is composed of three elements. ... the MD5 hash of the file, encoded in hex. Notice that pyc and pyo generated files will not have a hash because they are automatically produced from py files."

What if your distribution is not a source distribution and only provides pyc and pyo files?

Good question, I have never created such distribution. Aren't they read-only files ? What happens if someone change them, do we really want to keep them since it breaks the distribution ? How they can be changed in the first place, without the source ? Tarek

On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore wrote:

...

We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so)

"We" in this context denotes the people at the summit. Please remember that people who weren't there still have an opinion - and it may well differ. I'm not saying that it *does* differ, just that the view of the summit should not be viewed as conclusive (unless the way Python is developed is very different from how I understand it).

Sorry I didn't mean it this way. The summit was just the kickoff for this PEP, but with the strong desire to have a standard + some API for it. Also, notice that I put in "We" also the people that answered the survey I did before the summit. (http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/) I'll make other fixes in the PEP with the feedback you gave, thanks Tarek

Kevin Teague wrote:

On Jun 30, 2009, at 4:46 PM, Tarek Ziadé wrote:

...

On Tue, Jun 30, 2009 at 10:06 PM, Scott David Daniels wrote:

...

Tarek Ziadé wrote:

...

On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote:

...

[1] I'd actually like it if the PEP defined an uninstall command - something like "python -m distutils.uninstall packagename". It can be as minimalist as you like, but I'd like to see it present. it's already there: http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function That (at least as I read it) is a function, not a command. If it is a command, give an example of its use from the command line for us poor "don't want to research" people. If the following works: $ python setup.py uninstall some_package Then explicitly say so for us poor schlubs. Right, I'll add that. Although it will be a reference implementation only. Uninstall as a command feels a little weird.... I dunno what the right solution is. My two-cents is either to punt and only include an uninstall function as currently proposed, or for only supporting some form of the "python setup.py uninstall" style....

But for us poor schlubs, we want you brilliant packagers to actually come to a hard decision. If you want approval either admit you have no solution in the PEP (and detail the issues that prevent a decision), provide a minimally acceptable command, or expect that nobody sees the value of what you propose.... --Scott David Daniels Scott.Daniels@Acm.Org

I dunno what the right solution is.

I feel that it is straight forward. Either provide a /usr/bin/python-uninstall utility, or arrange to make python -mdistutils.uninstall work, so one would do python -mdistutils.uninstall some_package Regards, Martin

On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord wrote:

...

Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of:

$ python setup.py uninstall some_package

It could be:

  $ python -m distutils uninstall some_package

Asymmetrical with the install of course.

Yes exactly, I was going to add: $ python -m distutils.uninstall some_package The whole point of the RECORD file is to be able to uninstall without depending on the original archive used to install

On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore wrote:

2009/7/1 Tarek Ziadé :

...

...

That (at least as I read it) is a function, not a command. If it is a command, give an example of its use from the command line for us poor "don't want to research" people.  If the following works:

   $ python setup.py uninstall some_package

Then explicitly say so for us poor schlubs.

Right, I'll add that. Although it will be a reference implementation only.

-1. Where does the setup.py file come from? If I have docutils installed, and want to remove it, must I download the source again so that I can get the setup.py, so I can run the uninstall? This is daft - particularly given that the point of PEP 376 is to ensure that all of the required information is available from the installed package!

As I suggested before:

   python -m distutils.uninstall packagename

yes sorry if it was unclear, I was not thinking about adding something based on setup.py, but just saying that I was going to add this feature in the PEP. and it will be of the form: python -m distutils.uninstall packagename

Calling it a "reference implementation" should not imply that it's not built to be usable and complete. If it's there,m people should be able to use it.

It will be usable and complete, but very limited. As someone mentioned, it will not take care of dependencies and prevent you from removing a distribution that is mentioned in another distrubution in a setuptools' install_requires metadata. That said, when PEP 345 evolves like we have planned to (adding install_requires in the metadata) Then we should be able to provide this kind of warning with no pain.

2009/6/30 Paul Moore :

Thank you. I'll try to make the time to go through the PEP and comment more fully.

OK, I've now read the PEP through in full. My comments follow. I have deliberately avoided mentioning points that others have already raised, to keep things shorter. First of all, after I got over the use of setuptools terminology, the document is very good. It's clear what it's trying to achieve, and covers most of the points I'd want it to. Occasionally the presentation is confusing, see below, but I think everything was there. I do still feel that the setuptools focus will put some readers off. The introductory remarks, in particular, assume a reasonable level of familiarity from readers about setuptools, easy_install, pip and the like. Not having that familiarity isn't a problem in fact, but the assumption is there in the tone. (For example, personally, I hate the unnecessarily cute term "egg", but it's established by now, so that's a lost cause :-() Onto the PEP. "The problem is that many people use easy_install or pip to install their packages..." As already noted by others, it's not clear why this is a problem. But worse is the fact that the paragraph reads to me as saying that easy_install/pip/etc don't follow the *current* standard structure. Given this fact (which *is* a problem!) I fail to see why I should expect them to follow any *new* standard - so how does this PEP actually address the issue? Rather, it seems to say to me that the existing tools have a history of ignoring the standard approach, so this PEP is going to be useless :-( [I expect that isn't what you're trying to say, so you may just need to clarify your meaning. But I do think it's important to address the question of how this PEP is going to ensure that existing tools adapt. In particular, setuptools seems to have completely stagnated, so I see very little likelihood that setuptools is going to change to conform to the PEP - how does that affect things?] ".egg-info becomes a directory" Don't refer to the setuptools documentation (EggFormats)! It is obscure and user-hostile, as well as not actually being the same as the PEP's proposal. Rather, just document the structure of .egg-info. If you want, you can then add a cross-reference note, saying something like "The setuptools structure, as proposed in the EggFormats documentation for that package [ref], is a subset of this standard. In order to conform to this PEP, setuptools will have to be amended to only install .egg-info directories in the format defined by this PEP". "However, it will impact the setuptools and pip projects, but given the fact that..." Confusing. Will these tools need to change (I believe so) or not? If they will need to change, that hardly counts as "no deep consequences" - there's the whole backward compatibility issue for them to handle. "Adding a RECORD file" You say "at installation time" - please clarify. Do you only mean setup.py install? What about bdist_wininst and bdist_msi? What about third party bdist style tools? How will they ensure they get a RECORD file? "The RECORD format" The line separator shouldn't be os-dependent. What value is used for a pure-python (ie, platform independent) package? Unless the file is generated when the install is done, in which case see the previous point... Absolute file paths - this implies that RECORD has to be generated by the installer (which is the only place that knows absolute paths) which means that every bdist_xxx installer has to write its own RECORD file. Does the PEP offer no support for this? In any case, the bdist_wininst and bdist_msi code (which is core distutils) will need to be amended to manage RECORD files appropriately. "Adding an INSTALLER file in the .egg-info directory" Same question again - does the PEP offer supporting APIs to help bdist_xxx installers do this? What about the core bdist_xxx commands? "New APIs in pkgutil" You say "the best place to put these APIS seems to be pkgutil". You should be more definite - "these APIs will be added to the pkgutil module". The Distribution/ZippedDistribution and DistributionDir/ZippedDistributionDir pairs seem to imply that users need to explicitly instantiate these classes (and hence know whether a distribution is zipped). This is cleared up later, but you should add a note here - "Users will not need to create instances of these classes manually, they are returned by the public functions in pkgutil, such as get_distributions()". "DistributionDirMap class" This seems to be an internal implementation detail, and as such should not be documented. It's never returned from any of the public APIs, and the only created instance is a hidden internal global instance. The whole part about purging the cache is also unnecessary - the user has no interface to allow them to do this, so you don't need to document that it's possible. Actually, (Zipped)DistributionDir instances are never returned to the user via the public API, either - so these don't need to be documented. General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain. "Adding an uninstall function" With regard to my earlier comments, my apologies - I misread the text and assumed the documented function simply returned the files to remove, but didn't do the removal itself. (Which makes it far more useless than it actually is). One point - what happens if the uninstall method can't remove a file or directory? Will it leave things in a broken/partially installed state? This needs to be documented (although huge robustness and recovery measures don't need to be taken in a simple reference implementation). This is likely to be a somewhat common issue on Windows in particular, where you can't delete open files. "Installer marker" There are times when I would imagine it would be useful to force uninstallation of the basic package, even if that leaves stuff behind related to a particular installer. A force=True argument would be the easiest, but if you'd rather not add that, it would be useful to document the workaround: inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read() uninstall('docutils', installer=inst) If nothing else, it shows a simple example of how to use the APIs. A few more such examples in the PEP might be nice. But as I said, overall, I'm much happier with the PEP now that I've read it through and taken the time to digest it. Paul.

Kevin Teague wrote:

Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of:

$ python setup.py uninstall some_package

It could just be:

$ python setup.py uninstall

I was actually thinking of something more along the lines of: python -m distutils uninstall <whatever> (Older Pythons won't reliably let you execute a package like that, but 3.1+ and 2.7+ let you do it by including a __main__ module in the package) However, having uninstall as a command supported by setup.py also makes a certain amount of sense. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Right, the API part is almost empty, http://docs.python.org/distutils/apiref.html I'll complete it for the relevant part. On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore wrote:

2009/7/1 Paul Moore :

...

One other thought. You haven't documented the DistributionMetaData class. The

Just noticed that it's defined in distutils. But it's not documented there :-(

Maybe just add a bit to the PEP saying that the class exists in distutils, give a brief summary, and say that as part of implimenting the PEP it will be formally documented. (Even though distutils is a mess of undocumented functionality, I don't think that you can base new work on undocumented internals - you should document them first, or things will never get any better :-()

Paul.

-- Tarek Ziadé | http://ziade.org

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote:

"The problem is that many people use easy_install or pip to install their packages..."

As already noted by others, it's not clear why this is a problem. But worse is the fact that the paragraph reads to me as saying that easy_install/pip/etc don't follow the *current* standard structure. Given this fact (which *is* a problem!) I fail to see why I should expect them to follow any *new* standard - so how does this PEP actually address the issue? Rather, it seems to say to me that the existing tools have a history of ignoring the standard approach, so this PEP is going to be useless :-(

[I expect that isn't what you're trying to say, so you may just need to clarify your meaning. But I do think it's important to address the question of how this PEP is going to ensure that existing tools adapt. In particular, setuptools seems to have completely stagnated, so I see very little likelihood that setuptools is going to change to conform to the PEP - how does that affect things?]

Right it's unclear, I'll work on this part. To resume : - Phase 1 : introduction of the egg-info file in distutils Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476) alongside distutils-installed package, that contains the metadata of the distribution. - Phase 2: two new formats in the setuptools project Then he created two new formats in the setuptools project: 1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened because other files were added in that directory. 2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files. Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (–single-version-externally-managed *or* --root). - Phase 3: adopting a unified standard. It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.: - This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Set...) The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers. So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard. This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense.

".egg-info becomes a directory"

Don't refer to the setuptools documentation (EggFormats)! It is obscure and user-hostile, as well as not actually being the same as the PEP's proposal. Rather, just document the structure of .egg-info. If you want, you can then add a cross-reference note, saying something like "The setuptools structure, as proposed in the EggFormats documentation for that package [ref], is a subset of this standard. In order to conform to this PEP, setuptools will have to be amended to only install .egg-info directories in the format defined by this PEP".

I'll work that way.

"However, it will impact the setuptools and pip projects, but given the fact that..."

Confusing. Will these tools need to change (I believe so) or not? If they will need to change, that hardly counts as "no deep consequences" - there's the whole backward compatibility issue for them to handle.

I'll add this in a backward compatibility section, as suggested earlier by someone.

"Adding a RECORD file"

"The RECORD format"

"Adding an INSTALLER file in the .egg-info directory"

These section will have more details about the way they interact with bdist_* for the rest of the mail, I'll clarify the rest of the PEP using your feedback;

"New APIs in pkgutil"

You say "the best place to put these APIS seems to be pkgutil". You should be more definite - "these APIs will be added to the pkgutil module".

ok

General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain.

The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes. Maybe this should be removed from the PEP and but into another document targeted to developers ?

"Adding an uninstall function"

One point - what happens if the uninstall method can't remove a file or directory? Will it leave things in a broken/partially installed state? This needs to be documented (although huge robustness and recovery measures don't need to be taken in a simple reference implementation). This is likely to be a somewhat common issue on Windows in particular, where you can't delete open files.

Good question. I didn't go that far yet because I didn't implement that part in the prototype. Maybe the files could be moved to a temporary place before they are all deleted, allowing to cancel the uninstallation in case a file couldn't be moved.

"Installer marker"

There are times when I would imagine it would be useful to force uninstallation of the basic package, even if that leaves stuff behind related to a particular installer. A force=True argument would be the easiest, but if you'd rather not add that, it would be useful to document the workaround:

   inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read()    uninstall('docutils', installer=inst)

Ok,

If nothing else, it shows a simple example of how to use the APIs. A few more such examples in the PEP might be nice.

But as I said, overall, I'm much happier with the PEP now that I've read it through and taken the time to digest it.

Great thanks for the feedback. Tarek

2009/7/1 Tarek Ziadé :

Right it's unclear, I'll work on this part.

To resume :

- Phase 1 : introduction of the egg-info file in distutils

Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476)  alongside distutils-installed package, that contains the metadata of the distribution.

- Phase 2:  two new formats in the setuptools project

Then he created two new formats in the setuptools project:

1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened   because other files were added in that directory.

2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files.

Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (–single-version-externally-managed *or* --root).

- Phase 3: adopting a unified standard.

It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.:

- This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Set...)

The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers.

So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard.

This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense.

Ah, that makes a *lot* more sense. So, in a "compatibility" section, you could point out that the proposed standard is compatible with setuptools "single version externally managed" format, and that the setuptools multi-version format (the egg file/directory) is a non-standard format designed to allow multiple versions to be installed at once - something out of scope for this PEP. With that explanation, people like me who glaze over at the complexities of setuptools scenarios should be able to see what's going on. Thanks for clarifying.

...

General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain.

The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes.

Hmm, but you don't explain how they should do that. I certainly wouldn't know how to. As the public API is pkgutil.get_distribution (and the like) which isn't a class so cannot be subclassed, it's not clear how a 3rd party could hook in a subclass of (Zipped)Distribution. And as there's nothing in the public API that consumes these classes, creating them by hand is of no use, either.

Maybe this should be removed from the PEP and but into another document targeted to developers ?

No. If it's needed by developers, it should be defined in the PEP, not elsewhere. But it should be defined in a way that developers can use - not just for the sake of defining it. Unless there is a concrete use case (which should be stated as an example in the PEP) then *not* documenting the classes is of more help to developers, as they don't have to follow a definition that isn't used. Paul.

2009/7/1 R. David Murray :

I haven't read the PEP in detail since it's outside my area of interest and knowledge(*), but my understanding of the goal is that the PEP is providing an _infrastructure_ for system-level package management tools. The uninstall function is part of that infrastructure, but since distutils isn't a package manager itself (it's an install program), distutils as currently organized can't really handle uninstall except as outlined in a section you may have clipped from the above context (ie: when setup.py from the original package is available).

From a Windows user's POV, "install program" = "package manager". And an install program needs an uninstall feature. I know Linux users with their advanced dependency-managing package managers feel that this is a stone-age view, and they may be right, but the PEP needs to take the Windows situation into account.

The question is what do the people who do real package management (linux distribution level package management and the equivalent) think? I'm guessing they are happy with just having the function for their package management tools to call when needed, since (I'm hoping) they are part of the distutils sig....

Don't forget that the maintainers of the bdist_wininst, bdist_msi and bdist_rpm code *are* the distutils maintainers - so to that extent, the PEP has to say how *those* aspects of package managers are covered. Unless another PEP is accepted saying that support for bdist_xxx is being dropped [1], this part of distutils cannot be ignored.

So, if my understanding of the overall goal is correct, it looks to me like the PEP is missing a "motivation" section that talks about system package managers.

Possibly. If so, though, it must discuss the above 3 cases which are part of core distutils. Paul. [1] A PEP I plan on strongly opposing!

Paul Moore writes:

2009/7/1 R. David Murray :

...

The uninstall function is part of that infrastructure, but since distutils isn't a package manager itself (it's an install program), distutils as currently organized can't really handle uninstall except as outlined in a section you may have clipped from the above context (ie: when setup.py from the original package is available).

...

From a Windows user's POV, "install program" = "package manager". And an install program needs an uninstall feature. I know Linux users with their advanced dependency-managing package managers feel that this is a stone-age view, and they may be right, but the PEP needs to take the Windows situation into account.

Actually, your view seems quite reasonable to me: as a GNU user with my advanced dependency-managing package manager, I feel just as strongly that a program that installs is implicitly promising that it can also uninstall cleanly. I think that calling distutils an “install program” is confusing. When I discuss distutils, I don't call it a program at all; it's a library (or perhaps “framework”) that provides part of the job of package installation. -- \ “I think it would be a good idea.” —Mahatma Gandhi (when asked | `\ what he thought of Western civilization) | _o__) | Ben Finney

Tarek Ziadé wrote:

On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord wrote:

...

...

Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of:

$ python setup.py uninstall some_package It could be:

$ python -m distutils uninstall some_package

Asymmetrical with the install of course.

Yes exactly, I was going to add:

$ python -m distutils.uninstall some_package

A directly executable submodule is an even better idea than making distutils itself executable. Definitely worth mentioning in the PEP in the section on uninstallation support. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Paul Moore writes:

2009/7/2 Ben Finney :

...

I think that calling distutils an “install program” is confusing. When I discuss distutils, I don't call it a program at all; it's a library (or perhaps “framework”) that provides part of the job of package installation.

I agree, but I consider the fact that distutils "provides" python setup.py install means that it "is an install program" in some people's minds. These are the same people who complain about the lack of a distutils uninstall feature.

Well, I don't suffer from that confusion between the library and a specific program, yet I still want distutils to support uninstall :-) and hence am interested in seeing the fate of this PEP. -- \ “Men never do evil so completely and cheerfully as when they do | `\ it from religious conviction.” —Blaise Pascal (1623–1662), | _o__) Pensées, #894. | Ben Finney

2009/7/2 Tarek Ziadé :

For absolute paths now that gets installed, what would be the difference between the pre-generated RECORD file and the RECORD file installed on the win32 target system, if any ?

When is an absolute path generated? If you can give me a small sample of a distribution that installs a file in an absolute path, I'll do some testing on Windows. But the immediate answer is that there are *no* reliable "absolute" paths on Windows, so you're not looking at the likes of /usr/doc, but rather paths that are relative to sys.prefix, but not to the package directory. In that case, the key point is that if the installer is built on a system where Python is installed in a different directory than the system where the installer is run, paths need to be relocated. (E.g., C:\Python27 vs D:\Apps\Python27). Paul.

Guido van Rossum writes:

I noted an inconsistency: first you say that the RECORD file uses the excel dialect, but at the end of the same section you say it uses the default csv settings. Sounds like you need to delete one or the other.

If I remember correctly, the default csv dialect is just the Excel dialect, so this would be two different ways of saying the same things. Regards Antoine.

On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore wrote:

Ta. I'll give it a go tonight. But haven't you made the point yourself by saying I'll need to change the directory? This will fail for me as I don't have a "/tmp" directory. So I'd expect a bdist installer (*any* bdist installer) to fail, as it's got no way to tell where to put that file. I can only imagine this type of thing being done by someone packaging for Unix, with no interest in cross-platform portability (I suspect MAC OS directory names differ from Unix as well).

It will fail if your path starts with "/" and you are under win32 because there's a converter that will raise an error in that case at installation time. But you can use some code in your setup.py to provide the right directory name when install is called. I don't see it as a problem. I'm just afraid it's impossible to use efficiently under win32 because of the drive letter. But that's rather a bug.

So maybe absolute filenames should be banned in distutils? (OK, less radically, maybe someone should ask where that functionality is used in practice, and based on the answers come up with a more portable approach).

I'll try to see if I can collect that info out of PyPI with a script, to list the real-world usages. Maybe Martin you have a simple way to do this on PyPI ? Tarek

Antoine Pitrou wrote:

Guido van Rossum writes:

...

I noted an inconsistency: first you say that the RECORD file uses the excel dialect, but at the end of the same section you say it uses the default csv settings. Sounds like you need to delete one or the other.

If I remember correctly, the default csv dialect is just the Excel dialect, so this would be two different ways of saying the same things.

Right, but Guido's point is, decide which one is the is the definition and stick to talking about it in that form. --Scott David Daniels Scott.Daniels@Acm.Org

Tarek Ziadé wrote:

On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote:

...

Another general principle: even in the draft PEP, say "is", not "will be".

Ok I'll fix that. That's a French stuff : in french, "will be" isn't speculative at all.

I don't think "will be" is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, "X will be done." If "X" is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do "X". In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of "will be" in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu