
Hello, If no one objects, I'd like to push PEP 376 in the "accepted" status and go ahead with its implementation, with continuous feedback at Distutils-SIG as we did to build it. The next PEPs that are being discussed at Distutils-SIG are : - the new version of PEP 345 for the inclusion of fields like "installed_requires" (dependencies) - PEP 386, that talks about distribution version numbers (because it's needed to describe dependencies) While they are still under heavy discussion, I have good hope that they will both make it for 2.7 and 3.2. Regards Tarek -- Tarek Ziadé | http://ziade.org

Martin v. Löwis wrote:
Agreed. While Guido is highly likely to just accept the distutils-sig consensus on something like this, that doesn't eliminate the need for him to actually *say* that he is approving the PEP on that basis. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Tue, Jun 30, 2009 at 2:32 PM, Nick Coghlan<ncoghlan@gmail.com> wrote:
Sure, I didn't want to bypass the process, I was not really sure about the right move on this PEP since it was based on the summit decisions, I'll wait for Guido decision, thanks. Cheers Tarek -- Tarek Ziadé | http://ziade.org

On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan<ncoghlan@gmail.com> wrote:
Sure. :-) So what *is* the distutils-sig consensus? And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2009/6/30 Guido van Rossum <guido@python.org>:
And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)
My impression (as someone who does read the distutils SIG, but in all honesty has very little practical interest in distutils internals): - It's very focused on distutils internals. If it has an impact on end users (as opposed to packagers), it's very hard to discern. The only hint of such a thing is the mention of an uninstall function. But it's only an API. So still no end user impact [1] :-( - The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters... - It's quite dense for the casual reader not familiar with the terminology. I've never managed to read the whole thing through, personally. I'd suggest two things: - Add a section to the PEP describing the purely end user impact of the changes - Post that to python-list, with a note pointing to the PEP for people who care about distutils details If that gets no feedback, you've done as much as you can. Paul. [1] I'd actually like it if the PEP defined an uninstall command - something like "python -m distutils.uninstall packagename". It can be as minimalist as you like, but I'd like to see it present.

On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore@gmail.com> wrote:
- The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters...
setuptools implemented *needed* features, like a way for developers to browse installed packages. We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so) So I worked in PEP 376 to introduce them. Now if the fact that we want to introduce the good ideas of setuptools into distutils, (problems Phillip resolved) will make people push it back *even* they are good idead, needed features, is something we need to fight against.
I'll try to add a definitions section,
Ok.
If that gets no feedback, you've done as much as you can.
I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit'
it's already there: http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function Regards Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé wrote:
That doesn't imply that setuptools itself is needed. I can browse installed packages with dpkg -l, or "add-and-remove programs".
Assuming "we" have consensus before we start fighting against others. FWIW, I abstain from commenting on PEP 376. I don't need it, but it doesn't seem to hurt having it, especially since "egg-info" already managed to sneak in.
I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit'
Just be patient. Both Python 2.7 and 3.2 are still many months ahead, so there is no urgency (AFAICT). Regards, Martin

Tarek Ziadé wrote:
That (at least as I read it) is a function, not a command. If it is a command, give an example of its use from the command line for us poor "don't want to research" people. If the following works: $ python setup.py uninstall some_package Then explicitly say so for us poor schlubs. --Scott David Daniels Scott.Daniels@Acm.Org

On Jun 30, 2009, at 4:46 PM, Tarek Ziadé wrote:
Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of: $ python setup.py uninstall some_package It could just be: $ python setup.py uninstall Except then you'd need to have a complete distribution kicking around with which to run the "python setup.py uninstall" command just to tell the uninstall command the distribution name you want to uninstall. But then with the other uninstall format you could uninstall any distribution from within any other distribution, which is convenient, but weird ... e.g.: $ cd Spam-1.0/ $ python setup.py uninstall Foo Although even the other version of the command could do weird stuff: $ cd Spam-1.0/ $ python setup.py install $ cd ../Spam-2.0/ $ python setup.py uninstall Which would presumably remove the Spam 1.0 distribution when run from the 2.0 version of it! Or perhaps this command should only allow uninstall to be run from a distribution whose name and version match the one that it was installed from ... I dunno what the right solution is. My two-cents is either to punt and only include an uninstall function as currently proposed, or for only supporting some form of the "python setup.py uninstall" style since I would guess that the most common use-case for uninstall is: user downloads a distribution, runs "python setup.py install", tries out the package, decides they don't like package, then runs "python setup.py uninstall" to restore their python back to it's original state. For doing anything more complex than that, people should be encouraged to use another one of the existing tools for managing their distributions, IMHO.

Kevin Teague wrote:
But for us poor schlubs, we want you brilliant packagers to actually come to a hard decision. If you want approval either admit you have no solution in the PEP (and detail the issues that prevent a decision), provide a minimally acceptable command, or expect that nobody sees the value of what you propose.... --Scott David Daniels Scott.Daniels@Acm.Org

On Tue, 30 Jun 2009 at 20:06, Scott David Daniels wrote:
I haven't read the PEP in detail since it's outside my area of interest and knowledge(*), but my understanding of the goal is that the PEP is providing an _infrastructure_ for system-level package management tools. The uninstall function is part of that infrastructure, but since distutils isn't a package manager itself (it's an install program), distutils as currently organized can't really handle uninstall except as outlined in a section you may have clipped from the above context (ie: when setup.py from the original package is available). A possible implementation at the python/distutils level might be to have a 'pyuninstall' command installed (like pydoc et al are installed) which handles uninstallation. The question is what do the people who do real package management (linux distribution level package management and the equivalent) think? I'm guessing they are happy with just having the function for their package management tools to call when needed, since (I'm hoping) they are part of the distutils sig.... So, if my understanding of the overall goal is correct, it looks to me like the PEP is missing a "motivation" section that talks about system package managers. --David (*) I'll make time to read it anyway soon.

2009/7/1 R. David Murray <rdmurray@bitdance.com>:
Don't forget that the maintainers of the bdist_wininst, bdist_msi and bdist_rpm code *are* the distutils maintainers - so to that extent, the PEP has to say how *those* aspects of package managers are covered. Unless another PEP is accepted saying that support for bdist_xxx is being dropped [1], this part of distutils cannot be ignored.
Possibly. If so, though, it must discuss the above 3 cases which are part of core distutils. Paul. [1] A PEP I plan on strongly opposing!

Paul Moore <p.f.moore@gmail.com> writes:
Actually, your view seems quite reasonable to me: as a GNU user with my advanced dependency-managing package manager, I feel just as strongly that a program that installs is implicitly promising that it can also uninstall cleanly. I think that calling distutils an “install program” is confusing. When I discuss distutils, I don't call it a program at all; it's a library (or perhaps “framework”) that provides part of the job of package installation. -- \ “I think it would be a good idea.” —Mahatma Gandhi (when asked | `\ what he thought of Western civilization) | _o__) | Ben Finney

2009/7/2 Ben Finney <ben+python@benfinney.id.au>:
I agree, but I consider the fact that distutils "provides" python setup.py install means that it "is an install program" in some people's minds. These are the same people who complain about the lack of a distutils uninstall feature. Personally, I want distutils to provide all the individual bits that I get from bdist_wininst, so that I can put together an alternative for specialised situations (notably, non-site installs of some packages). So uninstall and being able to list installed packages are important to me - hence my interest in this PEP. Paul.

Paul Moore <p.f.moore@gmail.com> writes:
Well, I don't suffer from that confusion between the library and a specific program, yet I still want distutils to support uninstall :-) and hence am interested in seeing the fate of this PEP. -- \ “Men never do evil so completely and cheerfully as when they do | `\ it from religious conviction.” —Blaise Pascal (1623–1662), | _o__) Pensées, #894. | Ben Finney

Kevin Teague wrote:
It could be: $ python -m distutils uninstall some_package Asymmetrical with the install of course. Michael

Tarek Ziadé wrote:
A directly executable submodule is an even better idea than making distutils itself executable. Definitely worth mentioning in the PEP in the section on uninstallation support. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Kevin Teague wrote:
I was actually thinking of something more along the lines of: python -m distutils uninstall <whatever> (Older Pythons won't reliably let you execute a package like that, but 3.1+ and 2.7+ let you do it by including a __main__ module in the package) However, having uninstall as a command supported by setup.py also makes a certain amount of sense. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
-1. Where does the setup.py file come from? If I have docutils installed, and want to remove it, must I download the source again so that I can get the setup.py, so I can run the uninstall? This is daft - particularly given that the point of PEP 376 is to ensure that all of the required information is available from the installed package! As I suggested before: python -m distutils.uninstall packagename Calling it a "reference implementation" should not imply that it's not built to be usable and complete. If it's there,m people should be able to use it. Paul.

On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore<p.f.moore@gmail.com> wrote:
yes sorry if it was unclear, I was not thinking about adding something based on setup.py, but just saying that I was going to add this feature in the PEP. and it will be of the form: python -m distutils.uninstall packagename
It will be usable and complete, but very limited. As someone mentioned, it will not take care of dependencies and prevent you from removing a distribution that is mentioned in another distrubution in a setuptools' install_requires metadata. That said, when PEP 345 evolves like we have planned to (adding install_requires in the metadata) Then we should be able to provide this kind of warning with no pain.

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
Thank you. That is precisely the right level for the PEP, IMHO.
That's good, but not something that bothers me too much. (I'm still of the view that dependencies should be handled by documenting them properly - metadata isn't sufficient on its own). Paul.

2009/6/30 Tarek Ziadé <ziade.tarek@gmail.com>:
No problem with any of that. Please understand that on the whole, I'm in favour of the setuptools features. (While I dislike intensely some of the setuptools "ecosystem", such as easy_install. In fact, one of my goals in wanting this PEP accepted is to ensure that I can get the "good bits" of setuptools, without having to buy into the easy_install, dependency management, automated download infrastructure that currently goes with it).
We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so)
"We" in this context denotes the people at the summit. Please remember that people who weren't there still have an opinion - and it may well differ. I'm not saying that it *does* differ, just that the view of the summit should not be viewed as conclusive (unless the way Python is developed is very different from how I understand it).
So I worked in PEP 376 to introduce them.
And as with any PEP, that's a difficult and thankless task (I got a taste myself with PEP 302, and that was far easier), so just for the record, you have my thanks for doing this.
But there *are* issues with setuptools. Some users have mentioned them on a number of occasions. I've raised a few myself. By all means introduce the good ideas into the core - but make sure people agree that the ideas *are* good. And remember - all I said was that people may react against the setuptools-style terminology. Not that they dislike the *idea*, just that they might dislike the way it's presented. After all, one common complaint with setuptools (and one that I agree with) is that its documentation is obscure to the point of being hostile.
Thank you. I'll try to make the time to go through the PEP and comment more fully.
No, that defines an API, which as stated in the PEP, "allows a third-party application to use the uninstall function and make sure it's the only program that can remove a distribution it has previously installed". It does NOT define a standard uninstall command, to complement the standard install command. Paul.

On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Sorry I didn't mean it this way. The summit was just the kickoff for this PEP, but with the strong desire to have a standard + some API for it. Also, notice that I put in "We" also the people that answered the survey I did before the summit. (http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/) I'll make other fixes in the PEP with the feedback you gave, thanks Tarek

2009/6/30 Paul Moore <p.f.moore@gmail.com>:
Thank you. I'll try to make the time to go through the PEP and comment more fully.
OK, I've now read the PEP through in full. My comments follow. I have deliberately avoided mentioning points that others have already raised, to keep things shorter. First of all, after I got over the use of setuptools terminology, the document is very good. It's clear what it's trying to achieve, and covers most of the points I'd want it to. Occasionally the presentation is confusing, see below, but I think everything was there. I do still feel that the setuptools focus will put some readers off. The introductory remarks, in particular, assume a reasonable level of familiarity from readers about setuptools, easy_install, pip and the like. Not having that familiarity isn't a problem in fact, but the assumption is there in the tone. (For example, personally, I hate the unnecessarily cute term "egg", but it's established by now, so that's a lost cause :-() Onto the PEP. "The problem is that many people use easy_install or pip to install their packages..." As already noted by others, it's not clear why this is a problem. But worse is the fact that the paragraph reads to me as saying that easy_install/pip/etc don't follow the *current* standard structure. Given this fact (which *is* a problem!) I fail to see why I should expect them to follow any *new* standard - so how does this PEP actually address the issue? Rather, it seems to say to me that the existing tools have a history of ignoring the standard approach, so this PEP is going to be useless :-( [I expect that isn't what you're trying to say, so you may just need to clarify your meaning. But I do think it's important to address the question of how this PEP is going to ensure that existing tools adapt. In particular, setuptools seems to have completely stagnated, so I see very little likelihood that setuptools is going to change to conform to the PEP - how does that affect things?] ".egg-info becomes a directory" Don't refer to the setuptools documentation (EggFormats)! It is obscure and user-hostile, as well as not actually being the same as the PEP's proposal. Rather, just document the structure of .egg-info. If you want, you can then add a cross-reference note, saying something like "The setuptools structure, as proposed in the EggFormats documentation for that package [ref], is a subset of this standard. In order to conform to this PEP, setuptools will have to be amended to only install .egg-info directories in the format defined by this PEP". "However, it will impact the setuptools and pip projects, but given the fact that..." Confusing. Will these tools need to change (I believe so) or not? If they will need to change, that hardly counts as "no deep consequences" - there's the whole backward compatibility issue for them to handle. "Adding a RECORD file" You say "at installation time" - please clarify. Do you only mean setup.py install? What about bdist_wininst and bdist_msi? What about third party bdist style tools? How will they ensure they get a RECORD file? "The RECORD format" The line separator shouldn't be os-dependent. What value is used for a pure-python (ie, platform independent) package? Unless the file is generated when the install is done, in which case see the previous point... Absolute file paths - this implies that RECORD has to be generated by the installer (which is the only place that knows absolute paths) which means that every bdist_xxx installer has to write its own RECORD file. Does the PEP offer no support for this? In any case, the bdist_wininst and bdist_msi code (which is core distutils) will need to be amended to manage RECORD files appropriately. "Adding an INSTALLER file in the .egg-info directory" Same question again - does the PEP offer supporting APIs to help bdist_xxx installers do this? What about the core bdist_xxx commands? "New APIs in pkgutil" You say "the best place to put these APIS seems to be pkgutil". You should be more definite - "these APIs will be added to the pkgutil module". The Distribution/ZippedDistribution and DistributionDir/ZippedDistributionDir pairs seem to imply that users need to explicitly instantiate these classes (and hence know whether a distribution is zipped). This is cleared up later, but you should add a note here - "Users will not need to create instances of these classes manually, they are returned by the public functions in pkgutil, such as get_distributions()". "DistributionDirMap class" This seems to be an internal implementation detail, and as such should not be documented. It's never returned from any of the public APIs, and the only created instance is a hidden internal global instance. The whole part about purging the cache is also unnecessary - the user has no interface to allow them to do this, so you don't need to document that it's possible. Actually, (Zipped)DistributionDir instances are never returned to the user via the public API, either - so these don't need to be documented. General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain. "Adding an uninstall function" With regard to my earlier comments, my apologies - I misread the text and assumed the documented function simply returned the files to remove, but didn't do the removal itself. (Which makes it far more useless than it actually is). One point - what happens if the uninstall method can't remove a file or directory? Will it leave things in a broken/partially installed state? This needs to be documented (although huge robustness and recovery measures don't need to be taken in a simple reference implementation). This is likely to be a somewhat common issue on Windows in particular, where you can't delete open files. "Installer marker" There are times when I would imagine it would be useful to force uninstallation of the basic package, even if that leaves stuff behind related to a particular installer. A force=True argument would be the easiest, but if you'd rather not add that, it would be useful to document the workaround: inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read() uninstall('docutils', installer=inst) If nothing else, it shows a simple example of how to use the APIs. A few more such examples in the PEP might be nice. But as I said, overall, I'm much happier with the PEP now that I've read it through and taken the time to digest it. Paul.

2009/7/1 Paul Moore <p.f.moore@gmail.com>:
One other thought. You haven't documented the DistributionMetaData class. The And a minor nit: """ Distribution class A new class called Distribution is created with a the path of the .egg-info directory provided to the contructor. It reads the metadata contained in PKG-INFO when it is instanciated. """ instantiated Paul.

2009/7/1 Paul Moore <p.f.moore@gmail.com>:
One other thought. You haven't documented the DistributionMetaData class. The
Just noticed that it's defined in distutils. But it's not documented there :-( Maybe just add a bit to the PEP saying that the class exists in distutils, give a brief summary, and say that as part of implimenting the PEP it will be formally documented. (Even though distutils is a mess of undocumented functionality, I don't think that you can base new work on undocumented internals - you should document them first, or things will never get any better :-() Paul.

Right, the API part is almost empty, http://docs.python.org/distutils/apiref.html I'll complete it for the relevant part. On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore<p.f.moore@gmail.com> wrote:
-- Tarek Ziadé | http://ziade.org

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Right it's unclear, I'll work on this part. To resume : - Phase 1 : introduction of the egg-info file in distutils Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476) alongside distutils-installed package, that contains the metadata of the distribution. - Phase 2: two new formats in the setuptools project Then he created two new formats in the setuptools project: 1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened because other files were added in that directory. 2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files. Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (–single-version-externally-managed *or* --root). - Phase 3: adopting a unified standard. It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.: - This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Set...) The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers. So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard. This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense.
I'll work that way.
I'll add this in a backward compatibility section, as suggested earlier by someone.
These section will have more details about the way they interact with bdist_* for the rest of the mail, I'll clarify the rest of the PEP using your feedback;
ok
The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes. Maybe this should be removed from the PEP and but into another document targeted to developers ?
Good question. I didn't go that far yet because I didn't implement that part in the prototype. Maybe the files could be moved to a temporary place before they are all deleted, allowing to cancel the uninstallation in case a file couldn't be moved.
Ok,
Great thanks for the feedback. Tarek

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
Ah, that makes a *lot* more sense. So, in a "compatibility" section, you could point out that the proposed standard is compatible with setuptools "single version externally managed" format, and that the setuptools multi-version format (the egg file/directory) is a non-standard format designed to allow multiple versions to be installed at once - something out of scope for this PEP. With that explanation, people like me who glaze over at the complexities of setuptools scenarios should be able to see what's going on. Thanks for clarifying.
Hmm, but you don't explain how they should do that. I certainly wouldn't know how to. As the public API is pkgutil.get_distribution (and the like) which isn't a class so cannot be subclassed, it's not clear how a 3rd party could hook in a subclass of (Zipped)Distribution. And as there's nothing in the public API that consumes these classes, creating them by hand is of no use, either.
Maybe this should be removed from the PEP and but into another document targeted to developers ?
No. If it's needed by developers, it should be defined in the PEP, not elsewhere. But it should be defined in a way that developers can use - not just for the sake of defining it. Unless there is a concrete use case (which should be stated as an example in the PEP) then *not* documenting the classes is of more help to developers, as they don't have to follow a definition that isn't used. Paul.

Tarek Ziadé writes:
On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
That's a judgment you must make. However, Paul's opinion seems to be that it is internal, and not needed by third-parties who are working "on the top" of these classes. If upon consideration you agree, you should take those "details" out of the PEP proper. If you disagree, you should promote them to the "official"/public API. The point of a PEP is not to construct a duck; it is to explain what "quack" means.
Maybe this should be removed from the PEP and but into another document targeted to developers ?
Yes. In the reference implementation (aka prototype), which should have its own documentation as usual. The reference implementation for the PEP shows *how* these things can be done. It must do that, or the PEP has no force. However, the reference implementation need not be "industrial strength"; the actual implementation that eventually goes into the stdlib may very well be an optimized and enhanced version with many data structures and algorithms that differ from the reference implementation. Another general principle: even in the draft PEP, say "is", not "will be". If you're wrong, and that won't work; if it's insufficiently precise; if you find a more elegant way to express the solution; if you simply haven't thought carefully about something yet; if, for whatever reason, you may need to change the PEP, then you change it in the next draft. That's why we have a review cycle, so you can change it. If you yourself have a question, or the draft is incomplete at some point, then you can explain in a note (either a footnote or parentheses). You can even mark a whole section as "speculative" (eg, "nothing in this section has been implemented yet, so everything is subject to change"). But a rule should be stated as a rule, and precisely, so that reviewers can criticize it accurately. If you're uncomfortable saying "is", then maybe that part of the draft isn't ready for public review yet. On the other hand, you can use the reviewers' knowledge here: write "A is B", then as a note "(In some cases A is actually C, and this should be treated specially. But how?)"

On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull<stephen@xemacs.org> wrote:
Yes, while the APIs I have written in the prototype+PEP helped us claryfing what we wanted, I agree they would be better in a second document. They are two target audience, the users of distutils and the builders of package managers, so removing this details from the PEP will also make it easier to read for the first crowd.
Another general principle: even in the draft PEP, say "is", not "will be".
Ok I'll fix that. That's a French stuff : in french, "will be" isn't speculative at all. Thx

Tarek Ziadé wrote:
I don't think "will be" is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, "X will be done." If "X" is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do "X". In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of "will be" in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

Scott Dial wrote:
Someone feel free to correct me if I am incorrect about the desired tone and use of the document..
I don't think we're particularly consistent. For example, I actually ending up adding a note to PEP 343 indicating that people shouldn't read too much into the verb tense in that document because we were working on it for over a year and things that were in the future when we started were well in the past by the time we finished. In general, I agree that using present tense ("is") in specifications and documentation such as PEPs ends up standing the test of time better than using future tense ("will be"). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Since all bdist_* commands are using the install command in a temporary build directory to create a binary distribution, the RECORD file (and the INSTALLER) will be pre-generated then installed alongside the packages/modules. For absolute paths now that gets installed, what would be the difference between the pre-generated RECORD file and the RECORD file installed on the win32 target system, if any ?

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
When is an absolute path generated? If you can give me a small sample of a distribution that installs a file in an absolute path, I'll do some testing on Windows. But the immediate answer is that there are *no* reliable "absolute" paths on Windows, so you're not looking at the likes of /usr/doc, but rather paths that are relative to sys.prefix, but not to the package directory. In that case, the key point is that if the installer is built on a system where Python is installed in a different directory than the system where the installer is run, paths need to be relocated. (E.g., C:\Python27 vs D:\Apps\Python27). Paul.

On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Try this setup.py file: """ from distutils.core import setup setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) """ with this MANIFEST.in file: "" include data """ and add a 'data' file alongside If you install it, data is copied in /tmp. If you create a bdist distribution it will be created in the root of the dumb directory which is used to generate the binary distro. Of course you'll have to change '/tmp' to 'c:\tmp' (which makes me realizes that there's no way to force the installation data_files in another drive under windows: the binary distribution will be the same, no matter what drive you use in the absolute path used in data_files I'll add an issue on this)
I get the point: they are three levels we should handle in the RECORD file 1. absolute paths 2. paths relative to sys.prefix or sys.exec_prefix 3. paths relative to the directory where the .egg-info directory is located the RECORD file doesn't handle 2. indeed. For instance you can add a script: setup( .. scripts=['myscript.py'] ..) that will get installed in : sys.prefix + 'Scripts/myscript.py' under win32 for instance So a possible solution is to add 2. in the RECORD files by using a notation such as "$PREFIX" "$EXEC_PREFIX" when we detect that the file is under on of these paths. The query functions will then be able to use sys.prefix and sys.exec_prefix for recompose the absolute pat on the target system
Paul.
-- Tarek Ziadé | http://ziade.org

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
Ta. I'll give it a go tonight. But haven't you made the point yourself by saying I'll need to change the directory? This will fail for me as I don't have a "/tmp" directory. So I'd expect a bdist installer (*any* bdist installer) to fail, as it's got no way to tell where to put that file. I can only imagine this type of thing being done by someone packaging for Unix, with no interest in cross-platform portability (I suspect MAC OS directory names differ from Unix as well). So maybe absolute filenames should be banned in distutils? (OK, less radically, maybe someone should ask where that functionality is used in practice, and based on the answers come up with a more portable approach). Paul.

On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore<p.f.moore@gmail.com> wrote:
It will fail if your path starts with "/" and you are under win32 because there's a converter that will raise an error in that case at installation time. But you can use some code in your setup.py to provide the right directory name when install is called. I don't see it as a problem. I'm just afraid it's impossible to use efficiently under win32 because of the drive letter. But that's rather a bug.
I'll try to see if I can collect that info out of PyPI with a script, to list the real-world usages. Maybe Martin you have a simple way to do this on PyPI ? Tarek

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
Looks like bdist_wininst doesn't support absolute paths. Which isn't surprising, really. Same with bdist_msi. Actually, with setup.py install:
So how do absolute paths work??? It sure ain't like this! Changing '/tmp' to c:/tmp' gives this: directory name is invalid Changing it to 'c:\\tmp' gives (sort of) success: python setup.py bdist_wininst running bdist_wininst running build installing to build\bdist.win32\wininst running install_data creating build\bdist.win32\wininst\tmp copying data -> build\bdist.win32\wininst\tmp running install_egg_info Creating build\bdist.win32\wininst\PURELIB\ Writing build\bdist.win32\wininst\PURELIB\foo-1.0-py2.6.egg-info creating 'c:\users\gustav\appdata\local\temp\tmpcs9rmi.zip' and adding '.' to it adding 'PURELIB\foo-1.0-py2.6.egg-info' adding 'tmp\data' creating dist removing 'build\bdist.win32\wininst' (and everything under it) The installer packs a *relative* path tmp/data - which isn't right. bdist_msi copies the data file to c:\tmp - when building the msi!. This is very wrong. I can't introspect a msi file to see what ends up in it, but if I install it, no c:\tmp\data is installed, so it does look like it's screwed up. Frankly, handling of absolute paths looks so broken, that I'd explicitly state that it's not supported. I'd raise bugs for some of these things, but to be honest, I'm not at all sure what I'd expect to be correct behaviour (on Windows). In my view absolute paths shouldn't be supported - I can see that might be different on Linux, but I don't see how it's ever going to be cross-platform. Not sure where that leaves things. Absolute path handling seems broken enough (on Windows) that whatever the PEP says isn't much of an issue, as it's not usable anyway. One definite point, though - given the behaviour of bdist_msi, it thought it had installed something that it simply didn't - so there's a serious risk that the distutils uninstall function could uninstall something that the installer never installed - that's far worse than simply deleting the files and messing up the installer metadata. Maybe the answer is that distutils reject *all* uses of absolute paths on Windows. The PEP can then say whatever you want as far as I'm concerned, as it doesn't affect me. If the existing bugs don't get fixed, though, I'd say that the RECORD file should be explicitly disallowed from containing absolute paths on Windows (for safety reasons if nothing else). Paul.

2009/6/30 Guido van Rossum <guido@python.org>:
The consensus is to have one and only one way to install distributions in Python, and a way to retrieve information on installed distributions, and their files.
And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)
I am not sure who you are thinking about, I am blogging a lot on the topic and I am trying to get key players involved in this, like os packagers etc. We've built this PEP in respect with existing tools like setuptools, etc, and I am sending mails at python-dev to make sure evereyone involved in python development has a chance to provide some feedback,
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
-- Tarek Ziadé | http://ziade.org

On Wed, 1 Jul 2009 05:19:07 am Tarek Ziadé wrote:
"One and only one way"? Shouldn't that be "one obvious way"? There is a big difference -- the first implies that Python should actively prohibit (somehow!) or at least inhibit any other install mechanisms, while the second implies that there should be a preferred, obvious, but not compulsory, mechanism. If you say "one and only one", and I take you at your word, then I can only assume that you want to actively prevent me from manually dropping .py files into my PYTHONPATH and having them work. I would guess that's probably not what you mean, but that's what it sounds like you're proposing. I don't like guessing -- would you please clarify what you mean? Some questions and comments regarding the PEP: Rationale: http://www.python.org/dev/peps/pep-0376/#id13 "There are too many ways to do it." Why is having multiple ways of adding distributions a problem to be solved? On my Linux system, I can add packages with rpm/yum, or I can compile them from source and manage them myself. I possibly even could (but never have!) install apt-get and use it to manage packages. Another rationale which should be added: There is no standard way of uninstalling distributions. How distributions are installed: http://www.python.org/dev/peps/pep-0376/#id14 "The problem is that many people use easy_install (from the setuptools project [4]) or pip [5] to install their packages, and these third-party tools do not install packages in the same way that Distutils does" Why is that a problem to be solved? Uninstall information: http://www.python.org/dev/peps/pep-0376/#id15 "Under some circumstances, you might not be able to know for sure that you have removed everything, or that you didn't break another distribution by removing a file that is shared among several distributions." I don't see how this proposal will help in the second case. If you install distribution Spam, containing file spam.py, and then install distribution Ham, which requires spam.py, what is to prevent you from removing Spam and breaking Ham? If you don't propose a solution for the dependency problem, you should say so. The RECORD format: http://www.python.org/dev/peps/pep-0376/#id19 "Each record is composed of three elements. ... the MD5 hash of the file, encoded in hex. Notice that pyc and pyo generated files will not have a hash because they are automatically produced from py files." What if your distribution is not a source distribution and only provides pyc and pyo files? -- Steven D'Aprano

On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano<steve@pearwood.info> wrote:
one and only one way to install a distribution using its setup.py script, among the ways provided by easy_install, distutils and pip. e.g. avoiding having different format/location for the generated .egg.info;
If you choose not to manually manage them, the packaging system you use is installing files in standard places where you expect to find them, and provides tools to know what is installed. Right now we don't have this standard in Python stdlib to interact with the distributions in a Python installation. And since packaging systems that are used out there could use the same standard, since we would like to make distutils a reference package for this kind of need, it seems like a good idea to define this standard in here. then each packaging system is free to implement its features on the top of it, ala wsgiref.
Being able to retrieve the metadata of an installed distribution, or the list of its installed files, no matter which tool installed it.
This problem is solved as described later in the PEP, with the API that allows you to get the list of the distributions that use a given file. (thanks to the RECORD files) If Spam and Ham use smap.py, and if you uninstall Spam, this file will not be removed because the API will tell you its used in both distributions.
Good question, I have never created such distribution. Aren't they read-only files ? What happens if someone change them, do we really want to keep them since it breaks the distribution ? How they can be changed in the first place, without the source ? Tarek

2009/6/30 Guido van Rossum <guido@python.org>:
Looks like the discussion isn't quite over yet... I did read about a third of the PEP but still feel pretty lost about how it all fits together; I won't have time to read more until next week, probably. I made some minor edits for typos and grammar (being PEP editor and all :-). I noted an inconsistency: first you say that the RECORD file uses the excel dialect, but at the end of the same section you say it uses the default csv settings. Sounds like you need to delete one or the other. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Martin v. Löwis wrote:
Agreed. While Guido is highly likely to just accept the distutils-sig consensus on something like this, that doesn't eliminate the need for him to actually *say* that he is approving the PEP on that basis. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Tue, Jun 30, 2009 at 2:32 PM, Nick Coghlan<ncoghlan@gmail.com> wrote:
Sure, I didn't want to bypass the process, I was not really sure about the right move on this PEP since it was based on the summit decisions, I'll wait for Guido decision, thanks. Cheers Tarek -- Tarek Ziadé | http://ziade.org

On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan<ncoghlan@gmail.com> wrote:
Sure. :-) So what *is* the distutils-sig consensus? And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2009/6/30 Guido van Rossum <guido@python.org>:
And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)
My impression (as someone who does read the distutils SIG, but in all honesty has very little practical interest in distutils internals): - It's very focused on distutils internals. If it has an impact on end users (as opposed to packagers), it's very hard to discern. The only hint of such a thing is the mention of an uninstall function. But it's only an API. So still no end user impact [1] :-( - The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters... - It's quite dense for the casual reader not familiar with the terminology. I've never managed to read the whole thing through, personally. I'd suggest two things: - Add a section to the PEP describing the purely end user impact of the changes - Post that to python-list, with a note pointing to the PEP for people who care about distutils details If that gets no feedback, you've done as much as you can. Paul. [1] I'd actually like it if the PEP defined an uninstall command - something like "python -m distutils.uninstall packagename". It can be as minimalist as you like, but I'd like to see it present.

On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore@gmail.com> wrote:
- The terminology and focus feels setuptools-inspired (my apologies if that's not the case). Expect pushback from setuptools haters...
setuptools implemented *needed* features, like a way for developers to browse installed packages. We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so) So I worked in PEP 376 to introduce them. Now if the fact that we want to introduce the good ideas of setuptools into distutils, (problems Phillip resolved) will make people push it back *even* they are good idead, needed features, is something we need to fight against.
I'll try to add a definitions section,
Ok.
If that gets no feedback, you've done as much as you can.
I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit'
it's already there: http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function Regards Tarek -- Tarek Ziadé | http://ziade.org

Tarek Ziadé wrote:
That doesn't imply that setuptools itself is needed. I can browse installed packages with dpkg -l, or "add-and-remove programs".
Assuming "we" have consensus before we start fighting against others. FWIW, I abstain from commenting on PEP 376. I don't need it, but it doesn't seem to hurt having it, especially since "egg-info" already managed to sneak in.
I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit'
Just be patient. Both Python 2.7 and 3.2 are still many months ahead, so there is no urgency (AFAICT). Regards, Martin

Tarek Ziadé wrote:
That (at least as I read it) is a function, not a command. If it is a command, give an example of its use from the command line for us poor "don't want to research" people. If the following works: $ python setup.py uninstall some_package Then explicitly say so for us poor schlubs. --Scott David Daniels Scott.Daniels@Acm.Org

On Jun 30, 2009, at 4:46 PM, Tarek Ziadé wrote:
Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of: $ python setup.py uninstall some_package It could just be: $ python setup.py uninstall Except then you'd need to have a complete distribution kicking around with which to run the "python setup.py uninstall" command just to tell the uninstall command the distribution name you want to uninstall. But then with the other uninstall format you could uninstall any distribution from within any other distribution, which is convenient, but weird ... e.g.: $ cd Spam-1.0/ $ python setup.py uninstall Foo Although even the other version of the command could do weird stuff: $ cd Spam-1.0/ $ python setup.py install $ cd ../Spam-2.0/ $ python setup.py uninstall Which would presumably remove the Spam 1.0 distribution when run from the 2.0 version of it! Or perhaps this command should only allow uninstall to be run from a distribution whose name and version match the one that it was installed from ... I dunno what the right solution is. My two-cents is either to punt and only include an uninstall function as currently proposed, or for only supporting some form of the "python setup.py uninstall" style since I would guess that the most common use-case for uninstall is: user downloads a distribution, runs "python setup.py install", tries out the package, decides they don't like package, then runs "python setup.py uninstall" to restore their python back to it's original state. For doing anything more complex than that, people should be encouraged to use another one of the existing tools for managing their distributions, IMHO.

Kevin Teague wrote:
But for us poor schlubs, we want you brilliant packagers to actually come to a hard decision. If you want approval either admit you have no solution in the PEP (and detail the issues that prevent a decision), provide a minimally acceptable command, or expect that nobody sees the value of what you propose.... --Scott David Daniels Scott.Daniels@Acm.Org

On Tue, 30 Jun 2009 at 20:06, Scott David Daniels wrote:
I haven't read the PEP in detail since it's outside my area of interest and knowledge(*), but my understanding of the goal is that the PEP is providing an _infrastructure_ for system-level package management tools. The uninstall function is part of that infrastructure, but since distutils isn't a package manager itself (it's an install program), distutils as currently organized can't really handle uninstall except as outlined in a section you may have clipped from the above context (ie: when setup.py from the original package is available). A possible implementation at the python/distutils level might be to have a 'pyuninstall' command installed (like pydoc et al are installed) which handles uninstallation. The question is what do the people who do real package management (linux distribution level package management and the equivalent) think? I'm guessing they are happy with just having the function for their package management tools to call when needed, since (I'm hoping) they are part of the distutils sig.... So, if my understanding of the overall goal is correct, it looks to me like the PEP is missing a "motivation" section that talks about system package managers. --David (*) I'll make time to read it anyway soon.

2009/7/1 R. David Murray <rdmurray@bitdance.com>:
Don't forget that the maintainers of the bdist_wininst, bdist_msi and bdist_rpm code *are* the distutils maintainers - so to that extent, the PEP has to say how *those* aspects of package managers are covered. Unless another PEP is accepted saying that support for bdist_xxx is being dropped [1], this part of distutils cannot be ignored.
Possibly. If so, though, it must discuss the above 3 cases which are part of core distutils. Paul. [1] A PEP I plan on strongly opposing!

Paul Moore <p.f.moore@gmail.com> writes:
Actually, your view seems quite reasonable to me: as a GNU user with my advanced dependency-managing package manager, I feel just as strongly that a program that installs is implicitly promising that it can also uninstall cleanly. I think that calling distutils an “install program” is confusing. When I discuss distutils, I don't call it a program at all; it's a library (or perhaps “framework”) that provides part of the job of package installation. -- \ “I think it would be a good idea.” —Mahatma Gandhi (when asked | `\ what he thought of Western civilization) | _o__) | Ben Finney

2009/7/2 Ben Finney <ben+python@benfinney.id.au>:
I agree, but I consider the fact that distutils "provides" python setup.py install means that it "is an install program" in some people's minds. These are the same people who complain about the lack of a distutils uninstall feature. Personally, I want distutils to provide all the individual bits that I get from bdist_wininst, so that I can put together an alternative for specialised situations (notably, non-site installs of some packages). So uninstall and being able to list installed packages are important to me - hence my interest in this PEP. Paul.

Paul Moore <p.f.moore@gmail.com> writes:
Well, I don't suffer from that confusion between the library and a specific program, yet I still want distutils to support uninstall :-) and hence am interested in seeing the fate of this PEP. -- \ “Men never do evil so completely and cheerfully as when they do | `\ it from religious conviction.” —Blaise Pascal (1623–1662), | _o__) Pensées, #894. | Ben Finney

Kevin Teague wrote:
It could be: $ python -m distutils uninstall some_package Asymmetrical with the install of course. Michael

Tarek Ziadé wrote:
A directly executable submodule is an even better idea than making distutils itself executable. Definitely worth mentioning in the PEP in the section on uninstallation support. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Kevin Teague wrote:
I was actually thinking of something more along the lines of: python -m distutils uninstall <whatever> (Older Pythons won't reliably let you execute a package like that, but 3.1+ and 2.7+ let you do it by including a __main__ module in the package) However, having uninstall as a command supported by setup.py also makes a certain amount of sense. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
-1. Where does the setup.py file come from? If I have docutils installed, and want to remove it, must I download the source again so that I can get the setup.py, so I can run the uninstall? This is daft - particularly given that the point of PEP 376 is to ensure that all of the required information is available from the installed package! As I suggested before: python -m distutils.uninstall packagename Calling it a "reference implementation" should not imply that it's not built to be usable and complete. If it's there,m people should be able to use it. Paul.

On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore<p.f.moore@gmail.com> wrote:
yes sorry if it was unclear, I was not thinking about adding something based on setup.py, but just saying that I was going to add this feature in the PEP. and it will be of the form: python -m distutils.uninstall packagename
It will be usable and complete, but very limited. As someone mentioned, it will not take care of dependencies and prevent you from removing a distribution that is mentioned in another distrubution in a setuptools' install_requires metadata. That said, when PEP 345 evolves like we have planned to (adding install_requires in the metadata) Then we should be able to provide this kind of warning with no pain.

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
Thank you. That is precisely the right level for the PEP, IMHO.
That's good, but not something that bothers me too much. (I'm still of the view that dependencies should be handled by documenting them properly - metadata isn't sufficient on its own). Paul.

2009/6/30 Tarek Ziadé <ziade.tarek@gmail.com>:
No problem with any of that. Please understand that on the whole, I'm in favour of the setuptools features. (While I dislike intensely some of the setuptools "ecosystem", such as easy_install. In fact, one of my goals in wanting this PEP accepted is to ensure that I can get the "good bits" of setuptools, without having to buy into the easy_install, dependency management, automated download infrastructure that currently goes with it).
We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so)
"We" in this context denotes the people at the summit. Please remember that people who weren't there still have an opinion - and it may well differ. I'm not saying that it *does* differ, just that the view of the summit should not be viewed as conclusive (unless the way Python is developed is very different from how I understand it).
So I worked in PEP 376 to introduce them.
And as with any PEP, that's a difficult and thankless task (I got a taste myself with PEP 302, and that was far easier), so just for the record, you have my thanks for doing this.
But there *are* issues with setuptools. Some users have mentioned them on a number of occasions. I've raised a few myself. By all means introduce the good ideas into the core - but make sure people agree that the ideas *are* good. And remember - all I said was that people may react against the setuptools-style terminology. Not that they dislike the *idea*, just that they might dislike the way it's presented. After all, one common complaint with setuptools (and one that I agree with) is that its documentation is obscure to the point of being hostile.
Thank you. I'll try to make the time to go through the PEP and comment more fully.
No, that defines an API, which as stated in the PEP, "allows a third-party application to use the uninstall function and make sure it's the only program that can remove a distribution it has previously installed". It does NOT define a standard uninstall command, to complement the standard install command. Paul.

On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Sorry I didn't mean it this way. The summit was just the kickoff for this PEP, but with the strong desire to have a standard + some API for it. Also, notice that I put in "We" also the people that answered the survey I did before the summit. (http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/) I'll make other fixes in the PEP with the feedback you gave, thanks Tarek

2009/6/30 Paul Moore <p.f.moore@gmail.com>:
Thank you. I'll try to make the time to go through the PEP and comment more fully.
OK, I've now read the PEP through in full. My comments follow. I have deliberately avoided mentioning points that others have already raised, to keep things shorter. First of all, after I got over the use of setuptools terminology, the document is very good. It's clear what it's trying to achieve, and covers most of the points I'd want it to. Occasionally the presentation is confusing, see below, but I think everything was there. I do still feel that the setuptools focus will put some readers off. The introductory remarks, in particular, assume a reasonable level of familiarity from readers about setuptools, easy_install, pip and the like. Not having that familiarity isn't a problem in fact, but the assumption is there in the tone. (For example, personally, I hate the unnecessarily cute term "egg", but it's established by now, so that's a lost cause :-() Onto the PEP. "The problem is that many people use easy_install or pip to install their packages..." As already noted by others, it's not clear why this is a problem. But worse is the fact that the paragraph reads to me as saying that easy_install/pip/etc don't follow the *current* standard structure. Given this fact (which *is* a problem!) I fail to see why I should expect them to follow any *new* standard - so how does this PEP actually address the issue? Rather, it seems to say to me that the existing tools have a history of ignoring the standard approach, so this PEP is going to be useless :-( [I expect that isn't what you're trying to say, so you may just need to clarify your meaning. But I do think it's important to address the question of how this PEP is going to ensure that existing tools adapt. In particular, setuptools seems to have completely stagnated, so I see very little likelihood that setuptools is going to change to conform to the PEP - how does that affect things?] ".egg-info becomes a directory" Don't refer to the setuptools documentation (EggFormats)! It is obscure and user-hostile, as well as not actually being the same as the PEP's proposal. Rather, just document the structure of .egg-info. If you want, you can then add a cross-reference note, saying something like "The setuptools structure, as proposed in the EggFormats documentation for that package [ref], is a subset of this standard. In order to conform to this PEP, setuptools will have to be amended to only install .egg-info directories in the format defined by this PEP". "However, it will impact the setuptools and pip projects, but given the fact that..." Confusing. Will these tools need to change (I believe so) or not? If they will need to change, that hardly counts as "no deep consequences" - there's the whole backward compatibility issue for them to handle. "Adding a RECORD file" You say "at installation time" - please clarify. Do you only mean setup.py install? What about bdist_wininst and bdist_msi? What about third party bdist style tools? How will they ensure they get a RECORD file? "The RECORD format" The line separator shouldn't be os-dependent. What value is used for a pure-python (ie, platform independent) package? Unless the file is generated when the install is done, in which case see the previous point... Absolute file paths - this implies that RECORD has to be generated by the installer (which is the only place that knows absolute paths) which means that every bdist_xxx installer has to write its own RECORD file. Does the PEP offer no support for this? In any case, the bdist_wininst and bdist_msi code (which is core distutils) will need to be amended to manage RECORD files appropriately. "Adding an INSTALLER file in the .egg-info directory" Same question again - does the PEP offer supporting APIs to help bdist_xxx installers do this? What about the core bdist_xxx commands? "New APIs in pkgutil" You say "the best place to put these APIS seems to be pkgutil". You should be more definite - "these APIs will be added to the pkgutil module". The Distribution/ZippedDistribution and DistributionDir/ZippedDistributionDir pairs seem to imply that users need to explicitly instantiate these classes (and hence know whether a distribution is zipped). This is cleared up later, but you should add a note here - "Users will not need to create instances of these classes manually, they are returned by the public functions in pkgutil, such as get_distributions()". "DistributionDirMap class" This seems to be an internal implementation detail, and as such should not be documented. It's never returned from any of the public APIs, and the only created instance is a hidden internal global instance. The whole part about purging the cache is also unnecessary - the user has no interface to allow them to do this, so you don't need to document that it's possible. Actually, (Zipped)DistributionDir instances are never returned to the user via the public API, either - so these don't need to be documented. General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain. "Adding an uninstall function" With regard to my earlier comments, my apologies - I misread the text and assumed the documented function simply returned the files to remove, but didn't do the removal itself. (Which makes it far more useless than it actually is). One point - what happens if the uninstall method can't remove a file or directory? Will it leave things in a broken/partially installed state? This needs to be documented (although huge robustness and recovery measures don't need to be taken in a simple reference implementation). This is likely to be a somewhat common issue on Windows in particular, where you can't delete open files. "Installer marker" There are times when I would imagine it would be useful to force uninstallation of the basic package, even if that leaves stuff behind related to a particular installer. A force=True argument would be the easiest, but if you'd rather not add that, it would be useful to document the workaround: inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read() uninstall('docutils', installer=inst) If nothing else, it shows a simple example of how to use the APIs. A few more such examples in the PEP might be nice. But as I said, overall, I'm much happier with the PEP now that I've read it through and taken the time to digest it. Paul.

2009/7/1 Paul Moore <p.f.moore@gmail.com>:
One other thought. You haven't documented the DistributionMetaData class. The And a minor nit: """ Distribution class A new class called Distribution is created with a the path of the .egg-info directory provided to the contructor. It reads the metadata contained in PKG-INFO when it is instanciated. """ instantiated Paul.

2009/7/1 Paul Moore <p.f.moore@gmail.com>:
One other thought. You haven't documented the DistributionMetaData class. The
Just noticed that it's defined in distutils. But it's not documented there :-( Maybe just add a bit to the PEP saying that the class exists in distutils, give a brief summary, and say that as part of implimenting the PEP it will be formally documented. (Even though distutils is a mess of undocumented functionality, I don't think that you can base new work on undocumented internals - you should document them first, or things will never get any better :-() Paul.

Right, the API part is almost empty, http://docs.python.org/distutils/apiref.html I'll complete it for the relevant part. On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore<p.f.moore@gmail.com> wrote:
-- Tarek Ziadé | http://ziade.org

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Right it's unclear, I'll work on this part. To resume : - Phase 1 : introduction of the egg-info file in distutils Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476) alongside distutils-installed package, that contains the metadata of the distribution. - Phase 2: two new formats in the setuptools project Then he created two new formats in the setuptools project: 1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened because other files were added in that directory. 2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files. Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (–single-version-externally-managed *or* --root). - Phase 3: adopting a unified standard. It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.: - This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Set...) The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers. So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard. This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense.
I'll work that way.
I'll add this in a backward compatibility section, as suggested earlier by someone.
These section will have more details about the way they interact with bdist_* for the rest of the mail, I'll clarify the rest of the PEP using your feedback;
ok
The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes. Maybe this should be removed from the PEP and but into another document targeted to developers ?
Good question. I didn't go that far yet because I didn't implement that part in the prototype. Maybe the files could be moved to a temporary place before they are all deleted, allowing to cancel the uninstallation in case a file couldn't be moved.
Ok,
Great thanks for the feedback. Tarek

2009/7/1 Tarek Ziadé <ziade.tarek@gmail.com>:
Ah, that makes a *lot* more sense. So, in a "compatibility" section, you could point out that the proposed standard is compatible with setuptools "single version externally managed" format, and that the setuptools multi-version format (the egg file/directory) is a non-standard format designed to allow multiple versions to be installed at once - something out of scope for this PEP. With that explanation, people like me who glaze over at the complexities of setuptools scenarios should be able to see what's going on. Thanks for clarifying.
Hmm, but you don't explain how they should do that. I certainly wouldn't know how to. As the public API is pkgutil.get_distribution (and the like) which isn't a class so cannot be subclassed, it's not clear how a 3rd party could hook in a subclass of (Zipped)Distribution. And as there's nothing in the public API that consumes these classes, creating them by hand is of no use, either.
Maybe this should be removed from the PEP and but into another document targeted to developers ?
No. If it's needed by developers, it should be defined in the PEP, not elsewhere. But it should be defined in a way that developers can use - not just for the sake of defining it. Unless there is a concrete use case (which should be stated as an example in the PEP) then *not* documenting the classes is of more help to developers, as they don't have to follow a definition that isn't used. Paul.

Tarek Ziadé writes:
On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
That's a judgment you must make. However, Paul's opinion seems to be that it is internal, and not needed by third-parties who are working "on the top" of these classes. If upon consideration you agree, you should take those "details" out of the PEP proper. If you disagree, you should promote them to the "official"/public API. The point of a PEP is not to construct a duck; it is to explain what "quack" means.
Maybe this should be removed from the PEP and but into another document targeted to developers ?
Yes. In the reference implementation (aka prototype), which should have its own documentation as usual. The reference implementation for the PEP shows *how* these things can be done. It must do that, or the PEP has no force. However, the reference implementation need not be "industrial strength"; the actual implementation that eventually goes into the stdlib may very well be an optimized and enhanced version with many data structures and algorithms that differ from the reference implementation. Another general principle: even in the draft PEP, say "is", not "will be". If you're wrong, and that won't work; if it's insufficiently precise; if you find a more elegant way to express the solution; if you simply haven't thought carefully about something yet; if, for whatever reason, you may need to change the PEP, then you change it in the next draft. That's why we have a review cycle, so you can change it. If you yourself have a question, or the draft is incomplete at some point, then you can explain in a note (either a footnote or parentheses). You can even mark a whole section as "speculative" (eg, "nothing in this section has been implemented yet, so everything is subject to change"). But a rule should be stated as a rule, and precisely, so that reviewers can criticize it accurately. If you're uncomfortable saying "is", then maybe that part of the draft isn't ready for public review yet. On the other hand, you can use the reviewers' knowledge here: write "A is B", then as a note "(In some cases A is actually C, and this should be treated specially. But how?)"

On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull<stephen@xemacs.org> wrote:
Yes, while the APIs I have written in the prototype+PEP helped us claryfing what we wanted, I agree they would be better in a second document. They are two target audience, the users of distutils and the builders of package managers, so removing this details from the PEP will also make it easier to read for the first crowd.
Another general principle: even in the draft PEP, say "is", not "will be".
Ok I'll fix that. That's a French stuff : in french, "will be" isn't speculative at all. Thx

Tarek Ziadé wrote:
I don't think "will be" is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, "X will be done." If "X" is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do "X". In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of "will be" in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

Scott Dial wrote:
Someone feel free to correct me if I am incorrect about the desired tone and use of the document..
I don't think we're particularly consistent. For example, I actually ending up adding a note to PEP 343 indicating that people shouldn't read too much into the verb tense in that document because we were working on it for over a year and things that were in the future when we started were well in the past by the time we finished. In general, I agree that using present tense ("is") in specifications and documentation such as PEPs ends up standing the test of time better than using future tense ("will be"). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Since all bdist_* commands are using the install command in a temporary build directory to create a binary distribution, the RECORD file (and the INSTALLER) will be pre-generated then installed alongside the packages/modules. For absolute paths now that gets installed, what would be the difference between the pre-generated RECORD file and the RECORD file installed on the win32 target system, if any ?

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
When is an absolute path generated? If you can give me a small sample of a distribution that installs a file in an absolute path, I'll do some testing on Windows. But the immediate answer is that there are *no* reliable "absolute" paths on Windows, so you're not looking at the likes of /usr/doc, but rather paths that are relative to sys.prefix, but not to the package directory. In that case, the key point is that if the installer is built on a system where Python is installed in a different directory than the system where the installer is run, paths need to be relocated. (E.g., C:\Python27 vs D:\Apps\Python27). Paul.

On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore<p.f.moore@gmail.com> wrote:
Try this setup.py file: """ from distutils.core import setup setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) """ with this MANIFEST.in file: "" include data """ and add a 'data' file alongside If you install it, data is copied in /tmp. If you create a bdist distribution it will be created in the root of the dumb directory which is used to generate the binary distro. Of course you'll have to change '/tmp' to 'c:\tmp' (which makes me realizes that there's no way to force the installation data_files in another drive under windows: the binary distribution will be the same, no matter what drive you use in the absolute path used in data_files I'll add an issue on this)
I get the point: they are three levels we should handle in the RECORD file 1. absolute paths 2. paths relative to sys.prefix or sys.exec_prefix 3. paths relative to the directory where the .egg-info directory is located the RECORD file doesn't handle 2. indeed. For instance you can add a script: setup( .. scripts=['myscript.py'] ..) that will get installed in : sys.prefix + 'Scripts/myscript.py' under win32 for instance So a possible solution is to add 2. in the RECORD files by using a notation such as "$PREFIX" "$EXEC_PREFIX" when we detect that the file is under on of these paths. The query functions will then be able to use sys.prefix and sys.exec_prefix for recompose the absolute pat on the target system
Paul.
-- Tarek Ziadé | http://ziade.org

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
Ta. I'll give it a go tonight. But haven't you made the point yourself by saying I'll need to change the directory? This will fail for me as I don't have a "/tmp" directory. So I'd expect a bdist installer (*any* bdist installer) to fail, as it's got no way to tell where to put that file. I can only imagine this type of thing being done by someone packaging for Unix, with no interest in cross-platform portability (I suspect MAC OS directory names differ from Unix as well). So maybe absolute filenames should be banned in distutils? (OK, less radically, maybe someone should ask where that functionality is used in practice, and based on the answers come up with a more portable approach). Paul.

On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore<p.f.moore@gmail.com> wrote:
It will fail if your path starts with "/" and you are under win32 because there's a converter that will raise an error in that case at installation time. But you can use some code in your setup.py to provide the right directory name when install is called. I don't see it as a problem. I'm just afraid it's impossible to use efficiently under win32 because of the drive letter. But that's rather a bug.
I'll try to see if I can collect that info out of PyPI with a script, to list the real-world usages. Maybe Martin you have a simple way to do this on PyPI ? Tarek

2009/7/2 Tarek Ziadé <ziade.tarek@gmail.com>:
Looks like bdist_wininst doesn't support absolute paths. Which isn't surprising, really. Same with bdist_msi. Actually, with setup.py install:
So how do absolute paths work??? It sure ain't like this! Changing '/tmp' to c:/tmp' gives this: directory name is invalid Changing it to 'c:\\tmp' gives (sort of) success: python setup.py bdist_wininst running bdist_wininst running build installing to build\bdist.win32\wininst running install_data creating build\bdist.win32\wininst\tmp copying data -> build\bdist.win32\wininst\tmp running install_egg_info Creating build\bdist.win32\wininst\PURELIB\ Writing build\bdist.win32\wininst\PURELIB\foo-1.0-py2.6.egg-info creating 'c:\users\gustav\appdata\local\temp\tmpcs9rmi.zip' and adding '.' to it adding 'PURELIB\foo-1.0-py2.6.egg-info' adding 'tmp\data' creating dist removing 'build\bdist.win32\wininst' (and everything under it) The installer packs a *relative* path tmp/data - which isn't right. bdist_msi copies the data file to c:\tmp - when building the msi!. This is very wrong. I can't introspect a msi file to see what ends up in it, but if I install it, no c:\tmp\data is installed, so it does look like it's screwed up. Frankly, handling of absolute paths looks so broken, that I'd explicitly state that it's not supported. I'd raise bugs for some of these things, but to be honest, I'm not at all sure what I'd expect to be correct behaviour (on Windows). In my view absolute paths shouldn't be supported - I can see that might be different on Linux, but I don't see how it's ever going to be cross-platform. Not sure where that leaves things. Absolute path handling seems broken enough (on Windows) that whatever the PEP says isn't much of an issue, as it's not usable anyway. One definite point, though - given the behaviour of bdist_msi, it thought it had installed something that it simply didn't - so there's a serious risk that the distutils uninstall function could uninstall something that the installer never installed - that's far worse than simply deleting the files and messing up the installer metadata. Maybe the answer is that distutils reject *all* uses of absolute paths on Windows. The PEP can then say whatever you want as far as I'm concerned, as it doesn't affect me. If the existing bugs don't get fixed, though, I'd say that the RECORD file should be explicitly disallowed from containing absolute paths on Windows (for safety reasons if nothing else). Paul.

2009/6/30 Guido van Rossum <guido@python.org>:
The consensus is to have one and only one way to install distributions in Python, and a way to retrieve information on installed distributions, and their files.
And is there consensus outside of it? (Remember the ipaddr debacle. It's easy for people to miss an important PEP.)
I am not sure who you are thinking about, I am blogging a lot on the topic and I am trying to get key players involved in this, like os packagers etc. We've built this PEP in respect with existing tools like setuptools, etc, and I am sending mails at python-dev to make sure evereyone involved in python development has a chance to provide some feedback,
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
-- Tarek Ziadé | http://ziade.org

On Wed, 1 Jul 2009 05:19:07 am Tarek Ziadé wrote:
"One and only one way"? Shouldn't that be "one obvious way"? There is a big difference -- the first implies that Python should actively prohibit (somehow!) or at least inhibit any other install mechanisms, while the second implies that there should be a preferred, obvious, but not compulsory, mechanism. If you say "one and only one", and I take you at your word, then I can only assume that you want to actively prevent me from manually dropping .py files into my PYTHONPATH and having them work. I would guess that's probably not what you mean, but that's what it sounds like you're proposing. I don't like guessing -- would you please clarify what you mean? Some questions and comments regarding the PEP: Rationale: http://www.python.org/dev/peps/pep-0376/#id13 "There are too many ways to do it." Why is having multiple ways of adding distributions a problem to be solved? On my Linux system, I can add packages with rpm/yum, or I can compile them from source and manage them myself. I possibly even could (but never have!) install apt-get and use it to manage packages. Another rationale which should be added: There is no standard way of uninstalling distributions. How distributions are installed: http://www.python.org/dev/peps/pep-0376/#id14 "The problem is that many people use easy_install (from the setuptools project [4]) or pip [5] to install their packages, and these third-party tools do not install packages in the same way that Distutils does" Why is that a problem to be solved? Uninstall information: http://www.python.org/dev/peps/pep-0376/#id15 "Under some circumstances, you might not be able to know for sure that you have removed everything, or that you didn't break another distribution by removing a file that is shared among several distributions." I don't see how this proposal will help in the second case. If you install distribution Spam, containing file spam.py, and then install distribution Ham, which requires spam.py, what is to prevent you from removing Spam and breaking Ham? If you don't propose a solution for the dependency problem, you should say so. The RECORD format: http://www.python.org/dev/peps/pep-0376/#id19 "Each record is composed of three elements. ... the MD5 hash of the file, encoded in hex. Notice that pyc and pyo generated files will not have a hash because they are automatically produced from py files." What if your distribution is not a source distribution and only provides pyc and pyo files? -- Steven D'Aprano

On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano<steve@pearwood.info> wrote:
one and only one way to install a distribution using its setup.py script, among the ways provided by easy_install, distutils and pip. e.g. avoiding having different format/location for the generated .egg.info;
If you choose not to manually manage them, the packaging system you use is installing files in standard places where you expect to find them, and provides tools to know what is installed. Right now we don't have this standard in Python stdlib to interact with the distributions in a Python installation. And since packaging systems that are used out there could use the same standard, since we would like to make distutils a reference package for this kind of need, it seems like a good idea to define this standard in here. then each packaging system is free to implement its features on the top of it, ala wsgiref.
Being able to retrieve the metadata of an installed distribution, or the list of its installed files, no matter which tool installed it.
This problem is solved as described later in the PEP, with the API that allows you to get the list of the distributions that use a given file. (thanks to the RECORD files) If Spam and Ham use smap.py, and if you uninstall Spam, this file will not be removed because the API will tell you its used in both distributions.
Good question, I have never created such distribution. Aren't they read-only files ? What happens if someone change them, do we really want to keep them since it breaks the distribution ? How they can be changed in the first place, without the source ? Tarek

2009/6/30 Guido van Rossum <guido@python.org>:
Looks like the discussion isn't quite over yet... I did read about a third of the PEP but still feel pretty lost about how it all fits together; I won't have time to read more until next week, probably. I made some minor edits for typos and grammar (being PEP editor and all :-). I noted an inconsistency: first you say that the RECORD file uses the excel dialect, but at the end of the same section you say it uses the default csv settings. Sounds like you need to delete one or the other. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (15)
-
"Martin v. Löwis"
-
Antoine Pitrou
-
Ben Finney
-
Guido van Rossum
-
Kevin Teague
-
Michael Foord
-
Nick Coghlan
-
Paul Moore
-
R. David Murray
-
Scott David Daniels
-
Scott Dial
-
Stephen J. Turnbull
-
Steven D'Aprano
-
Tarek Ziadé
-
Terry Reedy