RFC : PEP 376 - egg.info
Hi, I have reworked the PEP a little bit with people feedback. It needs more feedback : http://svn.python.org/projects/peps/trunk/pep-0376.txt - install/uninstall script I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed. So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features. - MANIFEST (SOURCES.txt) Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP. The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations) What would be the interest of having the list of source files in egg-info ? Cheers Tarek -- Tarek Ziadé | http://ziade.org
On Thu, 30 Apr 2009 09:34:26 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
- install/uninstall script
I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed.
So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features.
Today I just was writing a package-deinstallation routine and doing small tests on the routine... :-) for my gui package manager. There are one or two issues here for you to think about... 1) - distutils already exists on the installed system - how can you install over it? 2) - The new distutils code won't be retrofitted to old systems. As far as I can see, it's just so much better to have a package uninstaller in a seperate gui tool. That can be installed over an existing installation.
- MANIFEST (SOURCES.txt)
Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP.
The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations)
I have done some limited testing on deinstalling packages over the last week. The big problem with using a manifest of installed files is that the setup.py in distutils can generate more files than were originally in the source package. Doing a build... for instance... leaves .o or .obj files. The way to solve the problem is to recursively delete the whole package directory. That gets rid of everything. Intermediate files and all...
What would be the interest of having the list of source files in egg-info ?
Not neccessary - recursively delete the whole package directory or .egg file. David
On Thu, Apr 30, 2009 at 2:11 PM, David Lyon <david.lyon@preisshare.net> wrote:
As far as I can see, it's just so much better to have a package uninstaller in a seperate gui tool. That can be installed over an existing installation.
We are planning to propose a backport for previous versions
The big problem with using a manifest of installed files is that the setup.py in distutils can generate more files than were originally in the source package. Doing a build... for instance... leaves .o or .obj files.
And creates scripts files, etc..
The way to solve the problem is to recursively delete the whole package directory. That gets rid of everything. Intermediate files and all...
What would be the interest of having the list of source files in egg-info ?
Not neccessary - recursively delete the whole package directory or .egg file.
No because you can have files installed anywhere The way to handle de-installation is to use the recorded list of file that can be created by the 'sdist' command at installation time It's the --record option and we want to put its output in "RECORD" in the egg.info Now my point is : do we want to add the MANIFEST (source file list) file as well. Setuptools and pip are adding a SOURCES.txt file at this point. Cheers Tarek
On Thu, 30 Apr 2009 14:23:41 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
We are planning to propose a backport for previous versions
ok
Not neccessary - recursively delete the whole package directory or .egg file.
No because you can have files installed anywhere
That's true that they can be installed anywhere. But there always needs to be an entry in a .PTH file along the python path to specify where the files were installed to.
The way to handle de-installation is to use the recorded list of file that can be created by the 'sdist' command at installation time
It's the --record option and we want to put its output in "RECORD" in the egg.info
Now my point is : do we want to add the MANIFEST (source file list) file as well. Setuptools and pip are adding a SOURCES.txt file at this point.
Well, you can certainly try that... But your logic fails in this case.... - you installed a package on a system in 1998... - that system didn't generate all of the manifest info that was just invented now. (2009) - therefore... the code you run now cannot deinstall that old package. Obviously.. there's ways around it... They're just my observations... David
On Thu, Apr 30, 2009 at 2:35 PM, David Lyon <david.lyon@preisshare.net> wrote:
No because you can have files installed anywhere
That's true that they can be installed anywhere. But there always needs to be an entry in a .PTH file along the python path to specify where the files were installed to.
You don't specify in this pth file that the package "foo" installed the script "bar" in the bin/ directory of your python installation.
The way to handle de-installation is to use the recorded list of file that can be created by the 'sdist' command at installation time
It's the --record option and we want to put its output in "RECORD" in the egg.info
Now my point is : do we want to add the MANIFEST (source file list) file as well. Setuptools and pip are adding a SOURCES.txt file at this point.
Well, you can certainly try that...
But your logic fails in this case....
- you installed a package on a system in 1998...
- that system didn't generate all of the manifest info that was just invented now. (2009)
- therefore... the code you run now cannot deinstall that old package.
There's no plan to provide the uninstall feature for old packages, but to provide a new egg-info standard that can be used on previous Python version. (and therefore the uninstall feature that goes with it)
Obviously.. there's ways around it...
I don't see how, since there were no standard way back then to keep track of installed files Tarek -- Tarek Ziadé | http://ziade.org
On Thu, 30 Apr 2009 14:43:42 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
That's true that they can be installed anywhere. But there always needs to be an entry in a .PTH file along the python path to specify where the files were installed to.
You don't specify in this pth file that the package "foo" installed the script "bar" in the bin/ directory of your python installation.
But during package installation, this information will be written into a .PTH file somewhere along the python path...
There's no plan to provide the uninstall feature for old packages, but to provide a new egg-info standard that can be used on previous Python version. (and therefore the uninstall feature that goes with it)
but if you can't uninstall an older package from the system... then how can you install the newer version with all the new information.. this leaves the user quite stuck...
Obviously.. there's ways around it...
I don't see how, since there were no standard way back then to keep track of installed files
That's my point.... Having a system where new packages can be deinstalled and old packages cannot be deinstalled will be totally confusing for users. It will drive most users crazy - if you think about it. If you have a deinstallation facility, then it must work for new packages as well as old. Otherwise, imho there's just no point. David
David Lyon wrote:
If you have a deinstallation facility, then it must work for new packages as well as old. Otherwise, imho there's just no point.
It is impossible to uninstall a package if you don't have a recording of what was installed. Removing every directories is a bad idea, as it may remove files which were not installed by the package. No packaging system does that. That's actually one of the very few rules which is followed by any installation method, be it windows installer, linux packages, etc... cheers, David
Hi David, On Thu, 30 Apr 2009 22:24:07 +0900, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
It is impossible to uninstall a package if you don't have a recording of what was installed.
Not true.
Removing every directories is a bad idea, as it may remove files which were not installed by the package.
In python, packages go into their own unique directories or .egg files.
No packaging system does that. That's actually one of the very few rules which is followed by any installation method, be it windows installer, linux packages, etc...
I don't disagree with your statement. However, the packaging system you are talking about didn't follow the abovementioned rules when it was designed. Forcing it to "behave" like others when it wasn't designed to work that way in the first place won't give you the desired results. I've been tracing through the old code.... There is much inner beauty in that old code... It seems illogical to me that on the one hand, programmers are told at conferences to delete package directories manually, and then here... people are discouraged from doing it programmatically. Anyway, I'm not going to discuss this with you guys too much more. All I was doing was sharing my experience because I have been debugging the python interpreter lately and seeing how the packaging stuff works. In summary... packages are just directories with an __init__.py file in them. Sometimes they are zipped into eggs. If directories get deleted, the python interpreter doesn't care. It moves right along. (down the python-path list) You can try putting more formality and burdens on the developer when they are trying to build a package (a directory). But somehow it goes against the simplicity of the 'old code'. Don't forget.... the whole point of "python setup.py" is to tell python to go "build a whole lot of stuff".. "and put it away" Now.. what you are saying is that you don't want that paradigm anymore... you want it to be changed to "here are all the files in a nice manifest" paradigm. The "new" paradigm is not pythonic.... It only works when you have all the files/drivers already built and you want them "plonked" in the right places. Anyway... I'm just bouncing feedback to your ideas... Discussing in the spirit of sharing ideas... David
At 10:17 AM 4/30/2009 -0400, David Lyon wrote:
In summary... packages are just directories with an __init__.py file in them. Sometimes they are zipped into eggs.
You are confusing "Python package" with "Python project". Projects are zipped into eggs, and may contain zero or more packages. Packages, conversely, may contain contents from multiple projects. A distribution found on PyPI such as "FooBar-2.79" is the 2.79 release of the *project* "FooBar"... this tells you absolutely nothing about the names of packages or modules contained in that project.
You can try putting more formality and burdens on the developer when they are trying to build a package (a directory). But somehow it goes against the simplicity of the 'old code'.
You don't understand packages, so the rest of your email here makes no sense. The simplicity that you are describing *never actually existed*, even before setuptools and eggs came on the scene.
On Thu, 30 Apr 2009 14:01:31 -0400, "P.J. Eby" <pje@telecommunity.com> wrote:
A distribution found on PyPI such as "FooBar-2.79" is the 2.79 release of the *project* "FooBar"... this tells you absolutely nothing about the names of packages or modules contained in that project.
Ok - but that's hardly my fault...
The simplicity that you are describing *never actually existed*, even before setuptools and eggs came on the scene.
Ok - Sure. So that's why what I am working on is something to make things work *more* simply in the future... Best Regards David
On Thu, Apr 30, 2009 at 3:32 PM, David Lyon <david.lyon@preisshare.net> wrote:
On Thu, 30 Apr 2009 14:43:42 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
That's true that they can be installed anywhere. But there always needs to be an entry in a .PTH file along the python path to specify where the files were installed to.
You don't specify in this pth file that the package "foo" installed the script "bar" in the bin/ directory of your python installation.
But during package installation, this information will be written into a .PTH file somewhere along the python path...
No, you just have a list of relative paths to installed package there that's it.
If you have a deinstallation facility, then it must work for new packages as well as old. Otherwise, imho there's just no point.
See David C. comment. How do you expect such a system to properly install a project that was installed in an unknown manner in the past ? You don't even know what packages were installed for your project unless you digg into the source distribution. For example, if a project installs two packages in your Python, how do you know it ? A project is not a self-contained directory we can remove recursively Cheers Tarek -- Tarek Ziadé | http://ziade.org
Hi Tarek, On Thu, 30 Apr 2009 15:50:19 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
But during package installation, this information will be written into a .PTH file somewhere along the python path...
No, you just have a list of relative paths to installed package there that's it.
It means the same thing.
You don't even know what packages were installed for your project unless you digg into the source distribution.
Yes I do... I have the following code: --code------- import pkg_resources ws = pkg_resources.WorkingSet() for i in ws: s = str(i) print s --end code--- That tells me what packages I have...
For example, if a project installs two packages in your Python, how do you know it ?
The above code will tell me... both packages will appear in the list. David
On Thu, Apr 30, 2009 at 4:34 PM, David Lyon <david.lyon@preisshare.net> wrote:
Hi Tarek,
On Thu, 30 Apr 2009 15:50:19 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
But during package installation, this information will be written into a .PTH file somewhere along the python path...
No, you just have a list of relative paths to installed package there that's it.
It means the same thing.
No, you don't have the exaustive list of the files installed.
You don't even know what packages were installed for your project unless you digg into the source distribution.
Yes I do...
I have the following code:
--code-------
import pkg_resources
ws = pkg_resources.WorkingSet()
for i in ws: s = str(i) print s
--end code---
That doesn't give you a list of installed package for your *project*. Remember that a Distutils project (one setup.py file) can install several packages in your python, and depending on the way you install(ed) it, you might not be abe to get back your packages. Try it yourself, here's such a package: create a dir with a setup.py file : """ from distutils.core import setup setup(name='ok', packages=['one', 'two']) """ now create a "one" subdirectory with a __init__ file in it, same for "two" next, run : $ python setup.py install Now tell me how you know, looking at your Python installation (site-packages), that "ok" installed "one" and "two"'... ;) Tarek -- Tarek Ziadé | http://ziade.org
On Thu, 30 Apr 2009 16:48:32 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
No, you don't have the exaustive list of the files installed.
My testing indicates that pkg_resources works ok.
--code-------
import pkg_resources
ws = pkg_resources.WorkingSet()
for i in ws: s = str(i) print s
--end code---
That doesn't give you a list of installed package for your *project*.
I am writing a package manager gui. I don't have a project. If there are projects or things like that, I tell the user to go get zc.buildout or pip.
Remember that a Distutils project (one setup.py file) can install several packages in your python, and depending on the way you install(ed) it, you might not be abe to get back your packages.
I am driving easy_install.. Seems to work reasonably well in many cases....
Try it yourself, here's such a package:
create a dir with a setup.py file :
""" from distutils.core import setup
setup(name='ok', packages=['one', 'two']) """
now create a "one" subdirectory with a __init__ file in it, same for "two"
next, run :
$ python setup.py install
Now tell me how you know, looking at your Python installation (site-packages), that "ok" installed "one" and "two"'... ;)
That isn't a typical use case in my scenario. A typical user just wants to install package x,y and z from pypi. They want them to go "into python" and it is as simple as that. Those packages will probably end up in site-packages but as likely as not, the user won't care where they go as long as they are available when they are needed. When they are finished, or they don't like them.. they want to deinstall them by clicking the remove button. In what I am doing, the nested directories would have been happily removed. I appreciate your feedback and it is nice discussing these things with you. Best Regards David
On Thu, Apr 30, 2009 at 5:08 PM, David Lyon <david.lyon@preisshare.net> wrote:
A typical user just wants to install package x,y and z from pypi. They want them to go "into python" and it is as simple as that. Those packages will probably end up in site-packages but as likely as not, the user won't care where they go as long as they are available when they are needed.
When they are finished, or they don't like them.. they want to deinstall them by clicking the remove button.
In what I am doing, the nested directories would have been happily removed.
What is your definition of a "package" and a "project" ? If by "package" you mean the packages that are available at PyPI for instance, they may have several packages included in them. If you use easy_install, they will stay nested in the same directory or zip file in your site-packages, but you still may have some files created outside. for instance, a console script generated by setuptools console_script entry point. In your case, you may find everything needed to uninstall though, by looking inside the EGG-INFO directory of the project. Now back to the initial discussion : if some projects are not installed with easy_install you won't have all the info needed to uninstall the various packages and scripts. So basically: we are trying to add in Distutils such a standard, (it differs slighlty from setuptools but it's the same spirit) And then, as we said earlier, it will be impossible to uninstall packages that were previously uninstalled with some other techniques since we won't have the info required. You said it yourself : your own uninstall system works only for packages installed with easy_install, so I guess you agree that your system cannot uninstall other packages properly (I show you an example of a package you couldn't uninstall in my previous mail.) Tarek -- Tarek Ziadé | http://ziade.org
Tarek, On Thu, 30 Apr 2009 17:29:40 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
And then, as we said earlier, it will be impossible to uninstall packages that were previously uninstalled with some other techniques since we won't have the info required.
Let's just accept this as your final position.. I'm led to believe that pip and enstaller both claim deinstallation of packages. It shows that people out there are doing - or at least claiming - deinstallation. Finally, if you know that easy_install has certain failings, then it is wrong to suggest that because I use it, that I am somehow responsible for it's internal faults. Anyway - Have a nice day David
David Lyon wrote:
Let's just accept this as your final position..
I'm led to believe that pip and enstaller both claim deinstallation of packages.
Yes - packages they have installed *themselves*. Nobody has ever claimed that uninstallation is impossible - uninstallation of something which was installed outside your knowledge is. cheers, David
Hi Tarek, On Thu, 30 Apr 2009 16:48:32 +0200, Tarek Ziadé wrote:
--code-------
import pkg_resources
ws = pkg_resources.WorkingSet()
for i in ws: s = str(i) print s
--end code--- ..
No, you just have a list of relative paths to installed package there that's it. It seems you are right... But ... when I check more closely .. I see that the above code doesn't really display all the installed packages in the site-lib directory. There are some directories in there that don't get included in the results. I would love to know why. So I am realising that the above code doesn't give the right answers. I thought if programmers used distutils and so forth everything would work "properly". or is this just an issue with pkg_resources ? Are these failures of the python subsystem? I would love to know... David
At 08:50 AM 5/1/2009 -0400, David Lyon wrote:
It seems you are right... But ... when I check more closely .. I see that the above code doesn't really display all the installed packages in the site-lib directory. There are some directories in there that don't get included in the results. I would love to know why. So I am realising that the above code doesn't give the right answers. I thought if programmers used distutils and so forth everything would work "properly".
The packages have to be installed using either setuptools, or Python 2.5+ distutils, in order for them to be listable via pkg_resources.
On Thu, Apr 30, 2009 at 9:34 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hi,
I have reworked the PEP a little bit with people feedback.
It needs more feedback : http://svn.python.org/projects/peps/trunk/pep-0376.txt
- install/uninstall script
I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed.
So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features.
- MANIFEST (SOURCES.txt)
Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP.
The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations)
What would be the interest of having the list of source files in egg-info ?
There's another thing I have thought of, for the file list in egg-info : make it configurable. So let's: - make the change in the "install_egg_info" command to create the egg-info directory - push the PKG-INFO file into it - provide a new option for the command for people to point other files to add - change the "install" command so it adds the RECORD file in this list With that change, third-party tools will be able to configure "install_egg_info" in order to list extra files to add into the egg-info directory when the package is installed, whitout having to override that command. The option would be a simple list of files path. Each path can be: - a path relative to the root of the project that is being installed, which allows you to provide a file that is present in your distribution tree - an absolute path, The name of each file will have to be normalized: all upper case with no extensions. Any opinions ? Cheers Tarek -- Tarek Ziadé | http://ziade.org
There's another point I was thinking about in PEP 376 What about dropping the 'egg' part in 'PROJECT.egg-info' ? and replace it with 'PROJECT.info' (and make the 2.7 version compatible with PROJECT.egg-info ) I know it's a minor change, but it seems that a lot of people are confused with this, when they don't know what is an egg, etc. Tarek On Thu, Apr 30, 2009 at 9:34 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hi,
I have reworked the PEP a little bit with people feedback.
It needs more feedback : http://svn.python.org/projects/peps/trunk/pep-0376.txt
- install/uninstall script
I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed.
So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features.
- MANIFEST (SOURCES.txt)
Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP.
The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations)
What would be the interest of having the list of source files in egg-info ?
Cheers Tarek
-- Tarek Ziadé | http://ziade.org
-- Tarek Ziadé | http://ziade.org
participants (4)
-
David Cournapeau -
David Lyon -
P.J. Eby -
Tarek Ziadé