Re: [Distutils] PEP 376 for Distutils

At 01:33 PM 2/24/2009 +0100, Tarek Ziadé wrote:
Philip wrote:
So, the uninstallation code should simply not remove file(s) that are referenced by more than one manifest in the target directory -- a relatively simple, future-proof safeguard, that doesn't require any specific knowledge of "namespace packages" per se.
Sounds good. Although, it requires scanning the files again which is not optimal, but the last point of this mail might be an idea to adress this.
Uninstallation isn't exactly a performance-critical activity. It's trivial to read targetdir/*/RECORD and build up a dictionary of file->package and package->file links; this is Python, after all. ;-)
2009/2/24 Joachim König <him@online.de>:
An other option could be to put the egg-info dir into the package itself, e.g.
zlib/ __init__.py egg-info/ PKG-INFO MANIFEST RECORD ...
This would require setuptools and pip to change the way they look for the packages,
More precisely, it would require pkg_resources to become ridiculously slow, by massively increasing the number of directories that need to be read at runtime to determine what packages are present.
Indeed. Having an index file would make things a whole lot simpler.
For *whom*? Certainly not for system packaging tools (rpm, deb, et al). A design goal should be to allow system packaging tools to install a static file footprint: i.e., independent files with predefined content, and no post-processing steps. You can't do that with a shared file, which is why setuptools uses a .pth hack to install namespace packages when building packages for rpm et al.
I am wondering then if this is not an evolution of the .pth files.
Although I find that having as many .pth file as we want is not robust. It make things slow down when you have too many of them.
So, I'd be in favor of a new, unique, optimized, index file, with a set of functions located in pkgutil to read and write in it
This index file could also index the namespace information, in order to speed up the work needed to uninstall a package that shares a namespace.
So, .pth files are bad... let's make more? In fact, let's make a new thing that'll have its own, new bugs? So that we can have uninstalls that take only 1/10th of a second instead of 1/2 a second? The standard to beat in this area, I believe, is PEP 262. At the very least, if you're making any major changes in direction from that PEP, the rationale for those differences should be documented. (And consolidated index files are explicitly rejected by that PEP, with good reason.)

On 24 Feb, 2009, at 16:20, P.J. Eby wrote:
Indeed. Having an index file would make things a whole lot simpler.
For *whom*? Certainly not for system packaging tools (rpm, deb, et al).
A design goal should be to allow system packaging tools to install a static file footprint: i.e., independent files with predefined content, and no post-processing steps. You can't do that with a shared file, which is why setuptools uses a .pth hack to install namespace packages when building packages for rpm et al.
What about another interoperability hook for system packages: specify a file that a (system) package manager can include into the egg-info directory (or egg-file) to tell setuptools/pip that this egg is managed by the system and hence shouldn't be removed by setuptools/pip. Which such a file the user of python package tool could be warned if he tries to uninstall an egg that's owned by the system, instead of invoking the wrath of a sysadmin by removing such files. Ronald

Ronald Oussoren wrote:
On 24 Feb, 2009, at 16:20, P.J. Eby wrote:
Indeed. Having an index file would make things a whole lot simpler.
For *whom*? Certainly not for system packaging tools (rpm, deb, et al).
A design goal should be to allow system packaging tools to install a static file footprint: i.e., independent files with predefined content, and no post-processing steps. You can't do that with a shared file, which is why setuptools uses a .pth hack to install namespace packages when building packages for rpm et al.
What about another interoperability hook for system packages: specify a file that a (system) package manager can include into the egg-info directory (or egg-file) to tell setuptools/pip that this egg is managed by the system and hence shouldn't be removed by setuptools/pip.
Which such a file the user of python package tool could be warned if he tries to uninstall an egg that's owned by the system, instead of invoking the wrath of a sysadmin by removing such files.
But that is already implemented via file/dir permissions. By your reasoning, we should also have something which warns users not to install to the system directory. These ideas are a duplication of functionality -- this functionality is implemented by the disabling write permissions of non-sysadmins into system directories. Or do you propose users put some stuff into their system directories not managed by their package managers and other stuff managed by the package managers?

On Feb 24, 2009, at 12:48 PM, Andrew Straw wrote:
But that is already implemented via file/dir permissions.
That's what's beautiful about GNU stow. Look: sudo mkdir /usr/local/stow/grozz sudo chown `whoami` /usr/local/stow/grozz python ./setup.py install --prefix=/usr/local/stow/grozz # LOOK NO SUDO cd /usr/local/stow sudo stow grozz The important point is that the install process doesn't have permission to write into the system, but GNU stow does. GNU stow is more trusted to behave well than the install scripts of the grozz package are, and it is extremely simple and fail-safe -- all it does is make symlinks from /usr/local/x/y -> /usr/local/stow/grozz/x/y . Note that GNU stow can therefore completely and correctly *uninstall* everything that it installed, by examining all of /usr/local looking for symlinks into /usr/local/stow/grozz and removing them. (Therefore it doesn't need a "RECORD" file -- the filesystem itself contains the exact record.) I hope that this new distutils work will make it possible to use GNU stow to manage Python packages (by making it so that installation consists only of *adding* files to the system, not requiring *editing* files such as .pth files). Regards, Zooko --- Tahoe, the Least-Authority Filesystem -- http://allmydata.org store your data: $10/month -- http://allmydata.com/?tracking=zsig

On Tue, Feb 24, 2009 at 11:48:13AM -0800, Andrew Straw wrote:
Ronald Oussoren wrote:
What about another interoperability hook for system packages: specify a file that a (system) package manager can include into the egg-info directory (or egg-file) to tell setuptools/pip that this egg is managed by the system and hence shouldn't be removed by setuptools/pip.
Which such a file the user of python package tool could be warned if he tries to uninstall an egg that's owned by the system, instead of invoking the wrath of a sysadmin by removing such files.
But that is already implemented via file/dir permissions.
Not all systems are managed by experienced sysadmins and on many single user systems `sudo' is all to easy to invoke.
By your reasoning, we should also have something which warns users not to install to the system directory. These ideas are a duplication of functionality -- this functionality is implemented by the disabling write permissions of non-sysadmins into system directories.
As pointed out by PJE simply removing the RECORD file should do the trick.
Or do you propose users put some stuff into their system directories not managed by their package managers and other stuff managed by the package managers?
It's interesting to point out what seems to be planned for Debian: http://lists.debian.org/debian-devel/2009/02/msg00431.html Quoting just the relevant part: """ Local installation path ----------------------- When installing Python modules using distutils, the resulting files end up in the same location wether they are installed by a Debian package, or by a local user or administrator, unless the installation path is overwritten on the command line. Compare this with most software based on autoconf, where an explicit prefix has to be provided for the packaging, while the default install installs into /usr/local. For new Python versions packaged in Debian this will change so that an installation into /usr (not /usr/local) requires an extra option to distutils install command (--install-layout=deb). To avoid breaking the packaging of existing code the distutils install command for 2.4 and 2.5 will just accept this option and ignore it. For the majority of packages we won't see changes in the packaging, provided that the python packaging helpers can find the files in the right location and move it to the expected target path. A second issue raised by developers was the clash of modules and extensions installed by a local python installation (with default prefix /usr/local) with the modules provided by Debian packages (/usr/local/lib/pythonX.Y/site-packages shared by the patched "system" python and the locally installed python. To avoid this clash the directory `site-packages' should be renamed to `dist-packages' in both locations: - /usr/lib/pythonX.Y/dist-packages (installation location for code packaged for Debian) - /usr/local/lib/pythonX.Y/dist-packages (installation location for locally installed code using distutils install without options). The path /usr/lib/pythonX.Y/site-packages is not found on sys.path anymore. About the name: Discussed this with Barry Warsaw and Martin v. Loewis, and we came to the conclusion that using the same directory name for both locations would be the most consistent way. This change should make the request to conditionalize the inclusion of /usr/local/lib/pythonX.Y/site-packages into sys.path obsolete. If needed we can provide a symlink /usr/lib/pythonX.Y/site-packages pointing to dist-packages. """ This should address the concern of sysadmins making mistakes when adding and removing python distributions (packages/modules). In other words, maybe it's not python's problem but the OS distribution's problem. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

At 10:03 PM 2/25/2009 +0000, Floris Bruynooghe wrote:
It's interesting to point out what seems to be planned for Debian: http://lists.debian.org/debian-devel/2009/02/msg00431.html
Quoting just the relevant part:
""" Local installation path -----------------------
When installing Python modules using distutils, the resulting files end up in the same location wether they are installed by a Debian package, or by a local user or administrator, unless the installation path is overwritten on the command line. Compare this with most software based on autoconf, where an explicit prefix has to be provided for the packaging, while the default install installs into /usr/local. For new Python versions packaged in Debian this will change so that an installation into /usr (not /usr/local) requires an extra option to distutils install command (--install-layout=deb). To avoid breaking the packaging of existing code the distutils install command for 2.4 and 2.5 will just accept this option and ignore it.
I wonder why they don't just use the sitewide distutils.cfg file, which would let them configure user-installed packages to go to somewhere else, e.g.: [install] prefix = /usr/local (And of course the build tools could specify different options.)

2009/2/25 P.J. Eby <pje@telecommunity.com>:
At 10:03 PM 2/25/2009 +0000, Floris Bruynooghe wrote:
It's interesting to point out what seems to be planned for Debian: http://lists.debian.org/debian-devel/2009/02/msg00431.html
Quoting just the relevant part:
""" Local installation path -----------------------
When installing Python modules using distutils, the resulting files end up in the same location wether they are installed by a Debian package, or by a local user or administrator, unless the installation path is overwritten on the command line. Compare this with most software based on autoconf, where an explicit prefix has to be provided for the packaging, while the default install installs into /usr/local. For new Python versions packaged in Debian this will change so that an installation into /usr (not /usr/local) requires an extra option to distutils install command (--install-layout=deb). To avoid breaking the packaging of existing code the distutils install command for 2.4 and 2.5 will just accept this option and ignore it.
I wonder why they don't just use the sitewide distutils.cfg file, which would let them configure user-installed packages to go to somewhere else, e.g.:
[install] prefix = /usr/local
(And of course the build tools could specify different options.)
There might be an issue with this while building python itself. I remember having issues failing to build python because I had [install] home=/home/flub in my .pydistutils.cfg. When python is building and installing it's standard library it uses distutils and having that in my configuration file messed everything up and the builds failed (this was both when building the upstream tarball as well as rebuilding the debian package). This is all IIRC however, I'd have to try this again to remind me of the exact issues (thinking about it now this is probably because python's makefile doesn't invoke setup.py with enough options to be sure to overwrite the configuration file). So you'd have to build-conflict against python, which would be annoying at best, but you also need python to build python these days (again IIRC, might be possible to get round this) so you're stuck. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Wed, Feb 25, 2009 at 4:16 PM, P.J. Eby <pje@telecommunity.com> wrote:
I wonder why they don't just use the sitewide distutils.cfg file, which would let them configure user-installed packages to go to somewhere else, e.g.:
[install] prefix = /usr/local
(And of course the build tools could specify different options.)
Some systems do this. I've found this problematic with virtualenv, as people get weird results because they don't realize this configuration file exists. Also there's no way currently in distutils to unset a value, so if something like prefix is given explicitly there's no way to indicate it should be calculated (you just have to give an explicit value for --prefix). -- Ian Bicking | http://blog.ianbicking.org

At 04:45 PM 2/24/2009 +0100, Ronald Oussoren wrote:
What about another interoperability hook for system packages: specify a file that a (system) package manager can include into the egg-info directory (or egg-file) to tell setuptools/pip that this egg is managed by the system and hence shouldn't be removed by setuptools/pip.
Which such a file the user of python package tool could be warned if he tries to uninstall an egg that's owned by the system, instead of invoking the wrath of a sysadmin by removing such files.
The absence of a RECORD file could suffice for that, I think. Any file that isn't positively listed in *exactly* one RECORD file shouldn't be removed. I'm not super-set on that position, just noting that the absence of a listing is sufficient to detect something that "shouldn't" (as opposed to "mustn't") be deleted. The only contradicting use case is the deletion of packages that were installed via an older distutils version.

At 02:57 PM 2/24/2009 -0500, P.J. Eby wrote:
At 04:45 PM 2/24/2009 +0100, Ronald Oussoren wrote:
What about another interoperability hook for system packages: specify a file that a (system) package manager can include into the egg-info directory (or egg-file) to tell setuptools/pip that this egg is managed by the system and hence shouldn't be removed by setuptools/pip.
Which such a file the user of python package tool could be warned if he tries to uninstall an egg that's owned by the system, instead of invoking the wrath of a sysadmin by removing such files.
The absence of a RECORD file could suffice for that, I think. Any file that isn't positively listed in *exactly* one RECORD file shouldn't be removed.
I'm not super-set on that position, just noting that the absence of a listing is sufficient to detect something that "shouldn't" (as opposed to "mustn't") be deleted. The only contradicting use case is the deletion of packages that were installed via an older distutils version.
Hm. This is also relevant for *install* operations... which should not overwrite (without warning) a file that's not listed in any RECORD files, even if it has the exact same content. (As that would mean a later uninstall would silently delete it.)
participants (6)
-
Andrew Straw
-
Floris Bruynooghe
-
Ian Bicking
-
P.J. Eby
-
Ronald Oussoren
-
zooko