Re: [Distutils] Some clarifications and/or corrections to PEP 376
If you move the files, other things can potentially break, too. For example, any scripts installed will have their shebangs adjusted to point to a particular interpreter, e.g. a venv's interpreter. If they get moved to a different location, the shebang lines may become invalid. I don't believe that we have to support such moving of files outside of the knowledge of packaging tools. Regards, Vinay Sajip
________________________________ From: Kevin Horn <kevin.horn@gmail.com> To: Vinay Sajip <vinay_sajip@yahoo.co.uk> Sent: Monday, 31 December 2012, 19:19 Subject: Re: [Distutils] Some clarifications and/or corrections to PEP 376
On Mon, Dec 31, 2012 at 12:59 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I agree that PEP 376 isn't ideal in what it specifies in this area. The simplest
solution would surely be absolute paths only. What are the downsides apart from the disk space used for the extra lengths of the filenames?
Regards,
Vinay Sajip
If you ever move the files, the absolute paths won't be valid any more. It seems kind of unlikely that this would cause problems, but it might.
-- Kevin Horn
On Mon, Dec 31, 2012 at 2:42 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If you move the files, other things can potentially break, too. For example, any scripts installed will have their shebangs adjusted to point to a particular interpreter, e.g. a venv's interpreter. If they get moved to a different location, the shebang lines may become invalid.
I don't believe that we have to support such moving of files outside of the knowledge of packaging tools.
FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing.
From: PJ Eby <pje@telecommunity.com>
FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing.
Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway. Regards, Vinay Sajip
On Fri, 2013-01-04 at 09:17 +0000, Vinay Sajip wrote:
From: PJ Eby <pje@telecommunity.com>
FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing.
Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway.
I've never really understood the idea that mere installation of a distribution should need to write to completely arbitrary locations on the filesystem. Personally, if the software in my distribution needs things to be put in places that aren't circumscribed by the install machinery (console scripts are an example of things that *are* circumscribed by the install machinery) I'll ship the distribution with a script (often a console script) that people can run that does the needful, but this never runs during installation. - C
On 4 January 2013 09:37, Chris McDonough <chrism@plope.com> wrote:
I've never really understood the idea that mere installation of a distribution should need to write to completely arbitrary locations on the filesystem. Personally, if the software in my distribution needs things to be put in places that aren't circumscribed by the install machinery (console scripts are an example of things that *are* circumscribed by the install machinery) I'll ship the distribution with a script (often a console script) that people can run that does the needful, but this never runs during installation.
I agree - on Windows, I would generally consider a package that installed anything outside of the Python installation to be pretty much broken (it wouldn't work properly in a virtualenv, for example). The problem seems to be that Unix/Linux packaging experts seem to have differing views (in particular around FHS-related concerns). I don't understand these issues well enough to comment, other than to say that even among the Unix community there does not seem to be consensus that there is a need for installation to write outside of site-packages. I do - very occasionally - take a Python installation and move it (for example onto a pen drive). It's no longer a "properly installed" Python, agreed, but for pretty much all practical purposes it still works and I wouldn't like to see that change as a result of absolute paths being mandated in a standard. I'm -1 on absolute paths. Sure, there's an issue with installing files no locations outside the Python installation, but I'd suggest (1) deprecating such usage, and (2) using absolute paths there. An install-time warning might be useful, if it's easy to do. Vinay - does your analysis of PyPI give any indication of what proportion of packages actually use the ability to install to arbitrary locations? I.e., how much of a real world issue is this? Paul.
Absolute path do make some sense in different contexts. For example a server (system wide installed) as administrator I'm expecting my config files to be under /etc. As user ("personal") I'm expecting my config files (for the same application) could be under ~/.somewhere A in-between could be the same application to be system wide installed (*but not configured system wide*) used as user so I'm expecting the config coming from merged between /etc and ~/.somewhere. I'm saying this thinking of a unix box, windows pretty much follows the same pattern, macos is someway different (unles fink/macports aren't taken into account). And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack). So far I haven't seen anything like a graph (uml or just nice) of the different roles explained and discussed: reason why there've been so many failures in rewriting distutils in my humble opinion. *better crawling in the right direction than running in the wrong one* Antonio PS. There's a the subtile (but very important) difference between installation and configuration: so far many fail to see it (until it bites them). Paul Moore wrote:
On 4 January 2013 09:37, Chris McDonough<chrism@plope.com> wrote:
I've never really understood the idea that mere installation of a distribution should need to write to completely arbitrary locations on
I agree - on Windows, I would generally consider a package that installed anything outside of the Python installation to be pretty much broken (it wouldn't work properly in a virtualenv, for example). The problem seems to be that Unix/Linux packaging experts seem to have differing views (in particular around FHS-related concerns). I don't understand these issues well enough to comment, other than to say that even among the Unix community there does not seem to be consensus that there is a need for installation to write outside of site-packages.
I do - very occasionally - take a Python installation and move it (for example onto a pen drive). It's no longer a "properly installed" Python, agreed, but for pretty much all practical purposes it still works and I wouldn't like to see that change as a result of absolute paths being mandated in a standard.
I'm -1 on absolute paths. Sure, there's an issue with installing files no locations outside the Python installation, but I'd suggest (1) deprecating such usage, and (2) using absolute paths there. An install-time warning might be useful, if it's easy to do. Vinay - does your analysis of PyPI give any indication of what proportion of packages actually use the ability to install to arbitrary locations? I.e., how much of a real world issue is this?
Paul. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On 4 January 2013 11:06, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:
And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack).
That's the point, though. In general, distutils installs Python packages, which are *libraries*, not *applications*. Paul
On Fri, 2013-01-04 at 11:43 +0000, Paul Moore wrote:
On 4 January 2013 11:06, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:
And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack).
That's the point, though. In general, distutils installs Python packages, which are *libraries*, not *applications*.
+1. A library might allow you to generate an application after the library itself is installed -- for example, it might include a console script which generates a config file and puts it somewhere -- but that needn't happen at library installation time (and IMO should not); it's a separate issue. - C
At Fri, 4 Jan 2013 11:43:13 +0000, Paul Moore wrote:
On 4 January 2013 11:06, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:
And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack).
That's the point, though. In general, distutils installs Python packages, which are *libraries*, not *applications*.
I've got a lot of python applications on my system and AFAIK those are all installed by distutils / distribute. They are almost all installed using the debian packages, but those debian packages are created using distutils / distribute. I don't think it's useful to the make the distinction and it is not easy to make anyway (django also installs the django-admin command, so it is a library or an application?). Kind regards, Jeroen Dekkers
The distinction is useful in the life cycle context: an "application" could depend on newer/older "libraries" than the one installed on the system. django-admin is an application from this life-cycle point of view. Django is distributed as an "application" that contains its own library (the package django): django-admin is upgraded along the django package. Things would get trickier if some other application would depend on django the package... so upgrading Django would break another application. btw did you notice you've used the word application ".. a lot of python applications on my system .."? Jeroen Dekkers wrote:
At Fri, 4 Jan 2013 11:43:13 +0000, Paul Moore wrote:
And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack). That's the point, though. In general, distutils installs Python
On 4 January 2013 11:06, Antonio Cavallo<a.cavallo@cavallinux.eu> wrote: packages, which are *libraries*, not *applications*.
I've got a lot of python applications on my system and AFAIK those are all installed by distutils / distribute. They are almost all installed using the debian packages, but those debian packages are created using distutils / distribute.
I don't think it's useful to the make the distinction and it is not easy to make anyway (django also installs the django-admin command, so it is a library or an application?).
Kind regards,
Jeroen Dekkers _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
From: Paul Moore <p.f.moore@gmail.com>
I agree - on Windows, I would generally consider a package that installed anything outside of the Python installation to be pretty much broken (it wouldn't work properly in a virtualenv, for example).
Not necessarily - for example, I did some work on virtualenvwrapper-like functionality for PowerShell which required installing stuff in users' PowerShell profile locations. I agree that care has to be taken by the distribution developer to consider use cases carefully, but surely Windows is one environment where installers (in general) put things in many different locations?
The problem seems to be that Unix/Linux packaging experts seem to have differing views (in particular around FHS-related concerns). I don't understand these issues well enough to comment, other than to say that even among the Unix community there does not seem to be consensus that there is a need for installation to write outside of site-packages.
If you do system installations of things, you're expected to put e.g. translations in /usr/share/locale, man pages in /usr/share/man etc. Of course, this doesn't apply when installing to a venv, but then venvs don't have iron-clad conventions about how you install into them. IIUC distutils / distribute allows you to override paths to put things in arbitrary locations.
I'm -1 on absolute paths. Sure, there's an issue with installing files no locations outside the Python installation, but I'd suggest (1) deprecating such usage, and (2) using absolute paths there. An install-time warning might be useful, if it's easy to do. Vinay - does your analysis of PyPI give any indication of what proportion of packages actually use the ability to install to arbitrary locations? I.e., how much of a real world issue is this?
I haven't looked closely into this area regarding existing PyPI distributions, and I'm not sure you can tell actual usage patterns, since you can override paths at installation time using command-line parameters. I do know that other installation systems like Bento offer a lot of options for directory paths, and I certainly don't get the feeling that all those paths are in site-packages. One possible problem I see is that if we don't get enough feedback from a wide enough range of respondents about what actual use cases are, we'll be in the dark when it comes to trying to tie some of this functionality down. I don't know enough about NumPy and SciPy environments and the needs of their users to be sure that we can deprecate features that they might need/already be using. By the way, if you move a distribution after installing it, it is unlikely to stop working just because the record of installed files uses only absolute paths. What that list is used for is uninstallation, which you perhaps don't need if you're moving stuff around the system by hand. If anything does stop working, it's likely to be any scripts which have shebang lines pointing to the original installation locations using absolute paths. IIUC those scripts wouldn't work in PJE's OSAF scenario, either, except perhaps on the machine which was used to install into the now shared location. Regards, Vinay Sajip
I've never really understood the idea that mere installation of a
distribution should need to write to completely arbitrary locations on the filesystem. Personally, if the software in my distribution needs things to be put in places that aren't circumscribed by the install machinery (console scripts are an example of things that *are* circumscribed by the install machinery) I'll ship the distribution with a script (often a console script) that people can run that does the needful, but this never runs during installation.
But that just seems to be deferring the work to a later phase - what happens when you uninstall that distribution? Do you expect the console script you provided to undo what it did when invoked? One example on Windows: PowerShell scripts are looked for by PowerShell in locations other than site-packages. Of course you can ask users to manually tweak environment variables etc. after installation, but I don't see why that's best practice. In general, if a Python library or application is self-contained, I don't disagree with your position. But interoperability with other systems and applications sometimes requires you to be more flexible. Regards, Vinay Sajip
Please consider a user scenario quite common: the use of DESTDIR while planning ahead for path resolution. Linux distro use DESTDIR (and I believe autoconf-related things) uses this env variable to set an alternative "root", so: prefix=/usr sysconfdir=/etc DESTDIR=/foo/bar -> bindir (if DESTDIR is set) = /foo/bar/usr/bin -> bindir (if DESTDIR is *not* set) = /foo/bar/usr/bin -> bindir (if DESTDIR is set) = /foo/bar/etc -> bindir (if DESTDIR is *not* set) = /foo/bar/etc The make install will write only under DESTDIR not requiring root permissions (generally regarded as best practice). Generated rpm can be afterward installed as root (or as required). Regards, Antonio PS. I'm generally against stand alone python "stacks" (isolated environment) because they're really a bad practice in respect to so many parameters (especially in highly regulated sectors). Vinay Sajip wrote:
From: PJ Eby<pje@telecommunity.com>
FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing.
Is this a mainstream use case, though? I'm not dead set against
relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway.
Regards,
Vinay Sajip wrote:
From: PJ Eby<pje@telecommunity.com>
FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing.
Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Fri, Jan 4, 2013 at 4:17 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones),
Actually, we already have a mechanism for that - IIRC, PEP 376 defines various system-managed file categories. So if you specify e.g. that something is a Powershell script, then the installation database should record a path relative to the "powershell root", which is defined elsewhere.
and that wouldn't work well in the OSAF scenario you described, anyway.
Actually, it'd work fine in the case where script paths are relative to "the directory where scripts go", when that directory is determined by the Python instance doing the looking/installing/repairing. Anyway, the point was that there's more than one reason to want relative paths, and as long as they can be relative to things besides the location of the library itself, the other cases can be taken care of. (The OSAF example is also a reason to want the paths to always be /-separated, even on Windows.)
On 1/4/2013 7:11 PM, PJ Eby wrote:
On Fri, Jan 4, 2013 at 4:17 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones),
Actually, we already have a mechanism for that - IIRC, PEP 376 defines various system-managed file categories. So if you specify e.g. that something is a Powershell script, then the installation database should record a path relative to the "powershell root", which is defined elsewhere.
and that wouldn't work well in the OSAF scenario you described, anyway.
Actually, it'd work fine in the case where script paths are relative to "the directory where scripts go", when that directory is determined by the Python instance doing the looking/installing/repairing.
That's true as long as "the directory where scripts go" can't be modified between installation and uninstallation. -- Eric.
participants (7)
-
Antonio Cavallo -
Chris McDonough -
Eric V. Smith -
Jeroen Dekkers -
Paul Moore -
PJ Eby -
Vinay Sajip