Changing the default install location, script versioning (Packages and PEP 370)

So, I've finally been able to kick my python 2.5 boat anchor, and I'm now relishing in the delightful future of 2.6. Two things struck me though, digging into PEP 370's (per user site-packages) behavior. The first is this - when you install a package (say, pip) into the .local directory via the --user command, the package ends up in the correct lib directory; but the binary get dropped into ./local/bin - this seems wrong. The reason being, is that what if I also have python 2.7 (which i do) installed, as well as python 3.1 and the release-which-will-not-be-named (3.0) - if I install that same package into one of the other versions, a new binary would be written with the same name - a script-pythonversion might also be installed, but the fact that the original script was overwritten seems broken to me. I think the "best" fix for this is to make the bin/ directory mirror the lib layout - each version would get it's own bin directory: .local/ bin/ python2.6/ python3.1/ lib/ python2.6/ python3.1/ Of course, doing this begs the question - why have /bin and /lib be top-level directories, instead favoring a layout which looks like: .local/ python2.6/ bin/ lib/ The data and doc directories which some packages might include should follow the same convention, to avoid conflicts in those directories as well. The second behavior is more of a "man I wish for a unicorn" type of thing - I think that --user should be removed, and by removed, I mean made the default action when "python setup.py install" is invoked. Users should need to instead pass in a --global flag to alter the behavior to install into the system's site-packages and bin directories. The reason is simple - installation to the global site is typically a super-user task, and can side-effect all users, it's growing into more of a no-no as more OS vendors grow to rely on the packaged/included versions of python. .002 cents jesse

Jesse Noller <jnoller@...> writes:
A script is a script. What version of Python it runs with shouldn't be your (the user's) concern. Do you have two Apache binaries, one compiled with gcc 3.x and one compiled with gcc 4.x?
-1. The point of .local/bin is that it's (supposedly) standard, so that you have only one path to add to $PATH. Putting scripts in versioned subdirectories totally defeats its purpose.

On Jul 20, 2009, at 4:32 AM, Antoine Pitrou wrote:
That analogy doesn't hold because the compiler isn't selected at runtime. The interpreter *is*, and at least with setuptools the script is hard-wired to run the version of the compiler used to install it (not just 2.6 vs. 2.7 but the actual path to the interpreter is embedded in the script). So if you install one package with 2.6 and another with 2.7, you end up with mingled scripts but they (rightly) don't see the other libraries. That's inconsistent.
+1 because using versioned subdirectories lets me install apps that may only work under one of the versions of Python I have installed on my system. Of course, a possibly better solution would be to have each app in its own directory with its own copy of its dependencies, but virtualenv makes that easy enough. Doug

Doug Hellmann <doug.hellmann@...> writes:
I don't follow the reasoning here. If app A only works with Python version 2.5, install it using Python 2.5, and if app B only works with Python version 2.6, install it using Python 2.6. The shebang line will point to the exact Python version (using setuptools, that is), and the executable's location is obviously orthogonal to selecting the proper Python interpreter. If, on the other hand, for whatever reason you want to install the same application A using two different Python interpreters, I think it's fair to say that it's a very unusual need and that Python shouldn't go out of its way to satisfy it (especially if it worsens the experience for everyone else). (I understand that the few applications which are themselves dedicated to package management, such as easy_install or pip, need some kind of multi-versioning of the executable. But, at least for easy_install, it /already/ comes multi-versioned (easy_install-2.6, etc.) and, besides, this situation is the exception, not the rule)
Yes, so bottom line: if you have very precise and exclusive requirements regarding what ends up where, you might just as well create a virtualenv, or manually override --install rather than require everyone else to switch to a more complicated setup. ~/.local/bin is simple and standard, please don't change that in order to better accomodate a couple of fringe use cases.

On Jul 20, 2009, at 12:18 PM, Antoine Pitrou wrote:
This comes up often enough, with people other than me, that calling it a "fringe case" seems like a mischaracterization. Why is the lib directory ~/.local/lib/pythonVERSION if ~/.local isn't intended to support more than one version of the interpreter? Doug

Doug Hellmann <doug.hellmann@...> writes:
Why is the lib directory ~/.local/lib/pythonVERSION if ~/.local isn't intended to support more than one version of the interpreter?
Because scripts specify which version of the interpreter they use in their shebang line, so they don't need a versioned location. Conversely, libraries have no way to specify for which version of the interpreter they were installed, so their location needs to be versioned. (besides, it is common for a library to be installed for several interpreters because it may be required by several different apps) This is the same difference as between /usr/bin and /usr/lib/python-X.Y/site-packages. Regards Antoine.

On Tue, Jul 21, 2009 at 4:49 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
I'm not dead yet! /monty python I want to point out that the ./local/bin directory isn't the only thing unversioned - any docs/ which get laid down by a package are also being plopped in .local/docs - not .local/version/docs.

On Wed, Jul 22, 2009 at 10:10 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
Because they're artifacts of a package installed into a version of the interpreter. Sure, "proper docs" will include a Python2.x section *and* the 3.x section; then again, why not only package the 3.x docs with the 3.x package?

Jesse Noller <jnoller@...> writes:
That's a good point. Incidentally, data files can be installed in the package's directory if `package_data` is used in the setup script. Which obviously versions them. Regards Antoine.

On Wed, Jul 22, 2009 at 6:10 PM, Antoine Pitrou<solipsis@pitrou.net> wrote:
Yup; and there's a sticking point - I'm more than willing with some of this to chalk this up to "bad package maintainers" - but given the sorry state of python-packaging in general (not disparaging Tarek's efforts) and the general confusion about the "One True Way to Do It", I think we need to do *something* to at least mitigate the issues. That's why I suggested versioning everything into versioned directories to begin with - sure, this adds a slight problem in that you have to do a export "PATH=$PATH:.local/python2.6/bin:local/python3.1/bin" and so on, but I feel that's a small problem compared to the "what version does this script apply to?" confusion. I'm trying to figure out the best way to handle this for people who are just trying to get something done, and we both know 2.x and 3.x are going to be sitting alongside one another for some time. People like those in this thread can easily work around both the $PATH and the ./local/bin issues - I just disable .local and stick with virtualenv, but that doesn't help Jim Bob sitting next to me who is a total newb. jesse

Antoine Pitrou wrote:
Also -1 because the -m switch was added to address exactly this problem of interpreter version specific copies of scripts without needing a proliferation of script files in the system bin directories. That said, does distutils have the equivalent of Python's "make altinstall" command to tell the distribution to install versioned scripts (e.g. easy-install-2.6) without altering non-versioned symlinks (e.g. easy-install)? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Mon, Jul 20, 2009 at 4:29 PM, Nick Coghlan<ncoghlan@gmail.com> wrote:
So python -m "setuptools.commands.easy_install" <args> or python `which easy_install` is a-ok for people? I find it much easier to tell someone "run easy_install" or "run pylint" rather than either one of the previous examples. I mean, if I have two versions of python (say I'm developing for 2.6 and 3.1) and I want my tool chain (pip, virtualenv, pylint, fabric, and so on) to be installed into both, I have two choices: 1: as soon as I install into --user remove the unversioned binary name (for example, pylint) and stick to pylint-2.6. This breaks down if the upstream package doesn't add the versioned name of the script, which mean I need to move it after install. 2. Skip scripts entirely, and execute the module by hand. python -m may have been added to stop the proliferation of script files - but from a pure usability standpoint, as someone who writes libraries which contain executables standpoint, running "myfoo" is much simpler to explain to noobs and easier to remember overall. jesse

2009/7/20 Jesse Noller <jnoller@gmail.com>:
No, but "python -m easy_install" is fine. The fact that setuptools doesn't provide a form designed for use with python -m doesn't mean that there's a problem with the feature, just that setuptools wasn't designed with it in mind. Paul.

On Mon, Jul 20, 2009 at 5:38 PM, Paul Moore<p.f.moore@gmail.com> wrote:
I don't know if easy_install does or doesn't - I simply used it as an example. What I don't parse is that python -m <module> is somehow a replacement for ./script - the logic within a script can do a lot more than just firing off the __main__ of a module. Are we saying that "scripts are considered harmful" and recommend people only support -m for this? There's over 7000 packages, applications and libraries in the cheeseshop right now. A fair number of them would run face first into the non-versioned binary problem. I guess distutils (in a future version) should just deprecate the scripts/entry points options entirely? I really don't think this is an edge case, or should be unsupported. Sure, the same problem exists outside of the .local directory - you run into this installing things into the default system-level site-packages and /usr/bin /usr/local/bin directories, but there's no real reason we can't make this work better in the context of .local jesse

2009/7/20 Jesse Noller <jnoller@gmail.com>:
My perspective is somewhat different, as I'm a Windows user, and many, many cases where scripts are used don't work cross-platform. Note: If you have a pure-python script foo.py, distributed with package bar, why not just put foo.py on sys.path as part of the install, and run it as python -m foo? How is that appreciably harder than putting it in a bin directory? Some cross-platform notes, which may not be obvious to people (if they are obvious to you, my apologies - I don't mean to be patronising): - Many Unix users don't like their scripts to end in .py - Scripts that don't end in .py don't work on Windows - .bat file wrappers on Windows have "odd" properties - You made the point that the #! line is an issue on Unix python -m module is cross-platform, and properly linked to the Python version. It's less transparent than an executable script, but you have to pick a compromise somewhere (and I appreciate that cross-platform isn't always foremost in people's minds). The biggest problem with -m is that not enough packages are designed to support it. I hope I can add a "yet" to that, but it needs more people seeing the benefits (you Unix people, remember us poor Windows users! :-) :-)) if that's to happen. Paul. PS If you really want an executable script, you can always write an alias or one-line shell script to do "python -m foo"... I don't know if that suits your requirement. PPS This is a defence of -m only, I don't have a view on the issue of whether .local/bin should be versioned, as I don't use the feature myself.

On Mon, Jul 20, 2009 at 5:50 PM, Jesse Noller<jnoller@gmail.com> wrote:
I'm personally OK with saying that scripts are no longer directly supported. That means that users will be forced to run scripts like: python -m mypackage.script -- --my-args Is this accessible to new users and Windows users?
This is what scares me about using 'python -m'. I think you would have to do this.
I currently have this issue all over the place. Every time I install software on my Buildbot I have to remember to move the script xyz to xyz$PYVER. I'm running 4 different versions of Python and as you can imagine this issues frustrates me. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek

Jesse Noller wrote:
As you know, Jesse, there is a precedent for this kind of layout: Python framework installs on Mac OS X. For those not familiar with it, the directory layout is something like this: {/System/Library/ | /Library | ~/Library }/Frameworks/Python.framework/ 2.6/ bin/ include/ lib/ share/ 3.1/ bin/ include/ ... where /System/Library/... contains the Apple-supplied Python(s), /Library/... contains the sysadmin-installed set of Pythons (for example, the default for python.org installers) ~/Library/... contains user-installed Pythons. So, not only do framework installs support multiple versions at the user level, they do so at the system and vendor level as well, and support multiple versions of essentially all Python-installed objects (i.e. shared libraries, executables, scripts, etc). (A side note: with OS X Python frameworks builds, ~/.local really isn't needed since ~/Library/Frameworks/... already provides its functionality and more.) As with any scheme supporting multiple versions, a downside to this is potential user confusion over how to manage access to these multiple versions. For those who don't regularly follow Python on OS X matters, this continues to be the source of some of the most frequent support questions. Those who understand the concept of search $PATH can figure out how to deal with it a lot easier than those who don't. As a convenience, the python.org 2.x and 3.1 installers by default also install versioned symbolic links in /usr/local/bin to the standard python command line objects in their respective framework bin directories., i.e. pythonx.x, pydocx.x, idlex.x. Something similar could be done automatically for distutil-installed scripts, of course. -- Ned Deily, nad@acm.org

On Mon, Jul 20, 2009 at 14:50, Jesse Noller <jnoller@gmail.com> wrote:
Taking Paul's follow-up email to this into account, I think we should definitely encourage people support runpy, but that doesn't do away with the usefulness of scripts. Ignoring the convenience factor of having short script names (``hg tip`` is much shorter than ``python2.6 -m mercurial -- tip``), there is also having to remember which interpreter you installed a tool under. Did I install hg under my Python 2.3, 2.4, 2.5 or 2.6 interpreter? I actually had to check the hg script to find that out. And there is also the situation where people are not targeting multiple operating systems and thus have a use for the features a script gives. If I am writing some Linux app in Python I might have some need to run something under bash. So while I totally support pushing people to start using runpy now that __main__.py support exists for packages, I don't think we can just fluff off script support. -Brett

2009/7/21 Brett Cannon <brett@python.org>:
Just to clarify - my personal view ("tainted" by Windows experience, certainly) is that *applications* (such as Mercurial) should be built as isolated, standalone executables, bundled with something like py2exe. I fully appreciate that this is the precise opposite of common practice on Linux (see below for some uninformed speculation on that). I see runpy as being appropriate for the types of small "supporting" scripts that come with general-use packages - things like Twisted's twistd, Nose's nosetests, etc. Essentially, where the package is the key thing, and the command is a related utility. When the command is the main focus, I see that as an "application". I don't know the right answer for "applications" under Linux, but I don't see any reason why a wrapper shell script wouldn't work - it's no less portable than a py2exe built executable. Installing a wrapper is then the responsibility of the packaging system, rather than Python. But as I say, my expertise is basically zero here. Paul.

Jesse Noller <jnoller@...> writes:
I don't think so. I interpret Paul's suggestion as "if you need a multi-versioned script, then you can use the 'python -m' feature instead". But you can continue running scripts in the other cases.
Well, that's your contention actually :) My contention is that a /very small number of them/ would run into the non-versioned binary problem, because only applications which are themselves used for deployment (easy_install, pip, etc.) may need to be invoked with an explicit version of the interpreter. Once again, I don't understand why you care which version of the interpreter random application X works with (for example Mercurial, as in Brett's message). /Apart/ from easy_install, pip and friends, that is.
Certainly not. Perhaps it could grow an option to generate versioned scripts easily, however. By the way, easy_install already comes versioned on the systems I work on (you can run `easy_install-2.6` explicitly, for example).
The real reason, IMO and as I've said, is that it makes life harder for people who don't care.

On Sun, Jul 19, 2009 at 6:57 PM, Jesse Noller<jnoller@gmail.com> wrote:
In addition to what you've mentioned here and in your blog post, the ~/.local directory is not well thought out when it comes to binary extension modules and It ignores the needs for multi-architecture systems that do not use fat binaries such as all x86_64 Linux/BSD/Solaris systems. It also ignores the fact that ~ may be shared across machines which are not the same architecture. Compiled modules .so, .dll, .whateveryouroses need to live in a subdirectory below the pythonX.Y directory unique to the OS + architecture they were compiled for. Even that still won't be perfect for shared home directories as it doesn't take into account which OS distro version (read: external shared library dependencies) an extension was compiled for but its a good start: .local/ python2.6/ linux-x86_64/ linux-i386/ linux-mipsel/ darwin-fatmultiarchbinaryorwhateverapplecallsthem/ darwin-x86_64/ netbsd-ppc/
+10 - Unicorns for everyone! The system python belongs to the OS and should -never- be touched by anything that isn't the OSs own package management system. People violate this all the time. That is the path to the dark side. Doing that on any system I'm near leads to anger. Anger leads to hate. Hate leads to suffering and revocation of root privileges. -gps

Jesse Noller <jnoller@...> writes:
A script is a script. What version of Python it runs with shouldn't be your (the user's) concern. Do you have two Apache binaries, one compiled with gcc 3.x and one compiled with gcc 4.x?
-1. The point of .local/bin is that it's (supposedly) standard, so that you have only one path to add to $PATH. Putting scripts in versioned subdirectories totally defeats its purpose.

On Jul 20, 2009, at 4:32 AM, Antoine Pitrou wrote:
That analogy doesn't hold because the compiler isn't selected at runtime. The interpreter *is*, and at least with setuptools the script is hard-wired to run the version of the compiler used to install it (not just 2.6 vs. 2.7 but the actual path to the interpreter is embedded in the script). So if you install one package with 2.6 and another with 2.7, you end up with mingled scripts but they (rightly) don't see the other libraries. That's inconsistent.
+1 because using versioned subdirectories lets me install apps that may only work under one of the versions of Python I have installed on my system. Of course, a possibly better solution would be to have each app in its own directory with its own copy of its dependencies, but virtualenv makes that easy enough. Doug

Doug Hellmann <doug.hellmann@...> writes:
I don't follow the reasoning here. If app A only works with Python version 2.5, install it using Python 2.5, and if app B only works with Python version 2.6, install it using Python 2.6. The shebang line will point to the exact Python version (using setuptools, that is), and the executable's location is obviously orthogonal to selecting the proper Python interpreter. If, on the other hand, for whatever reason you want to install the same application A using two different Python interpreters, I think it's fair to say that it's a very unusual need and that Python shouldn't go out of its way to satisfy it (especially if it worsens the experience for everyone else). (I understand that the few applications which are themselves dedicated to package management, such as easy_install or pip, need some kind of multi-versioning of the executable. But, at least for easy_install, it /already/ comes multi-versioned (easy_install-2.6, etc.) and, besides, this situation is the exception, not the rule)
Yes, so bottom line: if you have very precise and exclusive requirements regarding what ends up where, you might just as well create a virtualenv, or manually override --install rather than require everyone else to switch to a more complicated setup. ~/.local/bin is simple and standard, please don't change that in order to better accomodate a couple of fringe use cases.

On Jul 20, 2009, at 12:18 PM, Antoine Pitrou wrote:
This comes up often enough, with people other than me, that calling it a "fringe case" seems like a mischaracterization. Why is the lib directory ~/.local/lib/pythonVERSION if ~/.local isn't intended to support more than one version of the interpreter? Doug

Doug Hellmann <doug.hellmann@...> writes:
Why is the lib directory ~/.local/lib/pythonVERSION if ~/.local isn't intended to support more than one version of the interpreter?
Because scripts specify which version of the interpreter they use in their shebang line, so they don't need a versioned location. Conversely, libraries have no way to specify for which version of the interpreter they were installed, so their location needs to be versioned. (besides, it is common for a library to be installed for several interpreters because it may be required by several different apps) This is the same difference as between /usr/bin and /usr/lib/python-X.Y/site-packages. Regards Antoine.

On Tue, Jul 21, 2009 at 4:49 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
I'm not dead yet! /monty python I want to point out that the ./local/bin directory isn't the only thing unversioned - any docs/ which get laid down by a package are also being plopped in .local/docs - not .local/version/docs.

Jesse Noller <jnoller@...> writes:
Is it a problem? I understand the docs could be versioned by package version, but why should they be versioned by interpreter version?

On Wed, Jul 22, 2009 at 10:10 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
Because they're artifacts of a package installed into a version of the interpreter. Sure, "proper docs" will include a Python2.x section *and* the 3.x section; then again, why not only package the 3.x docs with the 3.x package?

Jesse Noller <jnoller@...> writes:
That's a good point. Incidentally, data files can be installed in the package's directory if `package_data` is used in the setup script. Which obviously versions them. Regards Antoine.

On Wed, Jul 22, 2009 at 6:10 PM, Antoine Pitrou<solipsis@pitrou.net> wrote:
Yup; and there's a sticking point - I'm more than willing with some of this to chalk this up to "bad package maintainers" - but given the sorry state of python-packaging in general (not disparaging Tarek's efforts) and the general confusion about the "One True Way to Do It", I think we need to do *something* to at least mitigate the issues. That's why I suggested versioning everything into versioned directories to begin with - sure, this adds a slight problem in that you have to do a export "PATH=$PATH:.local/python2.6/bin:local/python3.1/bin" and so on, but I feel that's a small problem compared to the "what version does this script apply to?" confusion. I'm trying to figure out the best way to handle this for people who are just trying to get something done, and we both know 2.x and 3.x are going to be sitting alongside one another for some time. People like those in this thread can easily work around both the $PATH and the ./local/bin issues - I just disable .local and stick with virtualenv, but that doesn't help Jim Bob sitting next to me who is a total newb. jesse

Antoine Pitrou wrote:
Also -1 because the -m switch was added to address exactly this problem of interpreter version specific copies of scripts without needing a proliferation of script files in the system bin directories. That said, does distutils have the equivalent of Python's "make altinstall" command to tell the distribution to install versioned scripts (e.g. easy-install-2.6) without altering non-versioned symlinks (e.g. easy-install)? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Mon, Jul 20, 2009 at 4:29 PM, Nick Coghlan<ncoghlan@gmail.com> wrote:
So python -m "setuptools.commands.easy_install" <args> or python `which easy_install` is a-ok for people? I find it much easier to tell someone "run easy_install" or "run pylint" rather than either one of the previous examples. I mean, if I have two versions of python (say I'm developing for 2.6 and 3.1) and I want my tool chain (pip, virtualenv, pylint, fabric, and so on) to be installed into both, I have two choices: 1: as soon as I install into --user remove the unversioned binary name (for example, pylint) and stick to pylint-2.6. This breaks down if the upstream package doesn't add the versioned name of the script, which mean I need to move it after install. 2. Skip scripts entirely, and execute the module by hand. python -m may have been added to stop the proliferation of script files - but from a pure usability standpoint, as someone who writes libraries which contain executables standpoint, running "myfoo" is much simpler to explain to noobs and easier to remember overall. jesse

2009/7/20 Jesse Noller <jnoller@gmail.com>:
No, but "python -m easy_install" is fine. The fact that setuptools doesn't provide a form designed for use with python -m doesn't mean that there's a problem with the feature, just that setuptools wasn't designed with it in mind. Paul.

On Mon, Jul 20, 2009 at 5:38 PM, Paul Moore<p.f.moore@gmail.com> wrote:
I don't know if easy_install does or doesn't - I simply used it as an example. What I don't parse is that python -m <module> is somehow a replacement for ./script - the logic within a script can do a lot more than just firing off the __main__ of a module. Are we saying that "scripts are considered harmful" and recommend people only support -m for this? There's over 7000 packages, applications and libraries in the cheeseshop right now. A fair number of them would run face first into the non-versioned binary problem. I guess distutils (in a future version) should just deprecate the scripts/entry points options entirely? I really don't think this is an edge case, or should be unsupported. Sure, the same problem exists outside of the .local directory - you run into this installing things into the default system-level site-packages and /usr/bin /usr/local/bin directories, but there's no real reason we can't make this work better in the context of .local jesse

2009/7/20 Jesse Noller <jnoller@gmail.com>:
My perspective is somewhat different, as I'm a Windows user, and many, many cases where scripts are used don't work cross-platform. Note: If you have a pure-python script foo.py, distributed with package bar, why not just put foo.py on sys.path as part of the install, and run it as python -m foo? How is that appreciably harder than putting it in a bin directory? Some cross-platform notes, which may not be obvious to people (if they are obvious to you, my apologies - I don't mean to be patronising): - Many Unix users don't like their scripts to end in .py - Scripts that don't end in .py don't work on Windows - .bat file wrappers on Windows have "odd" properties - You made the point that the #! line is an issue on Unix python -m module is cross-platform, and properly linked to the Python version. It's less transparent than an executable script, but you have to pick a compromise somewhere (and I appreciate that cross-platform isn't always foremost in people's minds). The biggest problem with -m is that not enough packages are designed to support it. I hope I can add a "yet" to that, but it needs more people seeing the benefits (you Unix people, remember us poor Windows users! :-) :-)) if that's to happen. Paul. PS If you really want an executable script, you can always write an alias or one-line shell script to do "python -m foo"... I don't know if that suits your requirement. PPS This is a defence of -m only, I don't have a view on the issue of whether .local/bin should be versioned, as I don't use the feature myself.

On Mon, Jul 20, 2009 at 5:50 PM, Jesse Noller<jnoller@gmail.com> wrote:
I'm personally OK with saying that scripts are no longer directly supported. That means that users will be forced to run scripts like: python -m mypackage.script -- --my-args Is this accessible to new users and Windows users?
This is what scares me about using 'python -m'. I think you would have to do this.
I currently have this issue all over the place. Every time I install software on my Buildbot I have to remember to move the script xyz to xyz$PYVER. I'm running 4 different versions of Python and as you can imagine this issues frustrates me. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek

Jesse Noller wrote:
As you know, Jesse, there is a precedent for this kind of layout: Python framework installs on Mac OS X. For those not familiar with it, the directory layout is something like this: {/System/Library/ | /Library | ~/Library }/Frameworks/Python.framework/ 2.6/ bin/ include/ lib/ share/ 3.1/ bin/ include/ ... where /System/Library/... contains the Apple-supplied Python(s), /Library/... contains the sysadmin-installed set of Pythons (for example, the default for python.org installers) ~/Library/... contains user-installed Pythons. So, not only do framework installs support multiple versions at the user level, they do so at the system and vendor level as well, and support multiple versions of essentially all Python-installed objects (i.e. shared libraries, executables, scripts, etc). (A side note: with OS X Python frameworks builds, ~/.local really isn't needed since ~/Library/Frameworks/... already provides its functionality and more.) As with any scheme supporting multiple versions, a downside to this is potential user confusion over how to manage access to these multiple versions. For those who don't regularly follow Python on OS X matters, this continues to be the source of some of the most frequent support questions. Those who understand the concept of search $PATH can figure out how to deal with it a lot easier than those who don't. As a convenience, the python.org 2.x and 3.1 installers by default also install versioned symbolic links in /usr/local/bin to the standard python command line objects in their respective framework bin directories., i.e. pythonx.x, pydocx.x, idlex.x. Something similar could be done automatically for distutil-installed scripts, of course. -- Ned Deily, nad@acm.org

On Mon, Jul 20, 2009 at 14:50, Jesse Noller <jnoller@gmail.com> wrote:
Taking Paul's follow-up email to this into account, I think we should definitely encourage people support runpy, but that doesn't do away with the usefulness of scripts. Ignoring the convenience factor of having short script names (``hg tip`` is much shorter than ``python2.6 -m mercurial -- tip``), there is also having to remember which interpreter you installed a tool under. Did I install hg under my Python 2.3, 2.4, 2.5 or 2.6 interpreter? I actually had to check the hg script to find that out. And there is also the situation where people are not targeting multiple operating systems and thus have a use for the features a script gives. If I am writing some Linux app in Python I might have some need to run something under bash. So while I totally support pushing people to start using runpy now that __main__.py support exists for packages, I don't think we can just fluff off script support. -Brett

2009/7/21 Brett Cannon <brett@python.org>:
Just to clarify - my personal view ("tainted" by Windows experience, certainly) is that *applications* (such as Mercurial) should be built as isolated, standalone executables, bundled with something like py2exe. I fully appreciate that this is the precise opposite of common practice on Linux (see below for some uninformed speculation on that). I see runpy as being appropriate for the types of small "supporting" scripts that come with general-use packages - things like Twisted's twistd, Nose's nosetests, etc. Essentially, where the package is the key thing, and the command is a related utility. When the command is the main focus, I see that as an "application". I don't know the right answer for "applications" under Linux, but I don't see any reason why a wrapper shell script wouldn't work - it's no less portable than a py2exe built executable. Installing a wrapper is then the responsibility of the packaging system, rather than Python. But as I say, my expertise is basically zero here. Paul.

Jesse Noller <jnoller@...> writes:
I don't think so. I interpret Paul's suggestion as "if you need a multi-versioned script, then you can use the 'python -m' feature instead". But you can continue running scripts in the other cases.
Well, that's your contention actually :) My contention is that a /very small number of them/ would run into the non-versioned binary problem, because only applications which are themselves used for deployment (easy_install, pip, etc.) may need to be invoked with an explicit version of the interpreter. Once again, I don't understand why you care which version of the interpreter random application X works with (for example Mercurial, as in Brett's message). /Apart/ from easy_install, pip and friends, that is.
Certainly not. Perhaps it could grow an option to generate versioned scripts easily, however. By the way, easy_install already comes versioned on the systems I work on (you can run `easy_install-2.6` explicitly, for example).
The real reason, IMO and as I've said, is that it makes life harder for people who don't care.

On Sun, Jul 19, 2009 at 6:57 PM, Jesse Noller<jnoller@gmail.com> wrote:
In addition to what you've mentioned here and in your blog post, the ~/.local directory is not well thought out when it comes to binary extension modules and It ignores the needs for multi-architecture systems that do not use fat binaries such as all x86_64 Linux/BSD/Solaris systems. It also ignores the fact that ~ may be shared across machines which are not the same architecture. Compiled modules .so, .dll, .whateveryouroses need to live in a subdirectory below the pythonX.Y directory unique to the OS + architecture they were compiled for. Even that still won't be perfect for shared home directories as it doesn't take into account which OS distro version (read: external shared library dependencies) an extension was compiled for but its a good start: .local/ python2.6/ linux-x86_64/ linux-i386/ linux-mipsel/ darwin-fatmultiarchbinaryorwhateverapplecallsthem/ darwin-x86_64/ netbsd-ppc/
+10 - Unicorns for everyone! The system python belongs to the OS and should -never- be touched by anything that isn't the OSs own package management system. People violate this all the time. That is the path to the dark side. Doing that on any system I'm near leads to anger. Anger leads to hate. Hate leads to suffering and revocation of root privileges. -gps
participants (9)
-
Antoine Pitrou
-
Brett Cannon
-
David Stanek
-
Doug Hellmann
-
Gregory P. Smith
-
Jesse Noller
-
Ned Deily
-
Nick Coghlan
-
Paul Moore