Re: [Python-Dev] 3.5 release schedule PEP
On 09/24/2014 10:00 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Subject: Re: [Python-Dev] 3.5 release schedule PEP
On 24 Sep 2014 15:15, "Tim Golden" <mail@timgolden.me.uk> wrote:
On 23/09/2014 18:05, Steve Dower wrote:
I'm also considering/experimenting with installing into "Program Files" by default, but I suspect that isn't going to work out yet.
I'd like to see that go forward: I think it's increasingly difficult to
justify Python's position at c:\pythonxx. But it does run the risk of
breaking All The Things.
It might be better to offer that as a supported option in 3.5, and then make it the default in 3.6.
That will offer a couple of years to work out the issues, rather than a few months.
Hi all, ProgramFiles was the default in Python 1.X. It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue. I've never once encountered a bug due to install to ProgramFiles, or heard of anyone who has, and have been using Python for everything since the year 2000. If any rare bugs happen to surface, they can likely be fixed or replaced with a line of code, or worked around by installing elsewhere. The potential existence of such bugs isn't enough reason to stay stuck in 1990 while leaving users vulnerable to tampering attacks for another few years. -Mike
On 24 September 2014 14:16, Mike Miller <python-dev@mgmiller.net> wrote:
It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt. Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use). I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch. Paul
Most Windows setup are desktop configured with a single user. I would not be shocked if pip installs modules only for the current user by default. Maybe it could be an option in Python installer (pip system wide or user). Victor Le mercredi 24 septembre 2014, Paul Moore <p.f.moore@gmail.com> a écrit :
On 24 September 2014 14:16, Mike Miller <python-dev@mgmiller.net <javascript:;>> wrote:
It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt.
Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use).
I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org <javascript:;> https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
On Wed, 24 Sep 2014 21:32:52 +0200 Victor Stinner <victor.stinner@gmail.com> wrote:
Most Windows setup are desktop configured with a single user. I would not be shocked if pip installs modules only for the current user by default. Maybe it could be an option in Python installer (pip system wide or user).
pip install --user
Paul Moore wrote: On 24 September 2014 14:16, Mike Miller <python-dev@mgmiller.net> wrote:
It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt.
It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning.
Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use).
This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not. There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite.
I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch.
A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6. Cheers, Steve
Paul
On Sep 24, 2014, at 4:24 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Paul Moore wrote: On 24 September 2014 14:16, Mike Miller <python-dev@mgmiller.net> wrote:
It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt.
It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning.
Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use).
This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not.
There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite.
I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch.
A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6.
One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft wrote:
One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?
Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings. I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users. Cheers, Steve
On 09/25/2014 08:50 AM, Steve Dower wrote:
Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.
There is the runas command, though it could be easier to use. There is also a third-part "sudo" clone.
I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users.
Shoul mention this won't take any choices away from users, such as IT managed environs, they could still install wherever they need to. I understand pip is not actually bundled with Python, just an installer script. That means it is decoupled from changes in the Python installer. Pypa can have pip default to, or fall back to --user later if it decides it is a good idea. -Mike
On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Donald Stufft wrote:
One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?
Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.
When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root. ChrisA
On 9/24/2014 6:59 PM, Chris Angelico wrote:
That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? Other programs do it.
Chris Angelico wrote:
On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Donald Stufft wrote:
One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?
Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.
When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root.
It's a compile time option (it's specified in the manifest you can embed into an executable) but it is possible to explicitly launch a separate process with the flag. Most programs that look like they can elevate on-demand are really launching a background process to do the work - you can't change the user for a process once it's started in Windows. This process can obviously be the same one you started with (maybe with different command line args), and the main observable difference is that you get fewer access denied errors. So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :) Cheers, Steve
ChrisA
On 25 September 2014 17:05, Steve Dower <Steve.Dower@microsoft.com> wrote:
So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :)
I thought one issue with running an elevated command line subprocess from a non-elevated one, was that the elevated one didn't have access to the non-elevated console, so it popped up its own independent console window, which disappeared immediately the process completed (hence losing any error messages). I definitely recall easy_install did that at one stage, and it was a real pain. Or is that something the parent process can affect, and the cmd/easy_install pair just didn't do so? Paul
Paul Moore wrote: On 25 September 2014 17:05, Steve Dower <Steve.Dower@microsoft.com> wrote:
So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :)
I thought one issue with running an elevated command line subprocess from a non-elevated one, was that the elevated one didn't have access to the non-elevated console, so it popped up its own independent console window, which disappeared immediately the process completed (hence losing any error messages). I definitely recall easy_install did that at one stage, and it was a real pain. Or is that something the parent process can affect, and the cmd/easy_install pair just didn't do so?
I'm not sure you can do it automatically, but if you own both sides of the application you can set up a pipe between the two processes and let the unelevated side forward stdio. Again, this isn't trivial to get right. The design for the elevation model seems to have focused mainly on GUI rather than console, probably assuming that people who need to elevate from the console will elevate the shell itself (this is a guess - I have no insight into how the Windows team designed this), so there are limitations we have to work within.
Paul
On 25 September 2014 18:13, Steve Dower <Steve.Dower@microsoft.com> wrote:
Again, this isn't trivial to get right. The design for the elevation model seems to have focused mainly on GUI rather than console, probably assuming that people who need to elevate from the console will elevate the shell itself (this is a guess - I have no insight into how the Windows team designed this), so there are limitations we have to work within.
Yeah, I think my view is that we don't really want to have to support code that complex, so let's keep it simple. To use pip to install into a protected location, pip must be run from an elevated shell. Given that doing so is inconvenient (specifically, even more inconvenient than sudo on Unix) let's ensure that for the normal case the user doesn't need to do that. Paul
On 09/25/2014 08:43 AM, Donald Stufft wrote:
One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?
There is the "runas" command, though it is a tad harder to use. There is a third-party windows sudo command also that works as you'd expect. -Mike
Paul Moore wrote:
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows?
Yes, that's the feature that protects from malicious users/code editing "import os" to run "format c:\", spam your address book, or look for credit card numbers, etc. It is the same on Mac and other Unix systems, even Windows since Vista came out, almost 10 years ago.
This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination),
real-life experience, certainly. But I would suggest carefully checking before making the switch.
Windows users know how to elevate now, especially developer types. It should be billed as a "feature for your protection" not something to be feared. Microsoft decided security was worth the pain of changing over its billions of users. Why not Python? In my experience pip --user works just fine also. We use it on our unmanned media players successfully. We also write Windows services, which run under a system account, where it is imperative everything is running from a secure file system.
A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6.
It's already an option! It always has been --> Custom install. I can't remember a time when it didn't work perfectly. -Mike
Mike Miller wrote: Paul Moore wrote:
One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows?
Yes, that's the feature that protects from malicious users/code editing "import os" to run "format c:\", spam your address book, or look for credit card numbers, etc.
It is the same on Mac and other Unix systems, even Windows since Vista came out, almost 10 years ago.
This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination),
real-life experience, certainly. But I would suggest carefully checking before making the switch.
Windows users know how to elevate now, especially developer types.
Sure, but there are plenty of not-quite developer types using Python, plenty using locked down machines, and plenty who simply don't trust arbitrary code when it wants to elevate. The ability to become an admin does not preclude the need to support non-admin users.
It should be billed as a "feature for your protection" not something to be feared. Microsoft decided security was worth the pain of changing over its billions of users. Why not Python?
In my experience pip --user works just fine also. We use it on our unmanned media players successfully.
This is good to know. Maybe we aren't as far away as we think.
We also write Windows services, which run under a system account, where it is imperative everything is running from a secure file system.
A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6.
It's already an option! It always has been --> Custom install. I can't remember a time when it didn't work perfectly.
Agreed. I mean it'll become an obvious option (one of the radio buttons at the start), and eventually the default option. As a slight aside, do you enable the option to compile PYC/PYO files on install? Unless your users are running as admin, you won't get caching on these for the stdlib when it's installed into Program Files. Cheers, Steve
-Mike
On 24 September 2014 22:29, Steve Dower <Steve.Dower@microsoft.com> wrote:
In my experience pip --user works just fine also. We use it on our unmanned media players successfully.
This is good to know. Maybe we aren't as far away as we think.
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice. Reports like this are great for confirming that it's a viable approach. Paul
On Wed, 24 Sep 2014 22:56:20 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 24 September 2014 22:29, Steve Dower <Steve.Dower@microsoft.com> wrote:
In my experience pip --user works just fine also. We use it on our unmanned media players successfully.
This is good to know. Maybe we aren't as far away as we think.
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.
What do you mean by well-exercised? Regards Antoine.
On 25 September 2014 02:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.
What do you mean by well-exercised?
Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. There's probably no way to improve that much other than by making --user the default and waiting for reports of any issues, but feedback like Mike's adds a certain level of confidence that there are no significant problems. Paul
FYI, homebrew's Python uses prefix option, so I can't use `--user`. Is it a bug? $ /usr/local/bin/pip -V pip 1.5.6 from /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg (python 2.7) $ /usr/local/bin/pip install --user tornado ... error: can't combine user with prefix, exec_prefix/home, or install_(plat)base $ cat /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/distutils.cfg [global] verbose=1 [install] force=1 prefix=/usr/local On Thu, Sep 25, 2014 at 3:34 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 25 September 2014 02:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.
What do you mean by well-exercised?
Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. There's probably no way to improve that much other than by making --user the default and waiting for reports of any issues, but feedback like Mike's adds a certain level of confidence that there are no significant problems.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
Le 25/09/2014 09:22, INADA Naoki a écrit :
FYI, homebrew's Python uses prefix option, so I can't use `--user`. Is it a bug?
$ /usr/local/bin/pip -V pip 1.5.6 from /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg (python 2.7)
$ /usr/local/bin/pip install --user tornado ... error: can't combine user with prefix, exec_prefix/home, or install_(plat)base
Yes, it is. http://bugs.python.org/issue22269 Regards Antoine.
On Thu, 25 Sep 2014 07:34:31 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 25 September 2014 02:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.
What do you mean by well-exercised?
Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations.
I'm using it often. I'm also unsure how broken it could be. The user site-packages is just another site-packages directory. Regards Antoine.
On Sep 25, 2014, at 4:54 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 25 Sep 2014 07:34:31 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 25 September 2014 02:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.
What do you mean by well-exercised?
Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations.
I'm using it often. I'm also unsure how broken it could be. The user site-packages is just another site-packages directory.
Broken like the prefix problem :) Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 25 September 2014 16:43, Donald Stufft <donald@stufft.io> wrote:
Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not.
I guess that "Using the python.org Python installer on Windows" is a limited enough subset that we probably could check that --user worked in that situation. The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in "Program Files" should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it). Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable. Paul
On Sep 25, 2014, at 11:54 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 25 September 2014 16:43, Donald Stufft <donald@stufft.io> wrote:
Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not.
I guess that "Using the python.org Python installer on Windows" is a limited enough subset that we probably could check that --user worked in that situation.
The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in "Program Files" should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it).
Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable.
Paul
My thoughts on the pip side has basically always been that pip should either: 1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window. 2) Switch to —user based on if the user has permission to write to the site-packages or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft <donald@stufft.io> wrote:
1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window.
Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window.
2) Switch to —user based on if the user has permission to write to the site-packages or not.
ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like: """ pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install """ "In the face of ambiguity, refuse the temptation to guess." -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Sep 25, 2014, at 6:44 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote: 1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window.
Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window.
2) Switch to —user based on if the user has permission to write to the site-packages or not.
ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like:
""" pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install """
"In the face of ambiguity, refuse the temptation to guess.”
I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else. Immediately switching to --user as a default (outside of a virtual environment) would break two important use cases: - A root/admin user attempting to install into a global site-packages - A user who uses complete Python installations to isolate themselves such as those created by pyenv. It would techincally break anyone relying on the fact that pip will currently throw an error if you attempt to ``pip install`` something without sudo, however I do not consider that to be an actual use case that is going to have wide signifcance. So we get down to how do we not break the two important use cases above. At first I thought about detecting uid 0 (and something similar on Windows?) and turning off the --user default in those cases. However that still left us dealing with something for the second use case. After thinking about it I realized there isn't a good way to determine if something is a user controlled Python installation instead of a system controlled one. This lead me to the realization that something that could be used to "detect" this is whether or not the current user has the ability to write to the site-packages as a means of determining "is my current user allowed to manage that particular Python". Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too). However this particular thing is somewhat offtopic for this list and the particulars of pip moving to --user by default is best discussed on our issue tracker here: https://github.com/pypa/pip/issues/1668. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 26 September 2014 01:38, Donald Stufft <donald@stufft.io> wrote:
Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too).
A couple of points come to mind (although they may be more suitable for distutils-sig). 1. Do user installs "leak" into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate). Note that both of these points apply both to venv and virtualenvs. Paul
On Sep 26, 2014, at 3:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 26 September 2014 01:38, Donald Stufft <donald@stufft.io> wrote:
Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too).
A couple of points come to mind (although they may be more suitable for distutils-sig).
1. Do user installs "leak" into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate).
Note that both of these points apply both to venv and virtualenvs.
1. No they don’t leak as far as I’m aware. 2. Yea, I think we throw an error when you use —user inside a virtual environment. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 26 September 2014 14:31, Donald Stufft <donald@stufft.io> wrote:
Yea, I think we throw an error when you use —user inside a virtual environment.
So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention? Paul
On Sep 26, 2014, at 9:53 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 26 September 2014 14:31, Donald Stufft <donald@stufft.io> wrote:
Yea, I think we throw an error when you use —user inside a virtual environment.
So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention?
Paul
default = “—user” if not running_inside_virtualenv() else “—system" --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft <donald@stufft.io> wrote:
2) Switch to —user based on if the user has permission to write to the
site-packages or not.
ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like:
I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else.
OK -- fine -- I think that history may be important -- at least for Py2, but I agree that --user is a better default. But what I'm objecting to is is switching the install mode based on the permissions that the user happens to be running with at the time. None of us should be routinely running as admin. So what I do (and I image a LOT of people do) is try to do whatever I need to do, and only if I get an error to I switch to admin mode (or add sudo, or ???). What I'm suggesting is that folks that simply expect stuff to get installed into the usual system python is not thinking ahead of time "oh, I need be admin for this", but rather, simply do it, and then, if you get a permission error, make the switch to either admin in mode, or add --user. If we do switch the default to --user, then this issue goes away. In short -- too much magic is bad. All this is also making me think that virtualenv and friends is the real solution to user installed packages anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 9/26/2014 1:03 PM, Chris Barker wrote:
On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote:
2) Switch to —user based on if the user has permission to write to the site-packages or not.
ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like:
I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else.
OK -- fine -- I think that history may be important -- at least for Py2, but I agree that --user is a better default.
But what I'm objecting to is is switching the install mode based on the permissions that the user happens to be running with at the time.
None of us should be routinely running as admin. So what I do (and I image a LOT of people do) is try to do whatever I need to do, and only if I get an error to I switch to admin mode (or add sudo, or ???). What I'm suggesting is that folks that simply expect stuff to get installed into the usual system python is not thinking ahead of time "oh, I need be admin for this", but rather, simply do it, and then, if you get a permission error, make the switch to either admin in mode, or add --user.
If we do switch the default to --user, then this issue goes away.
In short -- too much magic is bad.
Pip on Windows should act like a normal Windows program. If I install Python for all users, I expect pipped packages to be installed for all users too, unless I specify otherwise. If installation (for all users) requires admin privileges, I expect a UAC box to pop up and ask for the admin password. This is pretty routine, at least with Win7. Most every program I install does this either on installation or on first running. Some Windows operations also pop up a box. There are only a few things that require that I actually login as an admin user. -- Terry Jan Reedy
On 27 September 2014 06:08, Terry Reedy <tjreedy@udel.edu> wrote:
Pip on Windows should act like a normal Windows program. If I install Python for all users, I expect pipped packages to be installed for all users too, unless I specify otherwise. If installation (for all users) requires admin privileges, I expect a UAC box to pop up and ask for the admin password. This is pretty routine, at least with Win7. Most every program I install does this either on installation or on first running. Some Windows operations also pop up a box. There are only a few things that require that I actually login as an admin user.
The main problem is that there is little or no prior art on Windows for *console* programs that require elevation. Those few that do need it require you to start the program from an elevated prompt, but that's not a good user experience. But having said that, I agree with your point, the UAC changes are designed specifically to get people used to the "think about what you're doing" approach, and we should be doing the same. Paul
On 27 Sep 2014 14:19, "Chris Barker" <chris.barker@noaa.gov> wrote:
All this is also making me think that virtualenv and friends is the real
solution to user installed packages anyway. The main use case that doesn't cover is system scripting on Linux, where you may want access to all the platform specific libraries. We're well and truly off topic for python-dev at this point, though... Probably best to reboot the thread on distutils-sig, rather than continuing this one. Cheers, Nick.
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 26 Sep 2014 01:56, "Paul Moore" <p.f.moore@gmail.com> wrote:
Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable.
This sounds like a reasonable plan to me (I've been keen to have pip do this on the Linux side of things for a while). If the pip team were willing to start down this path, changing the default in 3.5 next year might still be cutting things a little fine. While I think it's worth aiming for that, I also wouldn't see it as a big deal if the change slipped to 3.6 in 2017. Regards, Nick.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Yes, we enable the compile script. As we require admin rights to install Python and system (not user) modules with pip, the stdlib .pycs do get created under ProgramFiles at install time. There might well be a situation where a system pipped module doesn't get compiled, but to be honest the difference is negligible on modern machines, those needed to run a supported Windows. I suppose this issue could be improved in the future by caching unwritable system .pyc in a user folder, however it feels like a micro-optimization that shouldn't hold up the change. -Mike On 09/25/2014 09:29 AM, Steve Dower wrote:
As a slight aside, do you enable the option to compile PYC/PYO files on install? Unless your users are running as admin, you won't get caching on these for the stdlib when it's installed into Program Files.
Cheers, Steve
-Mike
On 24 September 2014 23:16, Mike Miller <python-dev@mgmiller.net> wrote:
Hi all,
ProgramFiles was the default in Python 1.X.
It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.
I've never once encountered a bug due to install to ProgramFiles, or heard of anyone who has, and have been using Python for everything since the year 2000. If any rare bugs happen to surface, they can likely be fixed or replaced with a line of code, or worked around by installing elsewhere.
Right, the problem I was thinking of was actually a transient one during the 3.4 pre-release cycle (the ensurepip integration initially didn't work properly when installing into Program Files). So consider that a switch to +1 from me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (13)
-
Antoine Pitrou
-
Antoine Pitrou
-
Chris Angelico
-
Chris Barker
-
Donald Stufft
-
Glenn Linderman
-
INADA Naoki
-
Mike Miller
-
Nick Coghlan
-
Paul Moore
-
Steve Dower
-
Terry Reedy
-
Victor Stinner