Most Windows setup are desktop configured with a single user. I would not be shocked if pip installs modules only for the current user by default. Maybe it could be an option in Python installer (pip system wide or user). Victor Le mercredi 24 septembre 2014, Paul Moore a écrit :

On 24 September 2014 14:16, Mike Miller javascript:;> wrote:

...

It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.

One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt.

Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use).

I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch.

Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org javascript:; https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...

Paul Moore wrote: On 24 September 2014 14:16, Mike Miller wrote:

...

It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.

One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt.

It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning.

Assuming so, that means that if Python is installed there, the standard "pip install XXX" would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use).

This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not. There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite.

I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch.

A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6. Cheers, Steve

Paul

Donald Stufft wrote:

One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?

Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings. I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users. Cheers, Steve

On 09/25/2014 08:43 AM, Donald Stufft wrote:

One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?

There is the "runas" command, though it is a tad harder to use. There is a third-party windows sudo command also that works as you'd expect. -Mike

On 09/25/2014 08:50 AM, Steve Dower wrote:

Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.

There is the runas command, though it could be easier to use. There is also a third-part "sudo" clone.

I don't believe this is the right solution anyway, as very many Windows users won't be able to do this (particularly in IT managed environments). Having 'pip install' do a per-user install automatically is something that will actually work, at the cost/benefit of not affecting other users.

Shoul mention this won't take any choices away from users, such as IT managed environs, they could still install wherever they need to. I understand pip is not actually bundled with Python, just an installer script. That means it is decoupled from changes in the Python installer. Pypa can have pip default to, or fall back to --user later if it decides it is a good idea. -Mike

On Wed, 24 Sep 2014 22:56:20 +0100 Paul Moore wrote:

On 24 September 2014 22:29, Steve Dower wrote:

...

...

In my experience pip --user works just fine also. We use it on our unmanned media players successfully.

This is good to know. Maybe we aren't as far away as we think.

Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.

What do you mean by well-exercised? Regards Antoine.

On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote:

Donald Stufft wrote:

...

One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?

Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.

When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root. ChrisA

On 25 September 2014 02:08, Antoine Pitrou wrote:

...

Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.

What do you mean by well-exercised?

Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. There's probably no way to improve that much other than by making --user the default and waiting for reports of any issues, but feedback like Mike's adds a certain level of confidence that there are no significant problems. Paul

On Thu, 25 Sep 2014 07:34:31 +0100 Paul Moore wrote:

On 25 September 2014 02:08, Antoine Pitrou wrote:

...

...

Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice.

What do you mean by well-exercised?

Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations.

I'm using it often. I'm also unsure how broken it could be. The user site-packages is just another site-packages directory. Regards Antoine.

On 24 September 2014 23:16, Mike Miller wrote:

Hi all,

ProgramFiles was the default in Python 1.X.

It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue.

I've never once encountered a bug due to install to ProgramFiles, or heard of anyone who has, and have been using Python for everything since the year 2000. If any rare bugs happen to surface, they can likely be fixed or replaced with a line of code, or worked around by installing elsewhere.

Right, the problem I was thinking of was actually a transient one during the 3.4 pre-release cycle (the ensurepip integration initially didn't work properly when installing into Program Files). So consider that a switch to +1 from me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 September 2014 16:43, Donald Stufft wrote:

Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not.

I guess that "Using the python.org Python installer on Windows" is a limited enough subset that we probably could check that --user worked in that situation. The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in "Program Files" should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it). Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable. Paul

Chris Angelico wrote:

On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote:

...

Donald Stufft wrote:

...

One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell?

Unfortunately not. The "easy way" is for the executable to declare that it needs administrative privileges, and then the OS will take over and let you approve/reject/sign-in/etc. according to your settings.

When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root.

It's a compile time option (it's specified in the manifest you can embed into an executable) but it is possible to explicitly launch a separate process with the flag. Most programs that look like they can elevate on-demand are really launching a background process to do the work - you can't change the user for a process once it's started in Windows. This process can obviously be the same one you started with (maybe with different command line args), and the main observable difference is that you get fewer access denied errors. So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :) Cheers, Steve

ChrisA

Paul Moore wrote: On 25 September 2014 17:05, Steve Dower wrote:

...

So yes, pip can certainly do this, and if it's already running elevated then it shouldn't reprompt, but it's not entirely trivial to get this right ("are you denied write access to that directory because you're not admin or because it's on read-only media?") and it's considerably easier to try it, fail on access issues, but provide a flag for the user to force elevation. "pip --sudo install ..." would be fine by me :)

I thought one issue with running an elevated command line subprocess from a non-elevated one, was that the elevated one didn't have access to the non-elevated console, so it popped up its own independent console window, which disappeared immediately the process completed (hence losing any error messages). I definitely recall easy_install did that at one stage, and it was a real pain. Or is that something the parent process can affect, and the cmd/easy_install pair just didn't do so?

I'm not sure you can do it automatically, but if you own both sides of the application you can set up a pipe between the two processes and let the unelevated side forward stdio. Again, this isn't trivial to get right. The design for the elevation model seems to have focused mainly on GUI rather than console, probably assuming that people who need to elevate from the console will elevate the shell itself (this is a guess - I have no insight into how the Windows team designed this), so there are limitations we have to work within.

Paul

On 26 Sep 2014 01:56, "Paul Moore" wrote:

Basically, I'd like to hold off moving to "Program Files" as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first "we" in that was "python-dev" and the second was "PyPA". And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable.

This sounds like a reasonable plan to me (I've been keen to have pip do this on the Linux side of things for a while). If the pip team were willing to start down this path, changing the default in 3.5 next year might still be cutting things a little fine. While I think it's worth aiming for that, I also wouldn't see it as a big deal if the change slipped to 3.6 in 2017. Regards, Nick.

Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Sep 25, 2014, at 6:44 PM, Chris Barker wrote:

On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft mailto:donald@stufft.io> wrote: 1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window.

Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window.

2) Switch to —user based on if the user has permission to write to the site-packages or not.

ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like:

""" pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install """

"In the face of ambiguity, refuse the temptation to guess.”

I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else. Immediately switching to --user as a default (outside of a virtual environment) would break two important use cases: - A root/admin user attempting to install into a global site-packages - A user who uses complete Python installations to isolate themselves such as those created by pyenv. It would techincally break anyone relying on the fact that pip will currently throw an error if you attempt to ``pip install`` something without sudo, however I do not consider that to be an actual use case that is going to have wide signifcance. So we get down to how do we not break the two important use cases above. At first I thought about detecting uid 0 (and something similar on Windows?) and turning off the --user default in those cases. However that still left us dealing with something for the second use case. After thinking about it I realized there isn't a good way to determine if something is a user controlled Python installation instead of a system controlled one. This lead me to the realization that something that could be used to "detect" this is whether or not the current user has the ability to write to the site-packages as a means of determining "is my current user allowed to manage that particular Python". Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too). However this particular thing is somewhat offtopic for this list and the particulars of pip moving to --user by default is best discussed on our issue tracker here: https://github.com/pypa/pip/issues/1668. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sep 26, 2014, at 3:09 AM, Paul Moore wrote:

On 26 September 2014 01:38, Donald Stufft wrote:

...

Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too).

A couple of points come to mind (although they may be more suitable for distutils-sig).

1. Do user installs "leak" into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate).

Note that both of these points apply both to venv and virtualenvs.

1. No they don’t leak as far as I’m aware. 2. Yea, I think we throw an error when you use —user inside a virtual environment. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sep 26, 2014, at 9:53 AM, Paul Moore wrote:

On 26 September 2014 14:31, Donald Stufft wrote:

...

Yea, I think we throw an error when you use —user inside a virtual environment.

So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention?

Paul

default = “—user” if not running_inside_virtualenv() else “—system" --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 9/26/2014 1:03 PM, Chris Barker wrote:

On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft mailto:donald@stufft.io> wrote:

...

2) Switch to —user based on if the user has permission to write to the site-packages or not.

ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like:

I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else.

OK -- fine -- I think that history may be important -- at least for Py2, but I agree that --user is a better default.

But what I'm objecting to is is switching the install mode based on the permissions that the user happens to be running with at the time.

None of us should be routinely running as admin. So what I do (and I image a LOT of people do) is try to do whatever I need to do, and only if I get an error to I switch to admin mode (or add sudo, or ???). What I'm suggesting is that folks that simply expect stuff to get installed into the usual system python is not thinking ahead of time "oh, I need be admin for this", but rather, simply do it, and then, if you get a permission error, make the switch to either admin in mode, or add --user.

If we do switch the default to --user, then this issue goes away.

In short -- too much magic is bad.

Pip on Windows should act like a normal Windows program. If I install Python for all users, I expect pipped packages to be installed for all users too, unless I specify otherwise. If installation (for all users) requires admin privileges, I expect a UAC box to pop up and ask for the admin password. This is pretty routine, at least with Win7. Most every program I install does this either on installation or on first running. Some Windows operations also pop up a box. There are only a few things that require that I actually login as an admin user. -- Terry Jan Reedy

On 27 September 2014 06:08, Terry Reedy wrote:

Pip on Windows should act like a normal Windows program. If I install Python for all users, I expect pipped packages to be installed for all users too, unless I specify otherwise. If installation (for all users) requires admin privileges, I expect a UAC box to pop up and ask for the admin password. This is pretty routine, at least with Win7. Most every program I install does this either on installation or on first running. Some Windows operations also pop up a box. There are only a few things that require that I actually login as an admin user.

The main problem is that there is little or no prior art on Windows for *console* programs that require elevation. Those few that do need it require you to start the program from an elevated prompt, but that's not a good user experience. But having said that, I agree with your point, the UAC changes are designed specifically to get people used to the "think about what you're doing" approach, and we should be doing the same. Paul