On Tue, Sep 13, 2016, at 08:39 PM, Steve Dower wrote:

It would help if you could post the full error output (sanitizing paths if needed). But you may just need to upgrade pip (python -m install -U pip).

I think Ryan may have typed that command at a Python prompt rather than a system command prompt. Unfortunately the distinction often isn't clear in examples, because the experienced developers writing the instructions are used to guessing which commands are Python and which are system commands.

One thing I'd quite like to see Python grow is a standard function to install packages from inside Python. In R, the install.packages() function is the default way to get third party packages, and I think staying in one interactive prompt does make things easier for new users to understand.

Thomas

I think Ryan may have typed that command at a Python prompt rather than

...

a system command prompt. Unfortunately the distinction often isn't clear in examples, because the experienced developers writing the instructions are used to guessing which commands are Python and which are system commands.

indeed -- not a rare error for newbies.

One thing I'd quite like to see Python grow is a standard function to

...

install packages from inside Python.

indeed:

import pip

pip.install('package_name')

would make lots of sense, except:

1) It is going to have to be a different command than the shell command, so the above confusion would still be an issue.

2) are there any technicals reasons this would fail -- i.e. sys.path would have to be re-calculated at least -- any other re-initializaton that would have to be done?

3) permissions could be an issue, you don't necessarily want python to be running with the permissions to write to it's install dir....

In short -- I like the idea, but I'm not sure it'll really help the newbie confusion.

-CHB

In R, the install.packages()

...

function is the default way to get third party packages, and I think staying in one interactive prompt does make things easier for new users to understand.

Ah, I suspect you're right.

I have considered adding a "Python x.y Shell" start menu item that would configure PATH properly for commands like "python" and "pip". Instinctively, it seems this would be more useful than a direct shortcut to the executable, but at the same time I don't want to start competing with all the other app-specific shells out there.

Cheers, Steve

---

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 13 September 2016 at 19:00, Paul Moore p.f.moore@gmail.com wrote:

[...]

I'm not honestly sure how big the "installing while a process is running" issue would be - I did a few simple experiments and couldn't immediately trigger weirdness, but I believe it can happen. And things get significantly worse if we allow upgrades from the Python prompt rather than just installs (e.g., if you have already imported something from the old version and then upgrade).

The only weirdness on install I can imagine would happen is if the installation creates `.pth` files, as those wouldn't be re-read by the currently running Python.

The only package I remember having these was PIL, which can't be installed by pip anyway. Pillow, the PIL fork, doesn't use .pth files.

(easy_install creates and manages .pth files, but if you're using setuptools APIs to install packages from inside a running process then 1. you already have a system to automatically make the packages available in your namespace and 2. you know, or should know, what you're getting into).

Overall, it'd probably be a nice thing to have, but it's nowhere near

as simple as it seems at first glance. Paul

Considering that installing a package and importing a package are two very different things, I don't see why we couldn't handle the simple case of installing the package for the first time with a nice API. Just document the usual caveat emptors.

Cheers,

Leo

On 13Sep2016 1500, Paul Moore wrote:

On 13 September 2016 at 21:12, Thomas Kluyver thomas@kluyver.me.uk wrote:

...

One thing I'd quite like to see Python grow is a standard function to install packages from inside Python.

That's not too hard in principle - pip.main(['install', package]) is basically all you'd need (modulo various design details, and wrapping it in a nice user friendly function). The bigger problems are:

1. pip is an independent module (albeit shipped with Python) and it's

not clear to what extent Python would want a builtin function that depends on a non-stdlib module. But that's not a technical issue. 2. Python's module system isn't really designed around installing new modules while a process is running. There's caching involved, so the effects can be unpredictable. It's difficult to know whether the benefits (avoiding confused users who tried to install from a Python prompt) would outweigh the costs (confused users having installed a module but not being able to see it because the module system didn't notice things had changed).

I'm not honestly sure how big the "installing while a process is running" issue would be - I did a few simple experiments and couldn't immediately trigger weirdness, but I believe it can happen. And things get significantly worse if we allow upgrades from the Python prompt rather than just installs (e.g., if you have already imported something from the old version and then upgrade).

Overall, it'd probably be a nice thing to have, but it's nowhere near as simple as it seems at first glance. Paul

I think it's one of these things where we should suck it up and let the 90% case work fine, then display a big fat warning if anything weird may have happened and let users sort it out themselves.

A simple builtin that tries to run "sys.executable -m pip {args}" in a subprocess and displays "success", "success but you should probably restart Python now", "failed because you don't have pip", "failed and have all the output" is going to make lots of people happy, even if it upsets those who want it to be perfect in every scenario.

Cheers, Steve

Hi,

On Tue, Sep 13, 2016 at 3:55 PM, Donald Stufft donald@stufft.io wrote:

...

On Sep 13, 2016, at 6:41 PM, Steve Dower steve.dower@python.org wrote:

I think it's one of these things where we should suck it up and let the 90% case work fine, then display a big fat warning if anything weird may have happened and let users sort it out themselves.

I am unsure. One of the really egregious and hard to debug weirdness is going to be something like:

...

...

...

import foo.bar # foo, and foo.bar are in sys.modules pip.install(“thing”) # This implicitly upgrades foo import foo.widget # the old foo is in sys.modules, but the new foo.widget.

The 90% case works when it’s *only* pure python and there’s no upgrading/downgrading involved, however you can’t control whether there are going to be upgrades/downgrades or not when dependencies are in play because of ==, >=, >, <, <=, etc.

Another problem, particularly on Windows, is going to be stuff like:

...

...

...

import someclibrary pip.install(“thing”) # Implicitly upgrades someclibrary

In this case, this would explode because Python will have the someclibrary.dll locked and the upgrade will attempt to remove it.

My fears here are that people are going to get really confused when they make state modifications to their Python environment and they don’t see those modifications reflected (because of sys.modules caching or what have you). You can possibly get around that with reload(), but we already know the problems that reload() has. Of course, that doesn’t stop Python from offering reload(), but I think the difference is that reload() is targeted towards people with weird edge cases who can understand the caveats while the hypothetical ``pip.install()`` is targeted towards people who are beginners who are still having problems figuring out the difference between the REPL and their shell, much less the vagaries of Python’s global state and attempting to modify that in a now running Python program.

Perhaps a better idea would be to add some smarts to the REPL (but not to Python itself) that would detect something like:

...

...

...

pip install

And print a better error message that gives a better indication about what’s gone wrong besides a SyntaxError?

I was thinking the same thing. If it could also catch accidental

...

...

python3 -m pip install

commands, that would be even better.

Cheers,

Matthew

On 13Sep2016 1555, Donald Stufft wrote:

...

On Sep 13, 2016, at 6:41 PM, Steve Dower steve.dower@python.org wrote:

I think it's one of these things where we should suck it up and let the 90% case work fine, then display a big fat warning if anything weird may have happened and let users sort it out themselves.

I am unsure. One of the really egregious and hard to debug weirdness is going to be something like:

...

...

...

import foo.bar # foo, and foo.bar are in sys.modules pip.install(“thing”) # This implicitly upgrades foo import foo.widget # the old foo is in sys.modules, but the new foo.widget.

Except my version of this goes more like:

...

...

import foo.bar # foo, and foo.bar are in sys.modules pip.install("thing") # This implicitly upgrades foo

WARNING: Packages were updated. You need to restart Python now.

...

...

import foo.widget # because I happily ignore messages starting

with WARNING

Doesn't matter to me whether it's "safe" to keep going or not - if any files are overwritten at all then show the warning. (This is exactly the argument I was expecting when I said "those who want it to be perfect in every scenario" ;) )

Cheers, Steve

On 14 September 2016 at 21:43, Donald Stufft donald@stufft.io wrote:

...

On Sep 13, 2016, at 7:06 PM, Steve Dower steve.dower@python.org wrote:

This is exactly the argument I was expecting when I said "those who want it to be perfect in every scenario"

FWIW I don’t have a strong feeling on whether we should add it or not. I haven’t been a beginner in a long time and I am not very good at putting myself in their shoes. I legitimately don’t know what would be more or less confusing here, possible weird state (with or without a warning that they might be getting into weird state territory) or an error message telling them to use a real shell and not the Python REPL.

I think they're both confusing, but one of them we can be more certain the user will have the ability to resolve, regardless of their computing background.

Specifically, someone already familiar with the difference between a system shell and the Python REPL isn't likely to make the mistake in question in the first place - "$ pip install name" is clearly a shell command *if* you know what the "$" indicates, while the unqualified "pip install name" used in Warehouse for cross-platform consistency still looks more like a system command than it does a Python expression (it would be a potentially valid function call in some other languages though, most notably MATLAB in the context of this discussion).

For someone that *doesn't* know the difference though, the only thing we can be sure of is that they know how to run a Python REPL, and hence presumably how to stop and restart that REPL. It's also the case that for through-the-browser development, the user may not have system shell access at all, even if they do already know the difference between the two environments.

So if we're able to emit a warning that says "NOTE: Updated package installed for already loaded module '<name>'. Restart Python to load the new version.", then novice users are likely to take us at our word and restart Python. Intermediate users may say "Hah, I know imp.reload(), so I can ignore that warning!" and try to selective reload things. Experienced users more familiar with what a debugging nightmare selective reloading can be with a deep module dependency tree and usage of the "from X import Y" early binding syntax may then end up going back to just restarting their REPL session after installing new things :)

Cheers, Nick.

On 14 September 2016 at 08:55, Donald Stufft donald@stufft.io wrote:

Another problem, particularly on Windows, is going to be stuff like:

...

...

...

import someclibrary pip.install(“thing”) # Implicitly upgrades someclibrary

In this case, this would explode because Python will have the someclibrary.dll locked and the upgrade will attempt to remove it.

My fears here are that people are going to get really confused when they make state modifications to their Python environment and they don’t see those modifications reflected (because of sys.modules caching or what have you). You can possibly get around that with reload(), but we already know the problems that reload() has. Of course, that doesn’t stop Python from offering reload(), but I think the difference is that reload() is targeted towards people with weird edge cases who can understand the caveats while the hypothetical ``pip.install()`` is targeted towards people who are beginners who are still having problems figuring out the difference between the REPL and their shell, much less the vagaries of Python’s global state and attempting to modify that in a now running Python program.

While I'm normally not a fan, I think this is a case where a widely ignored warning is the most sensible option - if people see *any* kind of weirdness after doing a live package install, their first reaction should be to restart their Python environment and see if the problem goes away. In a lot of cases things will work fine (since a lot of modules are well-behaved in this regard), and for those that aren't, a restart is usually a sufficient remedial step.

To further reduce the support footprint, we can explicitly limit this API to supporting installation using the *default* configuration settings only - absolutely no configurability whatsoever other than an optional version constraint on the package being installed.

Since this is a new API, we could also make it only do --user installs when outside a virtual environment.

That said, to fend off requests for increased complexity in the beginner focused `pip.install()` API, it may be necessary to finally define a semi-supported `pip._custom_install()` API that exposes more of the bells and whistles supported by `pip install`. The underscore prefix would clearly indicate the officially unstable nature of the API, and beginners are unlikely to reach for "pip._custom_install()" when "pip.install()" exists.

Perhaps a better idea would be to add some smarts to the REPL (but not to Python itself) that would detect something like:

...

...

...

pip install

And print a better error message that gives a better indication about what’s gone wrong besides a SyntaxError?

I filed http://bugs.python.org/issue28140 for that - we should be able to hook into the same system that gives special error messages for "print message" and "exec code" in Python 3. (The related special casing is actually in the SyntaxError constructor, so this would affect non-interactive code as well, but I think that's fine for this case).

Cheers, Nick.

On 14 September 2016 at 18:02, Paul Moore p.f.moore@gmail.com wrote:

On 13 September 2016 at 23:55, Donald Stufft donald@stufft.io wrote:

...

The 90% case works when it’s *only* pure python and there’s no upgrading/downgrading involved, however you can’t control whether there are going to be upgrades/downgrades or not when dependencies are in play because of ==, >=, >, <, <=, etc.

I would be OK with an invocation that only ever installed and never did an upgrade, as that would avoid the worst situations. Of course, (a) AFAIK pip doesn't have such a method (other than --no-deps) at the moment (dependencies are always upgraded if needed) and (b) there is an ongoing proposal to change "pip install" to always upgrade, and this is the exactly the opposite of what we'd need from the Python prompt.

I understood Steve's proposal to be that we detect cases which do an upgrade rather than a pure install, and in that case report "you should restart your Python session". That's a common[1] pattern on Windows, so should be familiar to Windows users, who I suspect are more likely to hit this issue.

Paul

[1] Common, but not particularly loved - it's the source of the jokes about "your mouse has moved, do you want to restart Windows for this change to take effect?"

Agreed, but if we're able to limit the warning to cases where:

1. The requested package wasn't already present; and 2. Installing it upgraded one or more existing packages

that should eliminate a lot of the warning noise.

We could even enhance the filtering a little further by having pip look at the top level module names in the built wheel(s) and only emit the warning if at least one of those is already present in sys.modules. That way, doing:

>>> pip.install("i_need_this") >>> import i_need_this

in a clean REPL wouldn't emit a warning even if something was upgraded as part of the install step.

Cheers, Nick.

On 14 September 2016 at 22:27, Marius Gedminas marius@gedmin.as wrote:

On Tue, Sep 13, 2016 at 06:55:49PM -0400, Donald Stufft wrote:

...

...

...

...

pip install

And print a better error message that gives a better indication about what’s gone wrong besides a SyntaxError?

This way you could also make

...

...

...

exit

work like people expect it to work, without the ().

That's never going to happen, as aside from some additional builtins, the default REPL will always work the same way as non-interactive code.

Cheers, Nick.

P.S. It's also entirely unrelated to the current discussion, as it's not a SyntaxError, it's a valid reference to the exit builtin injected by the site module.

On Wed, Sep 14, 2016, at 04:31 PM, Paul Moore wrote:

Of course, it doesn't solve the problem of a user who doesn't know what a "shell prompt" is...

Especially since the interactive Python interpreter is also a shell prompt. The title of an IDLE window on my system is "Python 3.5.2 Shell" ;-). I'd suggest the terminology "system command prompt" to refer to... that thing that you should type 'pip' into.

Even if users don't know exactly what it means, indicating that they're running something in the wrong place gives them a better chance of working out what to do.

Thomas

On Thu, Sep 15, 2016, at 01:53 PM, Wes Turner wrote:

Shell commands can/could/should be clearly indicated with a '$' prefix:

$ python -m pip install numpy

That's a common convention, but:

- It comes from POSIX platforms where the default prompt ends in $. The default prompt in Windows ends with >, so $ is not clear. - On all platforms, new users may well not recognise that convention. - Users may think they have to type the $ as part of the command, leading to even more confusion.

On 16 September 2016 at 05:38, Wes Turner wes.turner@gmail.com wrote:

You can install bash on windows. (GoW, Cygwin, Ming), but Docker (and, soon, runC OCP 1.0) is probably your best bet for maximum compatibility with most {python, open source} tutorials.

Wes, none of these are beginner friendly recommendations - our target audience here is folks that *don't yet know what the phrase "command prompt" means*.

So while there are a myriad of ways to get access to a *nix prompt on Windows, they're all targeted at folks that are already experienced developers, and they're also the wrong answer when the goal is to help people to more effectively use the computer they already have, not the computer we happen to think they *should* have.

Cheers, Nick.