On 09/19/2016 09:25 AM, אלעזר wrote:

Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects).

This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type.

Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency.

This should start out as a library on PyPI. (Sorry, couldn't resist. ;) Actually, it should. Perhaps a name of "import_pip" would make sense? Any hurdles faced by this library would be (mostly) the same as a stdlib version. -- ~Ethan~

On 19 September 2016 at 19:55, אלעזר <elazarg@gmail.com> wrote:

A library in PyPi still requires installing it, which undermine many of the benefits. It won't help me with my gist/activestate recipe, code that I send to a friend, etc. I want to lower the barrier of inexperienced users.

As a documentation of dependencies it will suffice indeed.

Elazar

On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman <ethan@stoneleaf.us> wrote:

...

On 09/19/2016 09:25 AM, אלעזר wrote:

...

Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects).

This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type.

Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency.

This should start out as a library on PyPI. (Sorry, couldn't resist. ;)

Actually, it should. Perhaps a name of "import_pip" would make sense? Any hurdles faced by this library would be (mostly) the same as a stdlib version.

-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

using pip from within python is not that difficult already. as can be seen with a glance to: http://stackoverflow.com/questions/12332975/installing-python-module-within-...

I can definitely understand your point. The only issue with it (besides that it doesn't seem to be a good way for dependency management) is how do you manage the syntax involved here? Pip provides distributions. Each distribution contains a set of packages and modules. The latter can be imported, the former not. That's also due to the fact that the name of distribution can contain minuses: from __pip__ import nova-lxd # would this work? What about versions? from __pip__ import run-lambda>=0.1.0 # would this work? Maybe, I thinking too complicated here but if it works for, say, "requests" people tend to want it for special cases as well. :) Cheers, Sven On 19.09.2016 18:55, אלעזר wrote:

A library in PyPi still requires installing it, which undermine many of the benefits. It won't help me with my gist/activestate recipe, code that I send to a friend, etc. I want to lower the barrier of inexperienced users.

As a documentation of dependencies it will suffice indeed.

Elazar

On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman <ethan@stoneleaf.us <mailto:ethan@stoneleaf.us>> wrote:

On 09/19/2016 09:25 AM, אלעזר wrote:

> Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects). > > This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type. > > Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency.

This should start out as a library on PyPI. (Sorry, couldn't resist. ;)

Actually, it should. Perhaps a name of "import_pip" would make sense? Any hurdles faced by this library would be (mostly) the same as a stdlib version.

-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Obviously from __pip__ import "run-lambda>=0.1.0" Which is ugly but not my fault :) On Mon, Sep 19, 2016 at 8:16 PM Sven R. Kunze <srkunze@mail.de> wrote:

I can definitely understand your point.

The only issue with it (besides that it doesn't seem to be a good way for dependency management) is how do you manage the syntax involved here?

Pip provides distributions. Each distribution contains a set of packages and modules. The latter can be imported, the former not. That's also due to the fact that the name of distribution can contain minuses:

from __pip__ import nova-lxd # would this work?

What about versions?

from __pip__ import run-lambda>=0.1.0 # would this work?

Maybe, I thinking too complicated here but if it works for, say, "requests" people tend to want it for special cases as well. :)

Cheers,

Sven

On 19.09.2016 18:55, אלעזר wrote:

A library in PyPi still requires installing it, which undermine many of the benefits. It won't help me with my gist/activestate recipe, code that I send to a friend, etc. I want to lower the barrier of inexperienced users.

As a documentation of dependencies it will suffice indeed.

Elazar

On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman <ethan@stoneleaf.us> wrote:

...

On 09/19/2016 09:25 AM, אלעזר wrote:

...

Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects).

This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type.

Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency.

This should start out as a library on PyPI. (Sorry, couldn't resist. ;)

Actually, it should. Perhaps a name of "import_pip" would make sense? Any hurdles faced by this library would be (mostly) the same as a stdlib version.

-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 19 September 2016 at 18:20, אלעזר <elazarg@gmail.com> wrote:

Obviously

from __pip__ import "run-lambda>=0.1.0"

If we were going to go down this route (which I'm not at all convinced we should) it should be "from __pypi__ import" not "from __pip__ import" (as PyPI is where the code is coming from, not pip...) With importlib, it would probably be perfectly possible to write a module loader that worked like this. Of course, it's not accessible until you manually install it (with pip) and activate it, but for a prototype that demonstrates how the final version would work, and to thrash out how the various corner cases would be handled, that's a perfectly acceptable compromise. If the design of a 3rd-party implementation looks clean and useful, it would be much easier to debate whether it belongs in the stdlib (if for no other reason than the fact that many of the "how would X work" questions could be answered by reference to the prototype). Paul

Ryan I assume any error could be translated to your version through the `from ` statement too? Regarding the rabbit hole, the internet connection and privileges were required for running this script in the first place. I only suggest giving the developer - perhaps not a software engineer at all - an easy way not to require the (possibly inexperienced) user to do it manually. Put it another way, you a copy-pasted code from SO is not needed to use only stdlib now. On Mon, Sep 19, 2016 at 8:34 PM Ryan Gonzalez <rymg19@gmail.com> wrote:

This sounds like a great idea, but I've seen stuff like this done before, and it never ends well. You end up with a gargantuan-sized rabbit hole, since running a basic script could now involve using an internet connection and potentially root permissions.

IF one were to go this route, I prefer bundling a `requirements.txt` into the distribution and then doing something like this in the script:

import pkgcheck pkgcheck.check_requirements('requirements.txt')

If any of the requirements were missing, then something like this would happen:

Traceback (most recent call last): ... MissingPackageError: This script requires package(s) which are not installed: mypackage>=1.0, other_package

That way, you don't get the weird import errors, but you don't have to worry about all the subtleties of automatic downloading.

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ On Sep 19, 2016 11:26 AM, "אלעזר" <elazarg@gmail.com> wrote:

...

Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects).

This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type.

Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency.

Elazar

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Sep 20, 2016 at 4:43 AM, MRAB <python@mrabarnett.plus.com> wrote:

On 2016-09-19 18:20, אלעזר wrote:

...

Obviously

from __pip__ import "run-lambda>=0.1.0"

Which is ugly but not my fault :)

[snip] One possible problem I can see is that if it's quoted you might think that it's an expression and that you could also write:

package = "run-lambda>=0.1.0" from __pip__ import package

One surprisingly effective option is this: import blargle.spam # Solution: pip install blarg Comes out looking like this: Traceback (most recent call last): File "useblarg.py", line 1, in <module> import blargle.spam # Solution: pip install blarg ModuleNotFoundError: No module named 'blargle' (in older Pythons, that says "ImportError" rather than the more specific subclass, but same diff) You can put whatever you like onto that command line. ChrisA

Wow chris, that's a nice one! Of course it doesn't address the issues - for example, running a python script from the file manager will result in a failed execution, unexplained. On Mon, Sep 19, 2016 at 9:48 PM Chris Angelico <rosuav@gmail.com> wrote:

...

On 2016-09-19 18:20, אלעזר wrote:

...

Obviously

from __pip__ import "run-lambda>=0.1.0"

Which is ugly but not my fault :)

[snip] One possible problem I can see is that if it's quoted you might think

On Tue, Sep 20, 2016 at 4:43 AM, MRAB <python@mrabarnett.plus.com> wrote: that

...

it's an expression and that you could also write:

package = "run-lambda>=0.1.0" from __pip__ import package

One surprisingly effective option is this:

import blargle.spam # Solution: pip install blarg

Comes out looking like this:

Traceback (most recent call last): File "useblarg.py", line 1, in <module> import blargle.spam # Solution: pip install blarg ModuleNotFoundError: No module named 'blargle'

(in older Pythons, that says "ImportError" rather than the more specific subclass, but same diff)

You can put whatever you like onto that command line.

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 19 September 2016 at 19:52, אלעזר <elazarg@gmail.com> wrote:

Of course it doesn't address the issues - for example, running a python script from the file manager will result in a failed execution, unexplained.

What you're talking about here is deployment of Python "applications" (where in your case, you're focusing on single-file scripts, but the problem is broader). Beyond a certain point, as I'm sure you realise, you can't solve this issue - someone tries to run a Unix-only script on Windows, for example. So really, we're looking at ways of improving the current state of affairs. As things stand, deploying "standalone applications" in Python is not particularly easy, with dependency management being only one of the issues. Some options for deployment come to mind - I'm sure none of them seem quite as easy as "just run this script", but you should remember that even "just run this script" doesn't always work as well as you'd like - on Unix you need to make the script executable and have a shebang line, on Windows the file extension needs to be right and registered. Not all users have the right (or sometimes any) Python version on their PATH, etc. So even "just run this script" isn't always simple even if the script has no dependencies outside the stdlib. So, some options: 1. The previously-mentioned rwt makes "just run this script" work as you want, as long as you have rwt installed. Conceded, it isn't available by default, but it's not hard to install as a one-off task. And there's a discussion ongoing about including it with pip as "pip run" so "pip run myscript.py" (or "py -m pip run myscript.py" on Windows - remember what I said about PATH? ;-)) will run a script with its dependencies. 2. You can bundle your script with its dependencies using zipapp. On Unix, you can even make the resulting file executable, and on Windows, the .pyz extension is registered for this. 3. Someone could write a tool to scan a script for a specially-formatted comment at the top that described its dependencies, and installed them. That (in effect) adds an "install" step before the script can be run, but lots of utilities have that, so it's not completely unheard of. And if it's GUI users you're concerned about, you could make that a GUI utility. Drag the script onto it, it lists the dependencies and confirms you want them installed, then offers to run the script. 4. There are existing platform-specific options like py2exe or cx_Freeze. Probably a bit heavyweight for a "simple script", though. Of course, none of these options cover the use case of "I saw this nice script on Stack Overflow, so I copied it, and now I want to run it on my PC". But honestly, if it has some dependencies, and the script author didn't say "you need to install FOO and BAR", or the user doesn't know how to install those dependencies, you're probably out of luck anyway. Even if the proposed "from __pypi__ import foo" syntax existed, I doubt many scripts posted on the web would use it - if only because then, Python 3.5 users wouldn't be able to run the scripts! IMO, for command line use, rwt is 99% of the way to what you want. For GUI use, there's nothing specific to this problem right now, but someone could certainly write something. A syntax change to the language would likely be *less* useful, as people would not be able to use it unless they were deliberately only targeting Python 3.7 and later, which seems optimistic at best... Paul

Paul, I understand what you say, except the part of compatibility with Python3.5. Of course such a change is targeting future version more than the current and previous versions. If we have this syntax for Python3.6, users of Python3.9 will find many copy-pastable answers written this way, and other answers will have comments saying "just add this statement to the beginning of your script". On Mon, Sep 19, 2016 at 11:24 PM Paul Moore <p.f.moore@gmail.com> wrote:

On 19 September 2016 at 19:52, אלעזר <elazarg@gmail.com> wrote:

...

Of course it doesn't address the issues - for example, running a python script from the file manager will result in a failed execution, unexplained.

What you're talking about here is deployment of Python "applications" (where in your case, you're focusing on single-file scripts, but the problem is broader). Beyond a certain point, as I'm sure you realise, you can't solve this issue - someone tries to run a Unix-only script on Windows, for example. So really, we're looking at ways of improving the current state of affairs. As things stand, deploying "standalone applications" in Python is not particularly easy, with dependency management being only one of the issues.

Some options for deployment come to mind - I'm sure none of them seem quite as easy as "just run this script", but you should remember that even "just run this script" doesn't always work as well as you'd like - on Unix you need to make the script executable and have a shebang line, on Windows the file extension needs to be right and registered. Not all users have the right (or sometimes any) Python version on their PATH, etc. So even "just run this script" isn't always simple even if the script has no dependencies outside the stdlib.

So, some options:

1. The previously-mentioned rwt makes "just run this script" work as you want, as long as you have rwt installed. Conceded, it isn't available by default, but it's not hard to install as a one-off task. And there's a discussion ongoing about including it with pip as "pip run" so "pip run myscript.py" (or "py -m pip run myscript.py" on Windows - remember what I said about PATH? ;-)) will run a script with its dependencies. 2. You can bundle your script with its dependencies using zipapp. On Unix, you can even make the resulting file executable, and on Windows, the .pyz extension is registered for this. 3. Someone could write a tool to scan a script for a specially-formatted comment at the top that described its dependencies, and installed them. That (in effect) adds an "install" step before the script can be run, but lots of utilities have that, so it's not completely unheard of. And if it's GUI users you're concerned about, you could make that a GUI utility. Drag the script onto it, it lists the dependencies and confirms you want them installed, then offers to run the script. 4. There are existing platform-specific options like py2exe or cx_Freeze. Probably a bit heavyweight for a "simple script", though.

Of course, none of these options cover the use case of "I saw this nice script on Stack Overflow, so I copied it, and now I want to run it on my PC". But honestly, if it has some dependencies, and the script author didn't say "you need to install FOO and BAR", or the user doesn't know how to install those dependencies, you're probably out of luck anyway. Even if the proposed "from __pypi__ import foo" syntax existed, I doubt many scripts posted on the web would use it - if only because then, Python 3.5 users wouldn't be able to run the scripts!

IMO, for command line use, rwt is 99% of the way to what you want. For GUI use, there's nothing specific to this problem right now, but someone could certainly write something. A syntax change to the language would likely be *less* useful, as people would not be able to use it unless they were deliberately only targeting Python 3.7 and later, which seems optimistic at best...

Paul

Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the university, the person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip". Of course it still requires privileges and network connection, etc. And yes it can be solved in other ways automatically, but the fact is, it isn't. We get the side benefit of making more people aware of pypi and actually using it; many people are not aware and/or don't bother using it. In other words, the language will encourage people to use pypi. It might has its downsides (of introducing possibly unneeded dependencies) but I am under the impression that using pypi is something that is considered A Good Thing, in general. Elazar

On 19 September 2016 at 23:13, אלעזר <elazarg@gmail.com> wrote:

Alternative syntax might be that of import decorators

@from_pip("attrs") import attr

At which point you may as well just (insert disclaimer about "using pip's internal API isn't supported" here, and glossing over the fact that pip doesn't yet have an install function with this API) do import pip pip.install('attrs') import attr It's not as if that's functionally any different, or in practical terms any harder to read. And it needs no change to Python. Paul

...

...

import pip pip.main(['install', 'boltons']) Requirement already satisfied (use --upgrade to upgrade): boltons in ./anaconda/lib/python3.5/site-packages 0

This is easy, short, and uses existing syntax and an existing module. Yes, I know technically the API isn't promised; I wouldn't object to making that API official. But PLEASE, no new syntax for doing this! Btw. On failure:

...

...

pip.main(['install', 'nonesuch']) Collecting nonesuch Could not find a version that satisfies the requirement nonesuch (from versions: ) No matching distribution found for nonesuch 1

On Mon, Sep 19, 2016 at 3:46 PM, אלעזר <elazarg@gmail.com> wrote:

On Tue, Sep 20, 2016 at 1:40 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

On 19 September 2016 at 23:13, אלעזר <elazarg@gmail.com> wrote:

...

Alternative syntax might be that of import decorators

@from_pip("attrs") import attr

At which point you may as well just (insert disclaimer about "using pip's internal API isn't supported" here, and glossing over the fact that pip doesn't yet have an install function with this API) do

import pip pip.install('attrs') import attr

Please forgive me for my ignorance, but it doesn't work as written - what's the actual method?

Elazar

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

אלעזר writes:

Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the university, the person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip".

In other words, you're advocating a feature that allows script writers to download, install, and execute arbitrary, unsandboxed code on any machine where the script is run. That sounds ... *scary*, when put that way. Remember, you're advocating this on behalf of people who by assumption are infants years below the age of consent.

אלעזר writes:

...

Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the university, the person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip".

In other words, you're advocating a feature that allows script writers to download, install, and execute arbitrary, unsandboxed code on any machine where the script is run. That sounds ... *scary*, when put that way. Remember, you're advocating this on behalf of people who by assumption are infants years below the age of consent.

Let me understand. Your argument is "installing pip modules is unsafe, and

On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote: therefore we should make it less usable, where the appropriate amount of (un)usability is running cmd and then `pip install unsafe`" ?

Xavier, how is connecting and installing different from a windows popup "this software requires changes to your firewall settings" or "requires elevated privileges" which already happens. I am all for a two-step process, but I think it should be more user friendly, and it can be done as a Python command from inside the script. If I send you a small script, it should be treated in the same way as if I send you a program - an installer - not as a code that you should incorporate into your already existing code base. ‪On Tue, Sep 20, 2016 at 2:28 AM ‫אלעזר‬‎ <elazarg@gmail.com> wrote:‬

On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

...

אלעזר writes:

...

Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the university, the person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip".

In other words, you're advocating a feature that allows script writers to download, install, and execute arbitrary, unsandboxed code on any machine where the script is run. That sounds ... *scary*, when put that way. Remember, you're advocating this on behalf of people who by assumption are infants years below the age of consent.

Let me understand. Your argument is "installing pip modules is unsafe, and therefore we should make it less usable, where the appropriate amount of (un)usability is running cmd and then `pip install unsafe`" ?

On Tue, Sep 20, 2016 at 9:27 AM, Xavier Combelle <xavier.combelle@gmail.com> wrote:

I find the idea of tracking the dependencies in the script might be a good idea. However, magically downloading without warning the user is in my point of view for sure a bad idea. I would far prefer that pip could scan a script to know the dependencies. (A little bit like a requirements.txt but inside the script) A special comment or docstring would do the job. for example

""" pip_requirements: - requests >0.0 - asyncio """"

to run the script it would be at the first time a two step process for example:

python3 -m pip --script-dependencies [--user] my_script.py python3 my_script.py

How about this: python3 -m pip install -r requirements.txt python3 my_script.py That already works, but the requirements have to be in a separate text file, not in the .py file itself. If you want a docstring-based solution, I'd piggy-back it on requirements.txt, and use the same format (eg if the docstring starts "pip_requirements:", the rest of it is a requirements.txt file). Downside: Docstrings can be easily parsed and read *IF* you can successfully import the module. Which you can't, if you still need all those dependencies. So there would have to be some kind of import hook that crafts a dummy package with an infinite number of dummy subpackages, to allow all forms of import to trivially succeed - "import X", "import X.Y", "from X.Y import Z", etc. This may well be more hassle than it's worth. ChrisA

On 09/19/2016 04:38 PM, אלעזר wrote:

I was talking specifically about advanced courses, in which an assignment is "implement a side-channel attack using data" and you can use whatever library you like.

Am I misunderstanding, or did you just say you want this new functionality in order to implement attacks on people's computers? -- ~Ethan~

On Mon, Sep 19, 2016 at 08:34:41PM +0000, אלעזר wrote:

If we have this syntax for Python3.6

3.6 now is under feature-freeze. The earliest any new features can be added is 3.7. -- Steve

On Mon, Sep 19, 2016 at 04:55:26PM +0000, אלעזר wrote:

A library in PyPi still requires installing it, which undermine many of the benefits. It won't help me with my gist/activestate recipe, code that I send to a friend, etc. I want to lower the barrier of inexperienced users.

When you say users, are you talking about completely non-technical end-users who are not programmers at all? In their case, you should bundle the application and all its dependencies into a single frozen .exe file (for Windows), or equivalent for other platforms. Otherwise, the users you are talking about are *programmers*, or at least "technical people" (sys admin, etc), who can be expected to run a couple of commands to install needed dependencies. Perhaps they are inexperienced, and still learning. In that case, teaching them to use pip is a good thing, especially since it isn't that hard: python3 -m ensurepip pip install foo bar baz # whatever your dependencies are (In principle at least. In practice, I have found pip to be not quite as foolproof as advertised. But that's not something the std lib can fix.) https://docs.python.org/3/library/ensurepip.html -- Steve

On Tue, Sep 20, 2016 at 10:42 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Tue, Sep 20, 2016 at 09:34:12AM +1000, Chris Angelico wrote:

...

It's more than that, though. When a student is given an assignment, it's usually to help said student to learn to *write code*, not to learn how to walk into the plumbing store of PyPI and look for something that approximates to the job being done.

That might be sufficient for theoretical computer science courses, but it is *far* from sufficient for learning to be a professional programmer. I think that programmer education lets students down hugely. Knowing how to "walk into the plumbing store" to get a standard plumbing fixture is a skill programmers desperately need.

(We also need better standard plumbing fixtures, but that's another story.)

Failure to teach appropriate use of external dependencies, including when to use them and when not to, is part of the reason so many programmers suffer under Not Invented Here syndrome. It is not always appropriate to use PyPI, but being able to use it is an essential skill for programmers.

This is true, but by the time we're teaching people how to manage dependencies, we've already taught them to manage a *project*. That means a git repository, usually a README, a clear entry point, etc, etc, etc. With that sort of groundwork, it's easy to instruct them to use requirements.txt, rather than try to have a single self-contained .py file that identifies all its deps. Maybe it's different elsewhere, eg if you're starting people off in a scientific field and there's a single obvious library to use.

...

Maybe it's different at university, but with my students, it's always been "no external libraries" (and in some cases, a common-sense avoidance of obvious solutions from the standard library - if you teach someone how to implement a sort and the response is simply "lst.sort()", it's not exactly implementing anything).

This is off-topic, but unless you are specifically doing an algorithms course, why would you bother teaching anyone to implement a sort? Do you teach them to implement their own float multiplication as well?

Yes we are (at least in that instance). With more real-world tests, they're allowed to use all the standard library - this was the "in some cases" bit. (We don't teach float multiplication, because it's very complication, but we do teach some stuff about how bitwise operations work. So yes, we are teaching low level stuff in high level languages.) ChrisA

On 09/19/2016 06:45 PM, Chris Angelico wrote:

Yes we are (at least in that instance). With more real-world tests, they're allowed to use all the standard library - this was the "in some cases" bit. (We don't teach float multiplication, because it's very complication, but we do teach some stuff about how bitwise operations work. So yes, we are teaching low level stuff in high level languages.)

Good. Just remember that "bin(x)" does not print 2s-complement for negative numbers. Had me pulling my hair out for a while. -- ~Ethan~

I believe that at least some of these problems can be addressed given that pip *knows* that this import is an in-script import. So the list of corner cases will be shorter. Elazar On Tue, Sep 20, 2016 at 1:35 PM Paul Moore <p.f.moore@gmail.com> wrote:

On 19 September 2016 at 23:46, אלעזר <elazarg@gmail.com> wrote:

...

...

import pip pip.install('attrs') import attr

Please forgive me for my ignorance, but it doesn't work as written - what's the actual method?

As David Mertz said, pip.main(['install', 'attrs']) works right now, but it is NOT a supported use of pip[1]. To be 100% explicit, the only supported way of doing this is

import sys from subprocess import run run([sys.executable, '-m', 'pip', 'install', 'attrs'])

I suggested a hypothetical "pip.install" method as there is currently some discussion on the pip tracker about providing a supported install method. But it doesn't exist yet. Sorry for being confusing.

While on the whole subject of this, I should also point out that there are a lot of potential issues with installing new packages while a Python program is running. They are all low-probability, and easy to avoid if you're not doing weird things, but for a generally-promoted mechanism, we need to explain the corner cases[2], and an approach with a list of caveats longer than the main documentation is problematic.

1. If the install fails, you need to catch that and report it to the user, in a more friendly manner than pip's output. For example if the user has no C compiler and you need a C extension to be built. 2. You quite possibly want to suppress pip's output if it's *not* a failure, as it'll clutter up the program's real output. 3. If the code has already imported foo.bar, then you install a new version of foo (there are discussions as to whether pip install foo should automatically imply --upgrade, so even if it won't do that by default now, it might in the future), and maybe that new version doesn't have a bar submodule. So now you have a weird mix of old and new code in your process. 4. The install mechanism sometimes (I can't recall the details) caches the fact that it couldn't import a module. If it does that and then later you pip install that module, imports will still fail because the information is cached.

I'm still not at all clear why any of this is so much better than a comment at the top of the script

# To run this script, you need to "pip install attrs" first

Paul.

[1] We've had people report issues where pip breaks their logging config, for example, because pip uses logging but doesn't expect to be run from user code that also does so. [2] That "run([sys.executable, ...])" invocation doesn't work in an embedded program, for example, where sys.executable isn't "python".

On Tue, Sep 20, 2016 at 1:42 PM Paul Moore <p.f.moore@gmail.com> wrote:

On 20 September 2016 at 00:28, אלעזר <elazarg@gmail.com> wrote:

...

On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

...

אלעזר writes:

...

Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the

university,

...

the

...

person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip".

In other words, you're advocating a feature that allows script writers to download, install, and execute arbitrary, unsandboxed code on any machine where the script is run. That sounds ... *scary*, when put that way. Remember, you're advocating this on behalf of people who by assumption are infants years below the age of consent.

Let me understand. Your argument is "installing pip modules is unsafe, and therefore we should make it less usable, where the appropriate amount of (un)usability is running cmd and then `pip install unsafe`" ?

The argument is that if someone posts a script that says it does something innocuous, for example "benchmark showing that X is faster than Y", people will scan it, see that it looks OK, and run it. They have a reasonable expectation that it's not a security risk.

If it requires a benchmarking module from PyPI, they may not immediately notice that "from __pypi__ import benchmark" opens up a security risk. On the other hand, being explicitly told to run a command whose sole purpose is to download and install an external tool clearly indicates to them that they need to be aware of what's happening. Likely they will simply do so and it's no big deal. But in certain environments they may have to pause and possibly even check with their security team as to whether that tool has been approved.

It's not about "making it less usable", it's about ensuring that the implications are clear - explicit is better than implicit, in effect. Which is a particularly important principle when security risks such as "downloading arbitrary code from the internet" is involved.

So it should be something like from unsafe.__pip__ import benchmark Where unsafe is the hypothetical namespace in which exec(), eval() and subprocess.run() would have reside given your concerns. Elazar

On 20 September 2016 at 11:46, אלעזר <elazarg@gmail.com> wrote:

So it should be something like

from unsafe.__pip__ import benchmark

Where unsafe is the hypothetical namespace in which exec(), eval() and subprocess.run() would have reside given your concerns.

In my opinion, it should be # Please install benchmark using pip to run this script Or you should run the script using a dedicated runner like rwt. Or you can depend on a custom import hook that makes "from __pip__ install..." work as you want. I'm just saying that I don't want core Python to implicitly install packages for me. But that's simply a personal opinion. I'm not trying to persuade you you're wrong, just trying to explain my position. We can agree to differ. It certainly doesn't seem to me that there's any need for you to modify your proposal to suit me, it's unlikely I'll like any variation you're going to be happy with, which is fine (you're under no obligation to convince me). Paul

Moreover, being able to do it programmatically is a security risk, since it requires elevated privileges that I don't know how to drop, and most people will not think about doing, but a library implementation will. So if someone uses subprocess.run(), and the system asks the user for elevated privileges, a bug in later code can easily cause serious harm instead of failing. Yes, untrusted code should be sandboxed - but it isn't, more often than not. Elazar ‪On Tue, Sep 20, 2016 at 2:02 PM ‫אלעזר‬‎ <elazarg@gmail.com> wrote:‬

I think I generally understand concerns, and partially agree. I'm certainly not dismissing them. I only try to understand what are the precise problems and why the current situation - with dangerous functions at reach, easily buried deep in the code instead of marked on the top of the script - is so much better.

Elazar

On Tue, Sep 20, 2016 at 1:56 PM Paul Moore <p.f.moore@gmail.com> wrote:

...

On 20 September 2016 at 11:46, אלעזר <elazarg@gmail.com> wrote:

...

So it should be something like

from unsafe.__pip__ import benchmark

Where unsafe is the hypothetical namespace in which exec(), eval() and subprocess.run() would have reside given your concerns.

In my opinion, it should be

# Please install benchmark using pip to run this script

Or you should run the script using a dedicated runner like rwt. Or you can depend on a custom import hook that makes "from __pip__ install..." work as you want. I'm just saying that I don't want core Python to implicitly install packages for me. But that's simply a personal opinion. I'm not trying to persuade you you're wrong, just trying to explain my position. We can agree to differ. It certainly doesn't seem to me that there's any need for you to modify your proposal to suit me, it's unlikely I'll like any variation you're going to be happy with, which is fine (you're under no obligation to convince me).

Paul

On 20 September 2016 at 12:12, אלעזר <elazarg@gmail.com> wrote:

Moreover, being able to do it programmatically is a security risk, since it requires elevated privileges that I don't know how to drop, and most people will not think about doing, but a library implementation will.

So if someone uses subprocess.run(), and the system asks the user for elevated privileges, a bug in later code can easily cause serious harm instead of failing. Yes, untrusted code should be sandboxed - but it isn't, more often than not.

It's not possible to gain elevated privileges without asking the user (certainly not on Windows, and I don't believe so on Unix). So what you're talking about is getting people used to the idea that running a script they grabbed off the internet would ask them to run it elevated, and they should agree. That sounds to me like a very dangerous lesson to be teaching. (rwt gets round this by installing dependencies to a temporary location for the duration of the script. I *really* recommend that you look into it if you haven't already). Paul

On 20 September 2016 at 13:58, Random832 <random832@fastmail.com> wrote:

On Tue, Sep 20, 2016, at 07:12, אלעזר wrote:

...

Moreover, being able to do it programmatically is a security risk, since it requires elevated privileges that I don't know how to drop, and most people will not think about doing, but a library implementation will.

Maybe we should be thinking about why pip requires elevated privileges.

I'm not sure to what extent this was a rhetorical question, but basically because, by default pip installs into the Python installation directory, and if the user is running a system Python, that directory is only modifiable by an admin. You can use --user to make pip install into the user's site-packages. But that's not the default, and the proposal didn't discuss supplying any non-default options to pip. Pip could be changed to make the default --user, but that's not happened yet (and there are some compatibility issues holding it up). And even ignoring that, what about *other* pip options that might be needed (for example, specifying a proxy, or a non-default certificate store)? There's no capability to specify them in the proposal. Paul.

On Tue, Sep 20, 2016 at 9:48 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

Paul Moore writes:

...

I'm just saying that I don't want core Python to implicitly install packages for me.

+1

...

But that's simply a personal opinion.

+0.5

It's not *purely* a personal opinion, though. The original proposal is to include the facility and allow *script authors*, rather than script users, to invoke it. Since the (original) motivation is for it to work for naive users, you'll want it on by default to support them (they're going to click on an icon, so command line options and environment variables are right out, no?)

I have no objection to people who want to install this in their personal environments, but that doesn't serve the original use case of distributing code that depends on non-stdlib modules to people who aren't handy with pip. To serve those use cases, we all need to get it by default. That I personally don't want (for me or for the students I supervise), and I'm pretty sure my employer will take a very dim view of it.

I think that combining user convenience and security considerations, there should be some way to invoke a GUI version of pip with flashing screen asking for permissions to install the library. In situations where interaction with the user is not trivial (i.e. when you don't have GUI accessible) we can assume that the user is knowledgeable enough to install the dependencies by herself. The import statement will be self explanatory in this case. Elazar

On 21 September 2016 at 06:59, Chris Angelico <rosuav@gmail.com> wrote:

On Wed, Sep 21, 2016 at 4:58 AM, אלעזר <elazarg@gmail.com> wrote:

...

I think that combining user convenience and security considerations, there should be some way to invoke a GUI version of pip with flashing screen asking for permissions to install the library. In situations where interaction with the user is not trivial (i.e. when you don't have GUI accessible) we can assume that the user is knowledgeable enough to install the dependencies by herself. The import statement will be self explanatory in this case.

There have been talks of linking pip with Idle, which might do what you want. I don't know how that has progressed, but it'd be something to look into.

The issue for that is here: http://bugs.python.org/issue27051 It turns out the fact IDLE is sometimes used to teach *complete* novices creates a problem for that concept, as learning to safely navigate the free-for-all that is PyPI is actually a pretty advanced development skill vs using a more curated collection like a Linux distro or conda (where there are entities behind them that promise that the code you download will be both non-hostile and at least arguably useful, as opposed to PyPI where the only promise we make is "the code you download will be the code the publisher uploaded" without any attestation, good or otherwise, regarding the trustworthiness of the publisher). A whitelist where we pre-approve a bunch of known-safe components (perhaps even populated automatically from the conda ecosystem) might resolve that, but we really do need some form of curation if we're proposing to offer this as a default capability to learners that aren't even familar with their system command line yet. There's also an open issue at https://github.com/pypa/python-packaging-user-guide/issues/267 regarding improving the documentation on packaging.python.org that points folks towards the various bundling utilities that can given them installers and self-contained scripts for execution on end user systems. In that domain, a potentially useful addition would be a clear recipe for how to combine a script and a PyPI requirements file, into a zipapp archive that bundles all those dependencies along with the script as __main__.py. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia