Hi, Perl 5.26 succeeded to remove the current working directory from the default include path (our Python sys.path): https://metacpan.org/pod/release/XSAWYERX/perl-5.26.0/pod/perldelta.pod#Remo... Would it technically possible to make this change in Python? Or would it destroy the world? Sorry, it's a naive question (but honestly, I don't know the answer.) My main use case for "." in sys.path is to be to run an application without installing it: run ./hachoir-metadata which loads the Python "hachoir" module from the script directory. Sometimes, I run explicitly "PYTHONPATH=$PWD ./hachoir-metadata". But I don't think that running an application from the source without installing it is the most common way to run an application. Most users install applications to use them, no? Enabling the isolated mode already prevents "." to be added to sys.path: -I command line option. https://docs.python.org/dev/using/cmdline.html#cmdoption-I There is also an old idea of a "restricted" system Python which would use a "fixed" sys.path. Victor
On Fri, Jun 2, 2017 at 2:30 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.) All it'd take is one tiny change to Python, and then one tiny change to any multi-file non-package Python app. 1) Make the script directory implicitly function as a package. In effect, assume that there is an empty __init__.py in the same directory as the thing you just ran. 2) Any time a Python app wants to import from its own directory, it needs to "from . import blah" instead of simply "import blah". Then the removal you suggest could be done, without any loss of functionality. The change could alternatively be done as an import hack rather than an actual fake package if that's easier, such that "from . import blah" means either "import from the script directory" or "import from the current package" as appropriate. Or, it could be more simply done: 1) Make script-directory-local imports raise a warning, citing packages as the best solution. IMO it's a logical extension to relative imports, and a good solution to the "Idle crashes on startup" problem that comes from someone creating a "random.py" in the current directory. Big +1 from me for this. ChrisA
On Thu, Jun 1, 2017 at 4:46 PM, Chris Angelico <rosuav@gmail.com> wrote:
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.)
sys.path includes the current directory (i.e. an empty string) when there's no script, which includes the REPL, -c, and -m. It's removed by [-I]solated mode, which also removes the script directory.
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.)
Oh, it's very common that I run a script from its directory, so yeah script directory = current directory on such case. Sorry for the confusion. You are right, it's the script directory that it added to sys.path and I would like to know if it would be possible to change that? Victor
On Fri, Jun 2, 2017 at 8:58 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Yeah. The rest of my post assumed you meant script directory and, on that basis, wholeheartedly agrees with you. Ultimately, what I would like is for "import random" to be absolutely dependably going to grab the stdlib "random" module, or at very least, something that someone *deliberately* is shadowing that module with. You shouldn't be able to accidentally shadow a stdlib module. ChrisA
On Fri, Jun 02, 2017 at 09:22:16AM +1000, Chris Angelico wrote:
If that's the only problem you want to solve, then I would expect that moving the script/current directory to the *end* of sys.path instead of the start will accomplish that, without breaking any scripts that rely on '' to be in the path. I expect that moving '' to the end of sys.path will be a less disruptive change than removing it. -- Steve
On Fri, Jun 2, 2017 at 11:05 AM, Steven D'Aprano <steve@pearwood.info> wrote:
This is true. However, anything that depends on the current behaviour (intentionally or otherwise) would be just as broken as if it were removed, and it's still possible for "import foo" to mean a local import today and a stdlib import tomorrow. It'd just move the ambiguity around; instead of not being sure if it's been accidentally shadowed, instead you have to wonder if your code will break when a future Python version adds a new stdlib module. Or your code could break because someone pip-installs a third-party module. You can take your pick which thing shadows which by choosing where you place '' in sys.path, but there'll always be a risk one way or another. Different packages don't conflict with each other because intra-package imports are explicit. They're safe. ChrisA
On Fri, Jun 02, 2017 at 12:36:59PM +1000, Chris Angelico wrote: [...]
I don't think we've agreed that the current behaviour is broken. I think we agree that: - it is unfortunate when people accidentally shadow the stdlib; - it is a feature to be able to intentionally shadow the stdlib. I believe that it is also a feature for scripts to be able to depend on resources in their directory, including other modules. That's the current behaviour. I don't know if you agree, but if you want to argue that's "broken", you should do so explicitly. Broken or not, removing '' from the sys.path will break scripts that expect to import modules in their directory. So even if we conclude that the current behaviour is broken, we still need a deprecation period. How about... ? - in 3.7, we add a pair of command line flags, let's say: --script-directory # add '' to sys.path when running scripts --no-script-directory # don't add '' to sys.path with the default remaining to add it - add a warning to 3.7 whenever you import a module from '' - in 3.9, we move '' to the end of sys.path instead of the start - and in 3.9, the default changes to not adding '' to sys.path unless explicitly requested. Alternatively, we could use a single flag: --script-directory-location=FIRST|LAST|NONE
and it's still possible for "import foo" to mean a local import today and a stdlib import tomorrow.
Of course -- that's the downside of any search path. Without a central authority to allocate library names, how do you deal with name conflicts? Any library can clash with any other library, all you can do is choose the order in which clashes are resolved. The advantage of moving '' to the end is that shadowing the stdlib becomes an affirmative, deliberate act, rather than something easy to do by accident. Experts who want to shadow a stdlib module can be assumed to be able to cope with the consequences of moving '' back to the front (or whatever technique they use to shadow a module). The disadvantage is that now the std lib can shadow your modules! Naive users will instead find the std lib shadowing *their* modules, which will be no less mysterious when it happens, but at least you can't break things by shadowing a module you don't directly import. E.g. you import A, which requires B, but you've shadowed B, and now A is broken. That's an especially frustrating error for beginners to diagnose, even with help. Moving '' to the end of the path will, I think, all but eliminate that sort of shadowing error.
You don't have to wonder. You just have to read the What's New document. Your code can also break if you install a third-party library which (accidentally?) shadows the std lib, or another library which you rely on. There are only so many short, descriptive library names, and any time anyone (std lib or not) uses one, it risks clashing with somebody else's use of the same name.
Indeed.
Different packages don't conflict with each other because intra-package imports are explicit. They're safe.
Unless the package shadows another package of the same name. By the way, to give you an idea of how hairy this can get, there was a recent bug report (now closed) complaining that intra-package imports can shadow each other: spam/ +-- __init__.py +-- eggs/ +-- __init__.py +-- cheese +-- eggs.py Inside spam, the eggs.py module shadows the eggs subdirectory and makes it impossible to reach eggs.cheese. (Or perhaps the other way -- I don't think the language guarantees one way or the other.) -- Steve
On Sat, Jun 3, 2017 at 8:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
No, I'm not arguing that that behaviour is broken. Unideal, perhaps, but definitely not broken. What I said was that an application that depends on "import secrets" picking up secrets.py in the current directory is just as broken if '' is moved to the end as if it's removed altogether. By moving it to the end, we increase the chances that a minor version will break someone's code; by removing it altogether and forcing people to write "from . import secrets" (either with an implicit package or making people explicitly create __init__.py), we also force the issue to be fixed earlier. Instead of a potential future breakage, we have an immediate breakage with an easy and obvious solution. That's not to say that I don't think moving '' to the end would be an advantage. I just think that, if we're proposing to change the current behaviour and thus potentially break people's current code, we should fix the problem completely rather than merely reducing it. ChrisA
How about `import self.thing` (where "self" implies same dir as the current .py - if it's a dir-based package, then "self" is that dir) and `import super.thing` (where "super" implies parent package *locked to parent dir*. if there isn't any (top level package or main script), fail-by-default but let scripts override this behaviour (some scripts may want to reconfigure it to ignore "super" if there is no super)) With `import __future__.self_super_packages` to enable it. This should then allow '' to be completely removed, since you can just use `self` and/or `super`. Imports using `self.thing` should have their `super` set to the current `self`, e.g. ./main.py import self.xy ./xy/__init__.py import super.zy ./zy/__init__.py print "hello world" Should print "hello world" when you run main.py, even if there are modules `xy` and `zy` in the python path and no ''. On 2017-06-03 10:23 AM, Chris Angelico wrote:
Soni L. wrote:
How about `import self.thing` (where "self" implies same dir as the current .py
That wouldn't provide quite the same functionality, since currently a module alongside the main py file can be imported from anywhere, including .py files inside a package. Also I think it would be confusing to have two very similar but subtly different relative import mechanisms. -- Greg
How about `import self.thing` (where "self" implies same dir as the current .py - if it's a dir-based package, then "self" is that dir) and `import super.thing` (where "super" implies parent package *locked to parent dir*. if there isn't any (top level package or main script), fail-by-default but let scripts override this behaviour (some scripts may want to reconfigure it to ignore "super" if there is no super)) With `import __future__.self_super_packages` to enable it. This should then allow '' to be completely removed, since you can just use `self` and/or `super`. Imports using `self.thing` should have their `super` set to the current `self`, e.g. ./main.py import self.xy ./xy/__init__.py import super.zy ./zy/__init__.py print "hello world" Should print "hello world" when you run main.py, even if there are modules `xy` and `zy` in the python path and no ''. On 2017-06-03 10:23 AM, Chris Angelico wrote:
Is this really much of a security issue? Seems to me that for someone to exploit it, they would have to inject a malicious .py file alongside one of my script files. If they can do that, they can probably do all kinds of bad things directly. -- Greg
On 4 June 2017 at 10:00, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
There are genuine problems with it, which is why we have the -I switch to enable "isolated mode" (where pretty much all per-user settings get ignored). However, just dropping the current directory from sys.path without also disabling those other features (like user site-packages processing and environment variable processing) really doesn't buy you much. So the better answer from a security perspective is PEP 432 and the separate system-python binary (and Eric Snow recently got us started down that path by merging the initial aspects of that PEP as a private development API, so we can adopt the new settings management architecture incrementally before deciding whether or not we want to support it as a public API). So rather than anything security related, the key reasons I'm personally interested in moving towards requiring main-relative imports to be explicit are a matter of making it easier to reason about a piece of code just by reading it, as well as automatically avoiding certain classes of beginner bugs (i.e. essentially the same arguments PEP 328 put forward for the previous switch away from implicit relative imports in package submodules: https://www.python.org/dev/peps/pep-0328/#rationale-for-absolute-imports). Currently, main relative imports look like this: import helper This means that at the point of reading it, you don't know whether "helper" is independently redistributed, or if it's expected to be distributed alongside the main script. By contrast: from . import helper Makes it clear that "helper" isn't a 3rd party thing, it's meant to be distributed alongside the main script, and if it's missing, you don't want to pick up any arbitrary top level module that happens to be called "helper". Reaching a point where we require main relative imports to be written as "from . import helper" also means that a script called "socket.py" could include the statement "import socket" and actually get the standard library's socket module as it expected - the developer of such a script would have to write "from . import socket" in order to reimport the main script as a module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I really don't want people to start using the "from . import foo" idiom for their first steps into programming. It seems a reasonable "defensive programming" maneuver to put in scripts and apps made by professional Python programmers for surprise-free wide distribution, but (like many of those) should not be part of the learning experience. On Sun, Jun 4, 2017 at 12:35 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Sun, Jun 4, 2017 at 6:51 PM, Guido van Rossum <guido@python.org> wrote:
At the same time, someday you may want the support for 2.7-kind-of issues to stop. Requiring "from" (or other ways to make the source of the import unambiguous) is common in programming languages. Cheers, -- Juancarlo *Añez*
Le 5 juin 2017 00:52, "Guido van Rossum" <guido@python.org> a écrit : I really don't want people to start using the "from . import foo" idiom for their first steps into programming. It seems a reasonable "defensive programming" maneuver to put in scripts and apps made by professional Python programmers for surprise-free wide distribution, but (like many of those) should not be part of the learning experience. A minimum change would be to add the (empty string) at the end of sys.path in Python 3.7 rather than adding it at the start. It would increase Python usability since it avoids the "random has no randint() function" caused by a random.py file in the script directory. In my experience, this bug hits every developers starting to learn Python and it can be very strange when you get the error when trying to run IDLE. I don't think that a new command line parameter is required. It's already easy enough to prepend something to sys.path directly in the script. And I consider that it's a very rare use case. Victor
On 5 June 2017 at 20:55, Victor Stinner <victor.stinner@gmail.com> wrote:
The biggest problem with this approach is that it means that adding new standard library modules becomes a backwards compatibility break - scripts that used to work will now fail since they'll get the standard library module rather than the previously implicit main relative import. At the moment we don't have that problem - as with new builtins, adding a new standard library module may mean people have to rename things to get access to it, but their current code won't actually *break* as a result of the new name being assigned. Hence the "from . import helper" idea - that's unambiguous, so it will always get the co-located library, even if we later add "helper" to the standard library. That said, if we *just* wanted to fix the "random has no attribute randint" problem, without any other side effects, we could potentially special case __main__.__file__ in the import system such that we always ignored it, even if it could technically satisfy the current import request. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jun 5, 2017 at 4:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Python is a bit inconsistent about this. The standard library currently doesn't shadow modules in the script directory, but it does shadow site-packages, which means that new stdlib modules already can break working code. It also makes it impossible to pip install backport modules that intentionally shadow old stdlib modules, which might not be a great idea but is at least plausibly useful in some situations, while the kind of accidental shadowing one gets in the script directory is pretty much always bad IME. -n -- Nathaniel J. Smith -- https://vorpus.org
On 5 June 2017 at 22:25, Nathaniel Smith <njs@pobox.com> wrote:
And if folks want to *reliably* shadow the standard library (whether with their own modules or with third party ones), we already have a solution for that: zipapp and "pip install --target .". You do need to rename your scripts in development from "script.py" to "script/__main__.py" if you want to go down that path, though. I'm still somewhat inclined towards special casing __main__.__spec__.origin, but the point about standard library additions already shadowing site-packages does move me to being +0 on changing the relative precedence of the current directory on sys.path, rather than my previous -1. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jun 5, 2017 at 3:55 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
But it would add the "why won't python import my file?!??!" problem, which newbies also struggle with. Which leaves me with no suggestion for a solution... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Jun 5, 2017 at 10:52 AM Chris Barker <chris.barker@noaa.gov> wrote:
We already got rid of implicit relative imports within packages in Python 3. The primary value in continuing to treat the __main__ module differently is simplicity for learners. If the problem we're trying to solve (really - that needs to be nailed down) is that of overriding standard library modules with your own .py files being an issue (it is! it comes up time and time again)... This simple move from beginning to end is the right way to do it. It does not remove implicit relative imports for the main module but does remove implicit stdlib shadowing. which is something nobody ever wants. and when they do want that unwantable thing, they should be explicit about it instead of relying on magic that depends on code being executed as __main__ vs imported as a module from elsewhere. +0.667 on moving the empty string to the end of sys.path in 3.7 from me. -gps
On Sat, Jun 3, 2017 at 4:36 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Which is why the implicit sys.path entry probably can't go away, even though it would be nice in some ways. IIRC, in the past Guido has indicated he's opposed dropping the implicit sys.path entry for reasons along these lines.
In http://bugs.python.org/issue13475 spells these as "--path0" and "--nopath0". Also see http://www.python.org/dev/peps/pep-0395/ for "ways that the current automatic initialisation of sys.path[0] can go wrong" (quoted from the issue).
Both seem okay. Doing so would help with some of the reasons detailed in PEP 395. However we'd need to be sure the consequences are as minimal as they seem. :)
- and in 3.9, the default changes to not adding '' to sys.path unless explicitly requested.
We'd need to make sure there was a simple, obvious replacement. I'm not convinced dropping the implicit sys.path entry is worth doing though. -eric
On 2 June 2017 at 20:02, Victor Stinner <victor.stinner@gmail.com> wrote:
As long as user site packages are enabled, folks are pretty much hosed on that front (drop a *.pth file in there and you can run arbitrary code at startup). Hence isolated mode and the system-python idea (which can potentially be implemented even while PEP 432 is still a private API, although it would require several more config settings to be migrated to the new structure first). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I'd like to throw some cold water on this one, for the same reason I always add "." to the path in my shell, when some well-meaning soul has removed it. Why? It's 2017 and I've not shared a machine since the 1980's. I use immutable containers in the cloud that are not at this particular risk either. At a small company you might share a file server, but can trust fellow employees. At a large company, you might be at risk, but after many years at one I'd never heard of this actually happening. Guess that leaves hackers? Well, if they are already in... In short I submit this problem is mostly theoretical, as it hasn't occurred the decades(*cough*) of my experience. From small company to large, to the cloud. Has it ever occurred in the history of the world? Sure. On the other hand, requiring "from . " in front of many imports would make python a bit more tedious every single day, for everyone. -1 -Mike p.s. Rearranging sys.path should be tolerable. Have wondered why the current dir was first.
On Mon, Jun 05, 2017 at 12:06:45PM +0200, Stephan Houben <stephanh42@gmail.com> wrote:
What about doing it on the systems which *do* support it? (Which probably covers 99% of the installed base...)
That's the job of Linux distribution packagers, not Python Core developers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Would it not be a job for setuptools? Setuptools creates the scripts. Stephan Op 5 jun. 2017 12:12 schreef "Oleg Broytman" <phd@phdru.name>: On Mon, Jun 05, 2017 at 12:06:45PM +0200, Stephan Houben < stephanh42@gmail.com> wrote:
What about doing it on the systems which *do* support it? (Which probably covers 99% of the installed base...)
That's the job of Linux distribution packagers, not Python Core developers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Jun 05, 2017 at 09:59:35PM +1200, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In case of #!/usr/bin/env - yes. In case of #!/usr/bin/python - combined one-letter arguments can be passed.
-- Greg
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On 5 June 2017 at 19:49, Stephan Houben <stephanh42@gmail.com> wrote:
What about just adding the -I (isolated mode) flag to the #! line of installed scripts?
Fedora & derivatives generally do do that, but as others noted, it can sometimes cause issues with shebang line parsers. It's also easy to lose the setting when a subprocess gets started based on sys.executable. Wrapper scripts can be a little more robust (as long as they use -a to get sys.executable set appropriately), but things still end up being quite intricate and fiddly, and it's hard to prove you've plugged all the gaps. Providing a separate binary with different defaults baked in at build time doesn't magically fix everything (since you still need to change shebang lines to refer to that binary), but it does make it much easier to *stay* in system mode once you're there. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So it seems the best thing would be to have a system-python executable which always runs in isolated mode? In fact I could imagine that security-conscious distributions would only install system-python by default and relegate the ordinary python to some python-dev package. Stephan Op 5 jun. 2017 13:06 schreef "Nick Coghlan" <ncoghlan@gmail.com>:
On Thu, Jun 1, 2017 at 10:30 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
FYI, PEP 432 (Restructuring the CPython startup sequence) [1] facilitates efforts along these lines. It even proposes adding a system-python binary to the cpython build. More specific to the matter of sys.path[0], there's a long-standing issue (#13475) [2] that discusses the topic and some solutions. Note that progress on the issue was blocked by desire to clean up interpreter startup (i.e. PEP 432) first, which has now been accomplished as of the recent PyCon sprints. -eric [1] https://www.python.org/dev/peps/pep-0432/ [2] http://bugs.python.org/issue13475
On 2 June 2017 at 02:30, Victor Stinner <victor.stinner@gmail.com> wrote:
Scripts are very frequently run without installing them, as are things like Jupyter Notebooks, so any change along these lines would need to be carefully planned to avoid being unduly disruptive. It's entirely feasible at a technical level, though - https://bugs.python.org/issue29929 describes one way to move away from "import X" for __main__ relative imports and towards "from . import X", which essentially involves turning __main__ into a package in its own right when its a directly executed script. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Jun 4, 2017 at 12:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
A single-file script wouldn't be affected; only something that has more than one file "side by side" in an arbitrary directory, and imports one from the other. Do Jupyter notebooks do that? I've no idea how they work under the covers. ChrisA
On Jun 3, 2017 2:45 PM, "Chris Angelico" <rosuav@gmail.com> wrote: On Sun, Jun 4, 2017 at 12:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
A single-file script wouldn't be affected; only something that has more than one file "side by side" in an arbitrary directory, and imports one from the other. Do Jupyter notebooks do that? I've no idea how they work under the covers. ChrisA It seems to be pretty common in unit tests in my experience.
On Fri, Jun 2, 2017 at 2:30 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.) All it'd take is one tiny change to Python, and then one tiny change to any multi-file non-package Python app. 1) Make the script directory implicitly function as a package. In effect, assume that there is an empty __init__.py in the same directory as the thing you just ran. 2) Any time a Python app wants to import from its own directory, it needs to "from . import blah" instead of simply "import blah". Then the removal you suggest could be done, without any loss of functionality. The change could alternatively be done as an import hack rather than an actual fake package if that's easier, such that "from . import blah" means either "import from the script directory" or "import from the current package" as appropriate. Or, it could be more simply done: 1) Make script-directory-local imports raise a warning, citing packages as the best solution. IMO it's a logical extension to relative imports, and a good solution to the "Idle crashes on startup" problem that comes from someone creating a "random.py" in the current directory. Big +1 from me for this. ChrisA
On Thu, Jun 1, 2017 at 4:46 PM, Chris Angelico <rosuav@gmail.com> wrote:
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.)
sys.path includes the current directory (i.e. an empty string) when there's no script, which includes the REPL, -c, and -m. It's removed by [-I]solated mode, which also removes the script directory.
(AIUI, the *current directory* is never on Python's path, but the *script directory* is. They're the same thing a lot of the time.)
Oh, it's very common that I run a script from its directory, so yeah script directory = current directory on such case. Sorry for the confusion. You are right, it's the script directory that it added to sys.path and I would like to know if it would be possible to change that? Victor
On Fri, Jun 2, 2017 at 8:58 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Yeah. The rest of my post assumed you meant script directory and, on that basis, wholeheartedly agrees with you. Ultimately, what I would like is for "import random" to be absolutely dependably going to grab the stdlib "random" module, or at very least, something that someone *deliberately* is shadowing that module with. You shouldn't be able to accidentally shadow a stdlib module. ChrisA
On Fri, Jun 02, 2017 at 09:22:16AM +1000, Chris Angelico wrote:
If that's the only problem you want to solve, then I would expect that moving the script/current directory to the *end* of sys.path instead of the start will accomplish that, without breaking any scripts that rely on '' to be in the path. I expect that moving '' to the end of sys.path will be a less disruptive change than removing it. -- Steve
On Fri, Jun 2, 2017 at 11:05 AM, Steven D'Aprano <steve@pearwood.info> wrote:
This is true. However, anything that depends on the current behaviour (intentionally or otherwise) would be just as broken as if it were removed, and it's still possible for "import foo" to mean a local import today and a stdlib import tomorrow. It'd just move the ambiguity around; instead of not being sure if it's been accidentally shadowed, instead you have to wonder if your code will break when a future Python version adds a new stdlib module. Or your code could break because someone pip-installs a third-party module. You can take your pick which thing shadows which by choosing where you place '' in sys.path, but there'll always be a risk one way or another. Different packages don't conflict with each other because intra-package imports are explicit. They're safe. ChrisA
On Fri, Jun 02, 2017 at 12:36:59PM +1000, Chris Angelico wrote: [...]
I don't think we've agreed that the current behaviour is broken. I think we agree that: - it is unfortunate when people accidentally shadow the stdlib; - it is a feature to be able to intentionally shadow the stdlib. I believe that it is also a feature for scripts to be able to depend on resources in their directory, including other modules. That's the current behaviour. I don't know if you agree, but if you want to argue that's "broken", you should do so explicitly. Broken or not, removing '' from the sys.path will break scripts that expect to import modules in their directory. So even if we conclude that the current behaviour is broken, we still need a deprecation period. How about... ? - in 3.7, we add a pair of command line flags, let's say: --script-directory # add '' to sys.path when running scripts --no-script-directory # don't add '' to sys.path with the default remaining to add it - add a warning to 3.7 whenever you import a module from '' - in 3.9, we move '' to the end of sys.path instead of the start - and in 3.9, the default changes to not adding '' to sys.path unless explicitly requested. Alternatively, we could use a single flag: --script-directory-location=FIRST|LAST|NONE
and it's still possible for "import foo" to mean a local import today and a stdlib import tomorrow.
Of course -- that's the downside of any search path. Without a central authority to allocate library names, how do you deal with name conflicts? Any library can clash with any other library, all you can do is choose the order in which clashes are resolved. The advantage of moving '' to the end is that shadowing the stdlib becomes an affirmative, deliberate act, rather than something easy to do by accident. Experts who want to shadow a stdlib module can be assumed to be able to cope with the consequences of moving '' back to the front (or whatever technique they use to shadow a module). The disadvantage is that now the std lib can shadow your modules! Naive users will instead find the std lib shadowing *their* modules, which will be no less mysterious when it happens, but at least you can't break things by shadowing a module you don't directly import. E.g. you import A, which requires B, but you've shadowed B, and now A is broken. That's an especially frustrating error for beginners to diagnose, even with help. Moving '' to the end of the path will, I think, all but eliminate that sort of shadowing error.
You don't have to wonder. You just have to read the What's New document. Your code can also break if you install a third-party library which (accidentally?) shadows the std lib, or another library which you rely on. There are only so many short, descriptive library names, and any time anyone (std lib or not) uses one, it risks clashing with somebody else's use of the same name.
Indeed.
Different packages don't conflict with each other because intra-package imports are explicit. They're safe.
Unless the package shadows another package of the same name. By the way, to give you an idea of how hairy this can get, there was a recent bug report (now closed) complaining that intra-package imports can shadow each other: spam/ +-- __init__.py +-- eggs/ +-- __init__.py +-- cheese +-- eggs.py Inside spam, the eggs.py module shadows the eggs subdirectory and makes it impossible to reach eggs.cheese. (Or perhaps the other way -- I don't think the language guarantees one way or the other.) -- Steve
On Sat, Jun 3, 2017 at 8:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
No, I'm not arguing that that behaviour is broken. Unideal, perhaps, but definitely not broken. What I said was that an application that depends on "import secrets" picking up secrets.py in the current directory is just as broken if '' is moved to the end as if it's removed altogether. By moving it to the end, we increase the chances that a minor version will break someone's code; by removing it altogether and forcing people to write "from . import secrets" (either with an implicit package or making people explicitly create __init__.py), we also force the issue to be fixed earlier. Instead of a potential future breakage, we have an immediate breakage with an easy and obvious solution. That's not to say that I don't think moving '' to the end would be an advantage. I just think that, if we're proposing to change the current behaviour and thus potentially break people's current code, we should fix the problem completely rather than merely reducing it. ChrisA
How about `import self.thing` (where "self" implies same dir as the current .py - if it's a dir-based package, then "self" is that dir) and `import super.thing` (where "super" implies parent package *locked to parent dir*. if there isn't any (top level package or main script), fail-by-default but let scripts override this behaviour (some scripts may want to reconfigure it to ignore "super" if there is no super)) With `import __future__.self_super_packages` to enable it. This should then allow '' to be completely removed, since you can just use `self` and/or `super`. Imports using `self.thing` should have their `super` set to the current `self`, e.g. ./main.py import self.xy ./xy/__init__.py import super.zy ./zy/__init__.py print "hello world" Should print "hello world" when you run main.py, even if there are modules `xy` and `zy` in the python path and no ''. On 2017-06-03 10:23 AM, Chris Angelico wrote:
Soni L. wrote:
How about `import self.thing` (where "self" implies same dir as the current .py
That wouldn't provide quite the same functionality, since currently a module alongside the main py file can be imported from anywhere, including .py files inside a package. Also I think it would be confusing to have two very similar but subtly different relative import mechanisms. -- Greg
How about `import self.thing` (where "self" implies same dir as the current .py - if it's a dir-based package, then "self" is that dir) and `import super.thing` (where "super" implies parent package *locked to parent dir*. if there isn't any (top level package or main script), fail-by-default but let scripts override this behaviour (some scripts may want to reconfigure it to ignore "super" if there is no super)) With `import __future__.self_super_packages` to enable it. This should then allow '' to be completely removed, since you can just use `self` and/or `super`. Imports using `self.thing` should have their `super` set to the current `self`, e.g. ./main.py import self.xy ./xy/__init__.py import super.zy ./zy/__init__.py print "hello world" Should print "hello world" when you run main.py, even if there are modules `xy` and `zy` in the python path and no ''. On 2017-06-03 10:23 AM, Chris Angelico wrote:
Is this really much of a security issue? Seems to me that for someone to exploit it, they would have to inject a malicious .py file alongside one of my script files. If they can do that, they can probably do all kinds of bad things directly. -- Greg
On 4 June 2017 at 10:00, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
There are genuine problems with it, which is why we have the -I switch to enable "isolated mode" (where pretty much all per-user settings get ignored). However, just dropping the current directory from sys.path without also disabling those other features (like user site-packages processing and environment variable processing) really doesn't buy you much. So the better answer from a security perspective is PEP 432 and the separate system-python binary (and Eric Snow recently got us started down that path by merging the initial aspects of that PEP as a private development API, so we can adopt the new settings management architecture incrementally before deciding whether or not we want to support it as a public API). So rather than anything security related, the key reasons I'm personally interested in moving towards requiring main-relative imports to be explicit are a matter of making it easier to reason about a piece of code just by reading it, as well as automatically avoiding certain classes of beginner bugs (i.e. essentially the same arguments PEP 328 put forward for the previous switch away from implicit relative imports in package submodules: https://www.python.org/dev/peps/pep-0328/#rationale-for-absolute-imports). Currently, main relative imports look like this: import helper This means that at the point of reading it, you don't know whether "helper" is independently redistributed, or if it's expected to be distributed alongside the main script. By contrast: from . import helper Makes it clear that "helper" isn't a 3rd party thing, it's meant to be distributed alongside the main script, and if it's missing, you don't want to pick up any arbitrary top level module that happens to be called "helper". Reaching a point where we require main relative imports to be written as "from . import helper" also means that a script called "socket.py" could include the statement "import socket" and actually get the standard library's socket module as it expected - the developer of such a script would have to write "from . import socket" in order to reimport the main script as a module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I really don't want people to start using the "from . import foo" idiom for their first steps into programming. It seems a reasonable "defensive programming" maneuver to put in scripts and apps made by professional Python programmers for surprise-free wide distribution, but (like many of those) should not be part of the learning experience. On Sun, Jun 4, 2017 at 12:35 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Sun, Jun 4, 2017 at 6:51 PM, Guido van Rossum <guido@python.org> wrote:
At the same time, someday you may want the support for 2.7-kind-of issues to stop. Requiring "from" (or other ways to make the source of the import unambiguous) is common in programming languages. Cheers, -- Juancarlo *Añez*
Le 5 juin 2017 00:52, "Guido van Rossum" <guido@python.org> a écrit : I really don't want people to start using the "from . import foo" idiom for their first steps into programming. It seems a reasonable "defensive programming" maneuver to put in scripts and apps made by professional Python programmers for surprise-free wide distribution, but (like many of those) should not be part of the learning experience. A minimum change would be to add the (empty string) at the end of sys.path in Python 3.7 rather than adding it at the start. It would increase Python usability since it avoids the "random has no randint() function" caused by a random.py file in the script directory. In my experience, this bug hits every developers starting to learn Python and it can be very strange when you get the error when trying to run IDLE. I don't think that a new command line parameter is required. It's already easy enough to prepend something to sys.path directly in the script. And I consider that it's a very rare use case. Victor
On 5 June 2017 at 20:55, Victor Stinner <victor.stinner@gmail.com> wrote:
The biggest problem with this approach is that it means that adding new standard library modules becomes a backwards compatibility break - scripts that used to work will now fail since they'll get the standard library module rather than the previously implicit main relative import. At the moment we don't have that problem - as with new builtins, adding a new standard library module may mean people have to rename things to get access to it, but their current code won't actually *break* as a result of the new name being assigned. Hence the "from . import helper" idea - that's unambiguous, so it will always get the co-located library, even if we later add "helper" to the standard library. That said, if we *just* wanted to fix the "random has no attribute randint" problem, without any other side effects, we could potentially special case __main__.__file__ in the import system such that we always ignored it, even if it could technically satisfy the current import request. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jun 5, 2017 at 4:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Python is a bit inconsistent about this. The standard library currently doesn't shadow modules in the script directory, but it does shadow site-packages, which means that new stdlib modules already can break working code. It also makes it impossible to pip install backport modules that intentionally shadow old stdlib modules, which might not be a great idea but is at least plausibly useful in some situations, while the kind of accidental shadowing one gets in the script directory is pretty much always bad IME. -n -- Nathaniel J. Smith -- https://vorpus.org
On 5 June 2017 at 22:25, Nathaniel Smith <njs@pobox.com> wrote:
And if folks want to *reliably* shadow the standard library (whether with their own modules or with third party ones), we already have a solution for that: zipapp and "pip install --target .". You do need to rename your scripts in development from "script.py" to "script/__main__.py" if you want to go down that path, though. I'm still somewhat inclined towards special casing __main__.__spec__.origin, but the point about standard library additions already shadowing site-packages does move me to being +0 on changing the relative precedence of the current directory on sys.path, rather than my previous -1. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jun 5, 2017 at 3:55 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
But it would add the "why won't python import my file?!??!" problem, which newbies also struggle with. Which leaves me with no suggestion for a solution... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Jun 5, 2017 at 10:52 AM Chris Barker <chris.barker@noaa.gov> wrote:
We already got rid of implicit relative imports within packages in Python 3. The primary value in continuing to treat the __main__ module differently is simplicity for learners. If the problem we're trying to solve (really - that needs to be nailed down) is that of overriding standard library modules with your own .py files being an issue (it is! it comes up time and time again)... This simple move from beginning to end is the right way to do it. It does not remove implicit relative imports for the main module but does remove implicit stdlib shadowing. which is something nobody ever wants. and when they do want that unwantable thing, they should be explicit about it instead of relying on magic that depends on code being executed as __main__ vs imported as a module from elsewhere. +0.667 on moving the empty string to the end of sys.path in 3.7 from me. -gps
On Sat, Jun 3, 2017 at 4:36 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Which is why the implicit sys.path entry probably can't go away, even though it would be nice in some ways. IIRC, in the past Guido has indicated he's opposed dropping the implicit sys.path entry for reasons along these lines.
In http://bugs.python.org/issue13475 spells these as "--path0" and "--nopath0". Also see http://www.python.org/dev/peps/pep-0395/ for "ways that the current automatic initialisation of sys.path[0] can go wrong" (quoted from the issue).
Both seem okay. Doing so would help with some of the reasons detailed in PEP 395. However we'd need to be sure the consequences are as minimal as they seem. :)
- and in 3.9, the default changes to not adding '' to sys.path unless explicitly requested.
We'd need to make sure there was a simple, obvious replacement. I'm not convinced dropping the implicit sys.path entry is worth doing though. -eric
On 2 June 2017 at 20:02, Victor Stinner <victor.stinner@gmail.com> wrote:
As long as user site packages are enabled, folks are pretty much hosed on that front (drop a *.pth file in there and you can run arbitrary code at startup). Hence isolated mode and the system-python idea (which can potentially be implemented even while PEP 432 is still a private API, although it would require several more config settings to be migrated to the new structure first). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I'd like to throw some cold water on this one, for the same reason I always add "." to the path in my shell, when some well-meaning soul has removed it. Why? It's 2017 and I've not shared a machine since the 1980's. I use immutable containers in the cloud that are not at this particular risk either. At a small company you might share a file server, but can trust fellow employees. At a large company, you might be at risk, but after many years at one I'd never heard of this actually happening. Guess that leaves hackers? Well, if they are already in... In short I submit this problem is mostly theoretical, as it hasn't occurred the decades(*cough*) of my experience. From small company to large, to the cloud. Has it ever occurred in the history of the world? Sure. On the other hand, requiring "from . " in front of many imports would make python a bit more tedious every single day, for everyone. -1 -Mike p.s. Rearranging sys.path should be tolerable. Have wondered why the current dir was first.
On Mon, Jun 05, 2017 at 12:06:45PM +0200, Stephan Houben <stephanh42@gmail.com> wrote:
What about doing it on the systems which *do* support it? (Which probably covers 99% of the installed base...)
That's the job of Linux distribution packagers, not Python Core developers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Would it not be a job for setuptools? Setuptools creates the scripts. Stephan Op 5 jun. 2017 12:12 schreef "Oleg Broytman" <phd@phdru.name>: On Mon, Jun 05, 2017 at 12:06:45PM +0200, Stephan Houben < stephanh42@gmail.com> wrote:
What about doing it on the systems which *do* support it? (Which probably covers 99% of the installed base...)
That's the job of Linux distribution packagers, not Python Core developers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Jun 05, 2017 at 09:59:35PM +1200, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In case of #!/usr/bin/env - yes. In case of #!/usr/bin/python - combined one-letter arguments can be passed.
-- Greg
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On 5 June 2017 at 19:49, Stephan Houben <stephanh42@gmail.com> wrote:
What about just adding the -I (isolated mode) flag to the #! line of installed scripts?
Fedora & derivatives generally do do that, but as others noted, it can sometimes cause issues with shebang line parsers. It's also easy to lose the setting when a subprocess gets started based on sys.executable. Wrapper scripts can be a little more robust (as long as they use -a to get sys.executable set appropriately), but things still end up being quite intricate and fiddly, and it's hard to prove you've plugged all the gaps. Providing a separate binary with different defaults baked in at build time doesn't magically fix everything (since you still need to change shebang lines to refer to that binary), but it does make it much easier to *stay* in system mode once you're there. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So it seems the best thing would be to have a system-python executable which always runs in isolated mode? In fact I could imagine that security-conscious distributions would only install system-python by default and relegate the ordinary python to some python-dev package. Stephan Op 5 jun. 2017 13:06 schreef "Nick Coghlan" <ncoghlan@gmail.com>:
On Thu, Jun 1, 2017 at 10:30 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
FYI, PEP 432 (Restructuring the CPython startup sequence) [1] facilitates efforts along these lines. It even proposes adding a system-python binary to the cpython build. More specific to the matter of sys.path[0], there's a long-standing issue (#13475) [2] that discusses the topic and some solutions. Note that progress on the issue was blocked by desire to clean up interpreter startup (i.e. PEP 432) first, which has now been accomplished as of the recent PyCon sprints. -eric [1] https://www.python.org/dev/peps/pep-0432/ [2] http://bugs.python.org/issue13475
On 2 June 2017 at 02:30, Victor Stinner <victor.stinner@gmail.com> wrote:
Scripts are very frequently run without installing them, as are things like Jupyter Notebooks, so any change along these lines would need to be carefully planned to avoid being unduly disruptive. It's entirely feasible at a technical level, though - https://bugs.python.org/issue29929 describes one way to move away from "import X" for __main__ relative imports and towards "from . import X", which essentially involves turning __main__ into a package in its own right when its a directly executed script. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Jun 4, 2017 at 12:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
A single-file script wouldn't be affected; only something that has more than one file "side by side" in an arbitrary directory, and imports one from the other. Do Jupyter notebooks do that? I've no idea how they work under the covers. ChrisA
On Jun 3, 2017 2:45 PM, "Chris Angelico" <rosuav@gmail.com> wrote: On Sun, Jun 4, 2017 at 12:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
A single-file script wouldn't be affected; only something that has more than one file "side by side" in an arbitrary directory, and imports one from the other. Do Jupyter notebooks do that? I've no idea how they work under the covers. ChrisA It seems to be pretty common in unit tests in my experience.
participants (19)
-
Chris Angelico -
Chris Barker -
Eric Snow -
eryk sun -
Greg Ewing -
Gregory P. Smith -
Guido van Rossum -
Juancarlo Añez -
Mike Miller -
Nathaniel Smith -
Nick Coghlan -
Niki Spahiev -
Oleg Broytman -
Pavol Lisy -
Soni L. -
Stephan Houben -
Steven D'Aprano -
Todd -
Victor Stinner