Pseudo-package for current directory
Every once in a while, the issue pops up again of "import X" picking up X.py from the current directory, when a stdlib or pip-installed module named X was wanted. Python almost has a mechanism for protecting against this, in the form of explicit package relative imports; the only problem is that the current directory isn't a package.
So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
Is that something that would work with the current import system? It'd offer the same protection that Unix systems have had for decades: the current directory is not in $PATH, and if you want to run a script that's "next door to you", you explicitly request it as ./scriptname. Idle could be protected from accidentally importing someone's "html.py", because "import html" shouldn't ever import from the current directory.
ChrisA
On Mon, Mar 7, 2016, at 11:54, Chris Angelico wrote:
It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
"#!/usr/bin/env python3 -p" won't work on many systems. So I think it'd be better as a magic statement within the file (possibly simply a future-import, with the possibility of eventually making it the default behavior), rather than a command-line argument.
On Tue, Mar 8, 2016 at 4:07 AM, Random832 random832@fastmail.com wrote:
On Mon, Mar 7, 2016, at 11:54, Chris Angelico wrote:
It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
"#!/usr/bin/env python3 -p" won't work on many systems. So I think it'd be better as a magic statement within the file (possibly simply a future-import, with the possibility of eventually making it the default behavior), rather than a command-line argument.
It can't be a future import, though, as they apply to the file they're in, and not to anywhere else. This directive would change the way that the regular "import" statement behaves, for the whole process.
In theory, though, since it doesn't change syntax, it could be something like:
import sys sys.package_mode_ACTIVATE()
The trouble is that doing this after _any_ imports (including the ones that are done before your script starts) will risk those imports being resolved from local files. Having "import X" after activating package mode is useless if X gets resolved from sys.modules, and it'd be a nightmare to debug.
If the -p parameter is a problem, maybe there could be an environment variable that changes the default? Then people could opt-in to pseudo-package mode globally, run all their tests, and see what stops working.
FWIW, I do want this to eventually become the default behaviour. But it wouldn't be any time soon; in the meantime, it would be easy enough to recommend that people just "always do this to be safe" (in the same way that Python 2 didn't make new-style classes the default, but everyone's advised to "always subclass object"). By making it a long-term non-default feature, Python gets to provide safety without breaking anyone's code.
ChrisA
On 3/7/2016 12:22 PM, Chris Angelico wrote:
In theory, though, since it doesn't change syntax, it could be something like:
import sys sys.package_mode_ACTIVATE()
The trouble is that doing this after _any_ imports (including the ones that are done before your script starts) will risk those imports being resolved from local files.
Barring a setting in $PYTHONPATH, Python's startup imports will not be resolved from local files. It appears that '' or scriptdir are only added to sys.path afterwards. I base this on an experiment where I tried to shadow one of the Python-coded /lib modules with a local file.
C-coded builtin modules cannot be shadowed. I discovered this by experiment and then found
"Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files."
https://docs.python.org/3/reference/import.html#finders-and-loaders
I consider the behavior of local versus stdlib imports depending on the stdlib implementation language to be somewhat problematical. There was discussion of shadowing, I believe here, a few months ago (sometime after 3.5.0 was released).
Having "import X" after activating package mode is useless if X gets resolved from sys.modules, and it'd be a nightmare to debug.
If the -p parameter is a problem, maybe there could be an environment variable that changes the default? Then people could opt-in to pseudo-package mode globally, run all their tests, and see what stops working.
FWIW, I do want this to eventually become the default behaviour. But it wouldn't be any time soon; in the meantime, it would be easy enough to recommend that people just "always do this to be safe" (in the same way that Python 2 didn't make new-style classes the default, but everyone's advised to "always subclass object"). By making it a long-term non-default feature, Python gets to provide safety without breaking anyone's code.
You might want to look at previous discussions of shadowing if you can find them.
On Mon, 7 Mar 2016 at 15:02 Terry Reedy tjreedy@udel.edu wrote:
On 3/7/2016 12:22 PM, Chris Angelico wrote:
In theory, though, since it doesn't change syntax, it could be something
like:
import sys sys.package_mode_ACTIVATE()
The trouble is that doing this after _any_ imports (including the ones that are done before your script starts) will risk those imports being resolved from local files.
Barring a setting in $PYTHONPATH, Python's startup imports will not be resolved from local files. It appears that '' or scriptdir are only added to sys.path afterwards. I base this on an experiment where I tried to shadow one of the Python-coded /lib modules with a local file.
C-coded builtin modules cannot be shadowed. I discovered this by experiment and then found
"Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files."
https://docs.python.org/3/reference/import.html#finders-and-loaders
I consider the behavior of local versus stdlib imports depending on the stdlib implementation language to be somewhat problematical. There was discussion of shadowing, I believe here, a few months ago (sometime after 3.5.0 was released).
There was a discussion about shadowing modules that wasn't too long ago, but I don't remember when.
Basically this idea of dropping '' from sys.path so that the local directory isn't included comes up on occasion. Usually it's someone teaching a beginner who named a module something in the stdlib and then got bit by this that suggests it. This discussion usually comes down to "help the beginners" vs "don't break compatibility!" The discussion is slightly nuanced, though, because '' becomes the location of the file passed on the command-line when it's specified so that things don't have to be in a package to be run (Chris' proposal deals with this by forcing the package concept, although you don't need an imaginary __init__.py since we have the concept of namespace packages). But if you drop the directory that a script is contained in, how do you import any packages that are in that directory? What about __main__.py files in the top directory of code checkouts for easy testing of making executable zipfiles? It's a slippery slope, hence why the semantics have not changed.
-Brett
Having "import X" after activating package mode is useless if X gets resolved from sys.modules, and it'd be a nightmare to debug.
If the -p parameter is a problem, maybe there could be an environment variable that changes the default? Then people could opt-in to pseudo-package mode globally, run all their tests, and see what stops working.
FWIW, I do want this to eventually become the default behaviour. But it wouldn't be any time soon; in the meantime, it would be easy enough to recommend that people just "always do this to be safe" (in the same way that Python 2 didn't make new-style classes the default, but everyone's advised to "always subclass object"). By making it a long-term non-default feature, Python gets to provide safety without breaking anyone's code.
You might want to look at previous discussions of shadowing if you can find them.
-- Terry Jan Reedy
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Mar 9, 2016 at 5:47 AM, Brett Cannon brett@python.org wrote:
Basically this idea of dropping '' from sys.path so that the local directory isn't included comes up on occasion. Usually it's someone teaching a beginner who named a module something in the stdlib and then got bit by this that suggests it. This discussion usually comes down to "help the beginners" vs "don't break compatibility!" The discussion is slightly nuanced, though, because '' becomes the location of the file passed on the command-line when it's specified so that things don't have to be in a package to be run (Chris' proposal deals with this by forcing the package concept, although you don't need an imaginary __init__.py since we have the concept of namespace packages). But if you drop the directory that a script is contained in, how do you import any packages that are in that directory?
Packages in that directory would be subpackages of the pseudo-package. Just as "from . import module" would fetch module.py from the script dir, "from .pkg import module" would fetch module.py from the pkg subdirectory. If the package truly stands alone (such that you want it to be imported as "pkg.module" - you can't say "from . import pkg.module", at least not currently), then possibly you want to add the directory to sys.path.
What about __main__.py files in the top directory of code checkouts for easy testing of making executable zipfiles? It's a slippery slope, hence why the semantics have not changed.
Not sure what you mean. What about them? Wouldn't they be scripts just like any other, such that they'd be loading local files with "from . import X"?
ChrisA
On 9 March 2016 at 04:47, Brett Cannon brett@python.org wrote:
But if you drop the directory that a script is contained in, how do you import any packages that are in that directory? What about __main__.py files in the top directory of code checkouts for easy testing of making executable zipfiles? It's a slippery slope, hence why the semantics have not changed.
It's also not uncommon for setup.py files to expect to be able to import the package they relate to.
Accordingly, the proposal I'm most amenable to is the one to simply change the default precedence of the current directory on sys.path by moving it to the end of the filesystem search (after the standard library, site-packages, user site-packages, etc). This wouldn't affect the -m switch (or the runpy module in general), since that manipulates sys.path directly, and would mean creating a "socket.py" script to experiment with the socket modules would "just work".
Such a change poses its own compatibility problems (relating not only to deliberate shadowing of installed modules, but also to software that assumes the first entry in sys.path is the current directory or the script directory and removes it), but such software is already broken when run in isolated mode (which entirely skips adding the current/script directory to sys.path), so pushing folks to find more robust approaches to handling such cases wouldn't be a bad thing.
From a practical implementation perspective, one possible approach
would be to handle this as a second PathImporter on sys.meta_path (after the regular sys.path one), and lose the current/script directory entry from sys.path. That way the impact on other sys.path manipulation code would be minimal - the rest of the startup sequence could continue to just append to sys.path as it does today.
The compatibility break (even if only in limited circumstances) means any such change *would* require a PEP (and probably a way to opt back in to the old behaviour), but the fact this is also an opportunity to *simplify* the implementation (by giving the current/script directory a dedicated meta_path entry rather than using a special token in sys.path) makes me at least willing to consider the idea.
Regards, Nick.
Nick Coghlan <ncoghlan@...> writes:
Accordingly, the proposal I'm most amenable to is the one to simply change the default precedence of the current directory on sys.path by moving it to the end of the filesystem search (after the standard library, site-packages, user site-packages, etc). This wouldn't affect the -m switch (or the runpy module in general), since that manipulates sys.path directly, and would mean creating a "socket.py" script to experiment with the socket modules would "just work".
That would mean that any addition of a module in the standard libary is a compatibility break, because the new module in the standard library would shadow any module in the local directory.
To me the cleanest (from a user point of view) way would be to enable "from . import xxx" in scripts outside packages. When I learned about relative imports I was really surprised by this limitation. If it was possible I'd use it everywhere.
Joseph
On 9.03.2016 12:30, Joseph Martinot-Lagarde wrote:
To me the cleanest (from a user point of view) way would be to enable "from . import xxx" in scripts outside packages. When I learned about relative imports I was really surprised by this limitation. If it was possible I'd use it everywhere.
with python 3.5 import machinery its 3 line hack to fill sys.modules[''].__path__
Niki
Niki Spahiev <niki.spahiev@...> writes:
On 9.03.2016 12:30, Joseph Martinot-Lagarde wrote:
To me the cleanest (from a user point of view) way would be to enable "from . import xxx" in scripts outside packages. When I learned about relative imports I was really surprised by this limitation. If it was possible I'd use it everywhere.
with python 3.5 import machinery its 3 line hack to fill sys.modules[''].__path__
Niki
I wasn't aware of this, thanks for the tip. Still, a 3 line hack is not beginner-friendly, it's still a hack. If it's so simple and useful why not having it by default ? I see it's been discussed in PEP 338 and 366, but it would solve the issue discussed here.
On Tue, Mar 08, 2016 at 03:54:41AM +1100, Chris Angelico wrote:
Every once in a while, the issue pops up again of "import X" picking up X.py from the current directory, when a stdlib or pip-installed module named X was wanted.
Couldn't we (almost) fix that in an almost backwards-compatible way by moving the current directory to the end of the path, instead of the beginning?
If none of the modules in the current directory shadow the standard library, then you won't notice any difference.
If you are shadowing the standard library, then you're either doing it deliberately, which I expect will be rare, or it's an accident. If it's deliberate, then you probably know what you're doing and can work around the change of behaviour.
If it is an accident, then the standard library will shadow your module, rather than the other way around, which strikes me as better. (E.g. you won't accidently break IDLE by shadowing something IDLE depends on.)
Python almost has a mechanism for protecting against this, in the form of explicit package relative imports; the only problem is that the current directory isn't a package.
We shouldn't be talking about "current directory", since that's only relevant when you run Python without specifying a script to run. It is actually the directory containing the script being executed, not the current directory, which gets added to the path. If there is no script, the faux path '' is added to the path, and that gets treated as if it were '.' (the current directory).
So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
Seems awfully convoluted and hard to understand.
How will it interact with running real packages? How about running modules using -m?
Even if it works, it relies on the user being knowledgeable enough to run `python3 -p script` instead of `python3 script`, which means the people who most need this new feature are the least likely to use it. I think that's the fatal objection to this: if the user knows enough to run -p, she knows enough to diagnose an accidental shadowing.
[...]
Idle could be protected from accidentally importing someone's "html.py", because "import html" shouldn't ever import from the current directory.
That won't help when the user runs
python3 html.py
(well perhaps it will help *specifically* with IDLE, but not with the general problem of accidental shadowing).
On Mon, Mar 7, 2016, at 13:04, Steven D'Aprano wrote:
Idle could be protected from accidentally importing someone's "html.py", because "import html" shouldn't ever import from the current directory.
That won't help when the user runs
python3 html.py
It would if doing so *only* loads it as sys.modules['__main__'] and leaves sys.modules['html'] empty.
On Tue, Mar 8, 2016 at 5:04 AM, Steven D'Aprano steve@pearwood.info wrote:
On Tue, Mar 08, 2016 at 03:54:41AM +1100, Chris Angelico wrote:
Every once in a while, the issue pops up again of "import X" picking up X.py from the current directory, when a stdlib or pip-installed module named X was wanted.
Couldn't we (almost) fix that in an almost backwards-compatible way by moving the current directory to the end of the path, instead of the beginning?
Good question. Fortunately that one can be tested without major language hacking. I'm curious to know the answer.
Python almost has a mechanism for protecting against this, in the form of explicit package relative imports; the only problem is that the current directory isn't a package.
We shouldn't be talking about "current directory", since that's only relevant when you run Python without specifying a script to run. It is actually the directory containing the script being executed, not the current directory, which gets added to the path. If there is no script, the faux path '' is added to the path, and that gets treated as if it were '.' (the current directory).
Fair criticism. What I really mean is "script directory", which happens to align in the specific (albeit common) case of "python3 script.py" with no path on it. The rest of my proposal is the same, though.
So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
Seems awfully convoluted and hard to understand.
I know; that's the result of being pedantic. Here's an alternative description:
$ python3 -p somefile.py
pretends that the current directory is a package, and imports somefile from that package.
How will it interact with running real packages? How about running modules using -m?
I don't think it would be any different, but haven't tested. They'd be exactly the same thing, except that "python3 -m X.Y" looks for Y.py inside package X on sys.path, and "python3 -p somefile.py" takes a file name (which might include a path), and runs it directly. If you're running a real package that isn't on sys.path, you can either put it on sys.path, or use -p mode. Something like this, which currently works (and would still work if the current directory were removed from sys.path):
$ mkdir pkg $ echo 'print("init")' >pkg/__init__.py $ echo 'print("module")' >pkg/module.py $ PYTHONPATH=. python3 -m pkg.module init module
Or alternatively, this: $ python3 -p pkg/module.py
If the semantics of -p are "pretend that __init__.py exists", without precluding one that really does, this should behave correctly.
Even if it works, it relies on the user being knowledgeable enough to run `python3 -p script` instead of `python3 script`, which means the people who most need this new feature are the least likely to use it. I think that's the fatal objection to this: if the user knows enough to run -p, she knows enough to diagnose an accidental shadowing.
Right, this is a concern. But I don't want to break backward compatibility. Otherwise, the common situation of having multiple .py files in the same directory and having them import each other would be broken. Ultimately, I'd like to offer this protection by default. But I'm seeing this as similar to reminding people to type "python3" instead of just "python"; now we just teach people to type "python3 -p", and if they happen to miss off the 3, the system will now tell them (because "python2 -p" is an error).
If it's done as an environment variable as well or instead, would that be better? Python distributors and educators could choose to activate it by default, and a venv could apply the protection.
Idle could be protected from accidentally importing someone's "html.py", because "import html" shouldn't ever import from the current directory.
That won't help when the user runs
python3 html.py
(well perhaps it will help *specifically* with IDLE, but not with the general problem of accidental shadowing).
True, but if you're asking for help and you're running a thing called "html.py" or "complex.py", the people you talk to have a chance of noticing. But the presence of such a file in the current directory could mess stuff up, and a lot of people don't think to mention every other file they've ever worked on (in their "Python Tinkering" directory) when they run into difficulties.
ChrisA
On Tue, Mar 08, 2016 at 06:34:11AM +1100, Chris Angelico wrote:
[...]
So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like:
python3 -p somefile.py
which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile".
[...]
Even if it works, it relies on the user being knowledgeable enough to run `python3 -p script` instead of `python3 script`, which means the people who most need this new feature are the least likely to use it. I think that's the fatal objection to this: if the user knows enough to run -p, she knows enough to diagnose an accidental shadowing.
Right, this is a concern. But I don't want to break backward compatibility. Otherwise, the common situation of having multiple .py files in the same directory and having them import each other would be broken.
Only if they shadow something in the std lib. However we fix this, whatever the mechanism, it is going to break backwards compatibility for those shadowing the stdlib, which is of course the whole point of the exercise.
I don't think it's worth trying to gradually introduce this: those affected will either want this change or be able to work around it easily. The work around doesn't involve going through your entire code base changing (potentially) thousands of lines of code. It likely involves changing the name of one or two files, or explicitly adding "." to the front of your path. So even though it is a technical break of backwards compatibility, I think it is (1) minor enough and (2) important enough to warrant just doing it.
There are three cases where people would notice a difference:
(1) Deliberate shadowing of the std lib. In this case, if you are intentionally shadowing a module, I think you are probably capable of coming up with a work-around for the change. E.g. you might explicitly add '.' to your PYTHONPATH environment variable to force it to the front, so as to get the current behaviour.
I think that's a rare case and changing the default behaviour here is acceptible.
(2) Accidental shadowing of the std lib where they currently experience mysterious breakage. E.g.:
[steve@ando tmp]$ touch time.py [steve@ando tmp]$ idle Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/idlelib/run.py", line 7, in <module> import threading File "/usr/local/lib/python2.7/threading.py", line 13, in <module> from time import time as _time, sleep as _sleep ImportError: cannot import name time
This will be fixed, which is the whole purpose of the exercise.
(3) Accidental shadowing where they *don't* currently experience breakage, but will after the change.
E.g. suppose I have a main script "spam.py" and a file called "code.py" in the same directory, and spam.py imports code. Currently, that shadows the stdlib code.py, but harmlessly since few (if any) things depend on the code module. After the change, the stdlib code.py will now shadow the custom one, causing breakage.
I suspect that this will be rare in well-established or professional code, where the devs are more familiar with the std lib, but a bit more common in the case of beginners. This is unfortunate, but the fix is simple: rename your file and change a few imports: `import code` becomes `import mycode as code`.
On 07/03/2016 16:54, Chris Angelico wrote:
"import html" shouldn't ever import from the current directory.
I don't agree. When troubleshooting, you may wish to dig into packages (in the stdlib, or 3rd-party packages) to find out what is going on, and run your own version with (say) diagnostic messages added. I am sure I have done this a few times. Sorry, I can't give chapter and verse.
Rob Cliffe.
On Tue, Mar 8, 2016 at 10:48 AM, Rob Cliffe rob.cliffe@btinternet.com wrote:
On 07/03/2016 16:54, Chris Angelico wrote:
"import html" shouldn't ever import from the current directory.
I don't agree. When troubleshooting, you may wish to dig into packages (in the stdlib, or 3rd-party packages) to find out what is going on, and run your own version with (say) diagnostic messages added. I am sure I have done this a few times. Sorry, I can't give chapter and verse.
In the unusual cases where you actually want that, there are other solutions. You could either change the code to say "from . import html", or spin up a virtual environment and edit the actual code in html.py. Or you could explicitly add a directory to PYTHONPATH for your overrides (~/monkeypatch/html.py or something). The default import mechanism doesn't need to handle overrides IMO.
ChrisA
Sorry if I'm belabouring the point, I should have included this in my last e-mail. But you seem to be saying that when I write import xyz and I mean "import xyz from the current directory, not from the stdlib [or whatever]" I have to know the names of all the modules in the stdlib [or whatever]. Or else *always* add some boilerplate whenever I mean "import from the current directory". Well, maybe, but I am sure a lot of the time I couldn't be bothered, and then some day I'd run into a not-immediately-obvious bug. I.e.: Changing the current behaviour would surprise me. Rob Cliffe
On 08/03/2016 00:06, Chris Angelico wrote:
On Tue, Mar 8, 2016 at 10:48 AM, Rob Cliffe rob.cliffe@btinternet.com wrote:
On 07/03/2016 16:54, Chris Angelico wrote:
"import html" shouldn't ever import from the current directory.
I don't agree. When troubleshooting, you may wish to dig into packages (in the stdlib, or 3rd-party packages) to find out what is going on, and run your own version with (say) diagnostic messages added. I am sure I have done this a few times. Sorry, I can't give chapter and verse.
In the unusual cases where you actually want that, there are other solutions. You could either change the code to say "from . import html", or spin up a virtual environment and edit the actual code in html.py. Or you could explicitly add a directory to PYTHONPATH for your overrides (~/monkeypatch/html.py or something). The default import mechanism doesn't need to handle overrides IMO.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Mar 8, 2016 at 11:32 AM, Rob Cliffe rob.cliffe@btinternet.com wrote:
Sorry if I'm belabouring the point, I should have included this in my last e-mail. But you seem to be saying that when I write import xyz and I mean "import xyz from the current directory, not from the stdlib [or whatever]" I have to know the names of all the modules in the stdlib [or whatever]. Or else *always* add some boilerplate whenever I mean "import from the current directory". Well, maybe, but I am sure a lot of the time I couldn't be bothered, and then some day I'd run into a not-immediately-obvious bug. I.e.: Changing the current behaviour would surprise me.
Which is exactly why I am NOT proposing changing the default behaviour (at least, not any time soon). If you're prepared to accept that "import xyz" will only import from sys.path (which would then not have the script directory in it), then you enable this option, otherwise you don't. When you want that, you say "from . import xyz", and all works as you would expect.
ChrisA
participants (9)
-
Brett Cannon -
Chris Angelico -
Joseph Martinot-Lagarde -
Nick Coghlan -
Niki Spahiev -
Random832 -
Rob Cliffe -
Steven D'Aprano -
Terry Reedy