Suggest having a mechanism to distinguish import sources
Hi, I hope that his hasn't been discussed to death in the past but I quite often encounter cases where developers have unwittingly created a file or directory that has a name clash with either a system import or a pip installed library import. This can be very confusing to new users! And, of course, as the number of core libraries and the installable ecosystem grow this potential problem is growing all of the time. I know that we have use absolute imports to force importing local packages but I personally would like to have a notation to allow me to specify: 1. Import this name from the python system libraries only 2. Import this name from installed libraries only 3. Import this mane from the current project, i.e. the local directory or those above it 4. Import this name from local package only, i.e. only the current directory I know that this is possible with the .name notation but a lot od people seem to struggle with it 5. Import this name from wherever you can find it In C/C++ projects that I have worked on it has been possible to do something similar, (providing the correct include flags have been set in the build environment), by distinguishing between `#include <name>` and `#include "name"` where the first will only consider system libraries and the latter only local. I am not sure of what the notation would look like, (and of course the user would still have to take steps to avoid namespace clashes), but possibly something like: ``` from python import os, sys, glob # Only system libraries as candidates from installed import fred from local import os as myos # Personally I think that this would be clearer than from .package install ... import some_other_package # The current behaviour ``` Hopefully nobody would have a file or directory called os but you never know. Throwing this idea out there for discussion. Steve (Gadget) Barnes
On Nov 8, 2019, at 07:50, Steve Barnes <GadgetSteve@live.co.uk> wrote:,
I hope that his hasn’t been discussed to death in the past but I quite often encounter cases where developers have unwittingly created a file or directory that has a name clash with either a system import or a pip installed library import.
There was a somewhat similar thread a few months back about creating a std package that all of the stdlib modules could be imported from. So if you want to make sure you’re getting the system struct library as opposed to a local one, you could `from std import struct`. And, while I don’t think it was raised at the time, the idea could be extended to have packages for site (although the obvious name is already taken) and local, or maybe even allow you to add custom names to custom entries in sys.path. You probably want to find and read that old thread. IIRC, the idea foundered on bikeshedding issues, like: Are std.struct and struct the same module despite being imported differently? If so, what’s the qualified name of that module (and if that’s not just struct, is that a backward compatibility problem), and how does that work with sys.modules and the import system in general? If not, do we have to deprecate importing bare struct? What exactly is in std? Everything installed with the system? Everything that happens to be in a particular directory (plus statically linked in plus frozen into the bootstrap)? Just the stuff that’s available on all Python 3.9 implementations and platforms? Just what’s documented in the library reference? (Is there a platform or cpython or windows or whatever package for the stuff that’s part of CPython 3.9 for Windows but not part of Python 3.9? What about the stackless package that’s built into Stackless and PyPy? What about packages that are standard but optional, like curses, which is theoretically available on every platform but in practice Windows users never have it and only CPython and PyPy include it?) What happens with custom builds of Python, or custom embedding apps, where parts of the stdlib and third-party libs might both be statically linked into the binary, bootstrapped into the importer, and/or stored together in a python39.zip? (You wouldn’t want to build something that works, then package it for distribution with py2app or pyInstaller and have it stop working because stuff that used to be local is now system, and I don’t think you want py2app to have to edit code on the fly while packaging it to fix that, but maybe having it set up a list of packages in a single place would work or something?) There would also be some new questions from extending the idea. For example, are the user and system-wide site packages treated differently or merged together? What about (non-isolated) virtual environment vs. underlying installation? If someone ever updates Jython or Iron to 3.x, what namespace do Java or .NET packages live in? While we’re at it, this might also be an opportunity to solve the problem where running script.py means the same code is available as two separate modules script and __main__, but I’m not sure. Anyway, I think a big part of the problem last time around was that the OP didn’t have a clear idea of what the std package was trying to accomplish, so the answers weren’t obvious, while you seem to have a clear idea of exactly what you want to solve, and are interested in std specifically to solve that problem, so maybe it would be different. There’s a good chance if you went back and read over that thread, you could preemptively answer all of those questions and turn this into a proposal that covers everything but the implementation details (with maybe a few narrow open questions).
I have been bitten by this too. It seems to me like the import system could raise an error in these cases. This seems like it should solve these problems and avoid the need for new syntax. I guess there's a performance problem with this though, since the import system then can't early return?
Hi All I think we can learn from Unix/Linux here. Like python, it has a PATH variable for finding executables. It also has a command 'which'. According to the man page <BEGIN> DESCRIPTION which returns the pathnames of the files (or links) which would be executed in the current environment, had its arguments been given as commands in a strictly POSIX-conformant shell. It does this by searching the PATH for executable files matching the names of the arguments. It does not canonicalize path names. OPTIONS -a print all matching pathnames of each argument <END> Suppose we had a similar 'which' command in Python. It wouldn't by itself stop things going wrong. But when they do go wrong, it might help diagnose the problem. -- Jonathan
On 11/8/19 10:40 AM, Jonathan Fine wrote:
Python already has a "-v" option: -v Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded. When given twice, print a message for each file that is checked for when searching for a module. Also provides information on module cleanup at exit.
On 11/8/19 11:28 AM, Anders Hovmöller wrote:
On 8 Nov 2019, at 18:06, Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
On 11/8/19 10:40 AM, Jonathan Fine wrote:
I think we can learn from Unix/Linux here. Like python, it has a PATH variable for finding executables. It also has a command 'which'.
If I don't understand the problem, then I won't know to use a "which" command, either (or even to think that such a thing exists in the first place). All I know is that I get weird errors. Then again, why would I look for Python's "-v" option? ;-) This is an old problem: https://groups.google.com/forum/#!topic/comp.lang.python/Gy3YVSqr8DM Disallowing a shadowing import seems to be out of the question; a warning might be the right compromise.
throwing this idea out there, no idea if it is practical but it might be pretty nice/easily understood syntax. could a context manager be created such that anything imported under it is guaranteed to be imported from the standard library, and produce an error otherwise? perhaps by adding a level keyword argument to the __import__ built in. something like: with __import__(level="std"): # imports guaranteed to fail of not in the standard library from pathlib import Path from sys import argv with __import__(level="package"): # imports guaranteed to fail of not in the current package import mod1 import mod2 with __import__(level="local"): # imports guaranteed to fail of not in the local directory import mod1 import mod2 with __import__(level="site"): # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy as np
On Fri, Nov 8, 2019 at 10:01 AM Ricky Teachey <ricky@teachey.org> wrote:
__import__ already has a 'level' argument.
Not without frame inspection to know what import statements are in the context manager's block. This can all be done with code which calls importlib.import_module() and checks __spec__.origin to see where the module came from. Basically if you're willing to give up the syntax support of 'import' statements (which are just calls to __import__ with some assignments afterwards to bind things to names) you can have this protection today without adding syntax (which is always a massive ask).
__import__ already has a 'level' argument.
doh! maybe "context" is better, then.
Not without frame inspection to know what import statements are in the context manager's block.
I don't doubt you know what you're talking about, so this is a learning question: why couldn't you do it by simply adding an "inside_context_manager" flag, and producing an appropriate globals and locals based on the flag value? this is really lame, but maybe like this: class __import__: def __enter__(self): self.inside_context_manager = True return None def __exit__(self, *args): self.inside_context_manager = False def __call__(self, name, globals=None, locals=None, fromlist=(), level=0, context=None): am_i_outside_context_manager = not self.inside_context_manager if am_i_outside_context_manager: do_normal_import(name, globals, locals, fromlist, level) else: # inside context manager if context is None: raise ValueError("invalid import context") cglobals = contextualized_globals(globals, context) clocals = contextualized_locals(locals, context) do_special_import(name, cglobals, clocals, fromlist, level)
On Fri, Nov 8, 2019 at 11:07 AM Ricky Teachey <ricky@teachey.org> wrote:
Not thread-safe (we have a custom import lock to avoid this sort of problem) and the instant you start mucking with globals() and locals() you have (a) become a bit magical, and (b) made things extremely slow for PyPy. My opinion on this idea is while I appreciate the motivation behind it, this is proposing a lot of churn and change in Python to fix a single issue that only some people hit and most people learn to avoid the first time to realize what they have done. If it was a problem people ran into constantly throughout their life using Python then maybe I would personally be more receptive, but since it's a one-time mistake for vast majority of people I think I don't feel comfortable making every book on Python be outdated overnight over this. -Brett
On 2019-11-08 18:09, Brett Cannon wrote:
If we were to add syntax, the best I think I can come up with is: # imports guaranteed to fail of not in the standard library from pathlib in std import Path from sys in std import argv # imports guaranteed to fail of not in the current package import mod1 in package import mod2 in package # imports guaranteed to fail of not in the local directory import mod1 in local import mod2 in local # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy in site as np
From: Brett Cannon <brett@python.org> Sent: 08 November 2019 18:10 To: Ricky Teachey <ricky@teachey.org> Cc: Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com>; python-ideas <python-ideas@python.org> Subject: [Python-ideas] Re: Suggest having a mechanism to distinguish import sources On Fri, Nov 8, 2019 at 10:01 AM Ricky Teachey <ricky@teachey.org<mailto:ricky@teachey.org>> wrote: throwing this idea out there, no idea if it is practical but it might be pretty nice/easily understood syntax. could a context manager be created such that anything imported under it is guaranteed to be imported from the standard library, and produce an error otherwise? perhaps by adding a level keyword argument to the __import__ built in. __import__ already has a 'level' argument. [Steve Barnes] To be honest this is the first that I have heard of __import__(level) but looking at the docs and help for 3.8 doesn’t read like what you describe below – maybe it is an un/under-documented feature – it certainly wasn’t easy to find in the documentation (https://docs.python.org/3/library/functions.html#__import__)! level specifies whether to use absolute or relative imports. 0 (the default) means only perform absolute imports. Positive values for level indicate the number of parent directories to search relative to the directory of the module calling __import__()<https://docs.python.org/3/library/functions.html#__import__> (see PEP 328<https://www.python.org/dev/peps/pep-0328> for the details). PEP 328 doesn’t seem to mention any of the names detailed below. something like: with __import__(level="std"): # imports guaranteed to fail of not in the standard library from pathlib import Path from sys import argv with __import__(level="package"): # imports guaranteed to fail of not in the current package import mod1 import mod2 with __import__(level="local"): # imports guaranteed to fail of not in the local directory import mod1 import mod2 with __import__(level="site"): # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy as np Not without frame inspection to know what import statements are in the context manager's block. This can all be done with code which calls importlib.import_module() and checks __spec__.origin to see where the module came from. Basically if you're willing to give up the syntax support of 'import' statements (which are just calls to __import__ with some assignments afterwards to bind things to names) you can have this protection today without adding syntax (which is always a massive ask).
On Sat, Nov 9, 2019 at 10:57 AM Steve Barnes <GadgetSteve@live.co.uk> wrote:
It has to do with handling relative imports. It doesn't have anything to do with the idea being proposed beyond being a parameter name that already exists.
I strongly advise reading PEPs as documentation once their work has landed. At that point they are mostly historical documents and will not be kept up to date going forward. IOW do not read any import-related PEPs for help; everything will be in the stdlib docs for importlib or in the language reference for imports. -Brett
Jonathan Fine wrote:
This thread got me thinking about something exactly along these lines. It looks like the needed information is provided by, e.g., inspect.getfile, which on quick testing works for both modules and for objects imported from modules. For builtins, a TypeError is raised:
For nonbuiltins, the full path to the relevant .py is returned:
inspect.getfile(inspect) 'C:\\...\\Python\\Python37\\Lib\\inspect.py'
I'm sure there are other ways of inferring this, too. So, I guess the first question is whether it's worth actually implementing something into Python to make surfacing this information easier, or more automatic, or whatever -- or if it's really just a marketing campaign to make people more aware of, e.g., `inspect.getfile()` as a handy import debugging tool. From there is the thornier question of whether implementing a feature to enable active filtering of imports based on location/category is a good idea or not. It seems like there's enough variability in installation locations/contexts/etc. that coming up with a consistent set of filtering rules would be a piece of a nightmare....
On Nov 8, 2019, at 18:14, Brian Skinn <brian.skinn@gmail.com> wrote:
What does it do for zipimported modules, bootstrapped modules, modules distributed as a .pyc file without the .py source, etc., plain old .py filed but that are found in an unusual way (a 3.x equivalent of Apple’s 2.7 Extras directory, or even a custom importlib finder), whatever Pythonista does on iOS, etc., much less modules that use a custom loader? Also, even for normal .py imports on a stock CPython on Windows or Mac installation, can a non-sophisticated user (or automated software) reliably distinguish system, site, added-by-old-school-egg, user, venv, script-dir, and custom based on the pathname?
Andrew Barnert wrote:
All excellent questions, of which I had not thought. Research required!
The way I see it, if someone is trying to diagnose a bug coming from a wrong-location import, in order to succeed they are inevitably going to have to become rather *un*-non-sophisticated about the import path cascade in the process. And no, I suspect it would be *quite* hard to write automated software able to reliably assign a given module's path into those categories. It raises basically the same set of problems as writing an automated mechanism for _constraining_ imports only to certain of those category/ies; it "... would be a piece of a nightmare...."
On Nov 8, 2019, at 18:59, Brian Skinn <brian.skinn@gmail.com> wrote:
And no, I suspect it would be *quite* hard to write automated software able to reliably assign a given module's path into those categories. It raises basically the same set of problems as writing an automated mechanism for _constraining_ imports only to certain of those category/ies; it "... would be a piece of a nightmare...."
I’m not sure it would be that hard. The standard install directories have to get set up at Python install time, and the installer knows which of those directories are “stdlib” vs. “site”, is storing that information somewhere is probably not a huge deal. Custom Python installers (like Anaconda) and packaging tools (like py2app) would have to change as well, but again, it shouldn’t be too hard for them. And note that this works fine for zipimports, or custom “paths” that are handled by custom importers (although I suspect you’d want to leave that functionality up to custom finders to do whatever they wanted), because it doesn’t have to be a literal directory path; it can be a zipfile path, or an arbitrary key or whatever. Anything added by the package site is additional “site” packages. The empty entry that the interpreter adds is “local”. Anything else is “other”, but tools/packagers/etc. that want to do something different with their extra directories can have a way to specify one of them. This doesn’t cover the current package directory, but if we can already handle that for relative imports, if we need it here too, we can presumably do it here. This all assumes that we actually want the distinction to be between groups of directories/finders (as opposed to, say, “everything that’s documented as the Python standard library is std but nothing else”), but if that’s what we want, I think it’s doable without an implementation nightmare.
On Nov 8, 2019, at 07:50, Steve Barnes <GadgetSteve@live.co.uk> wrote:
I hope that his hasn’t been discussed to death in the past but I quite often encounter cases where developers have unwittingly created a file or directory that has a name clash with either a system import or a pip installed library import. This can be very confusing to new users! And, of course, as the number of core libraries and the installable ecosystem grow this potential problem is growing all of the time.
One issue I didn’t think of at first: Sometimes you deliberately create a module with the same name as a stdlib module _deliberately_. And we want to be careful not to gratuitously break any legit uses of that. Usually, packages that exist in Python 3.x but are on PyPI for 3.(x-1) and earlier have different names (statistics vs. stats, enum34 vs. enum, etc.), so the user code has to explicitly try one and except import the fallback. But when the fallback is inside your own application, sometimes you just use be same name and inject it into the sys.path if needed. I don’t think the most obvious ways of solving or problem would break things (and if brand-new code starts using try from std import except from local import instead of screwing with sys.path, that’s not a problem, it’s a good thing). But it’s something to keep in mind.
On Nov 8, 2019, at 07:50, Steve Barnes <GadgetSteve@live.co.uk> wrote:,
I hope that his hasn’t been discussed to death in the past but I quite often encounter cases where developers have unwittingly created a file or directory that has a name clash with either a system import or a pip installed library import.
There was a somewhat similar thread a few months back about creating a std package that all of the stdlib modules could be imported from. So if you want to make sure you’re getting the system struct library as opposed to a local one, you could `from std import struct`. And, while I don’t think it was raised at the time, the idea could be extended to have packages for site (although the obvious name is already taken) and local, or maybe even allow you to add custom names to custom entries in sys.path. You probably want to find and read that old thread. IIRC, the idea foundered on bikeshedding issues, like: Are std.struct and struct the same module despite being imported differently? If so, what’s the qualified name of that module (and if that’s not just struct, is that a backward compatibility problem), and how does that work with sys.modules and the import system in general? If not, do we have to deprecate importing bare struct? What exactly is in std? Everything installed with the system? Everything that happens to be in a particular directory (plus statically linked in plus frozen into the bootstrap)? Just the stuff that’s available on all Python 3.9 implementations and platforms? Just what’s documented in the library reference? (Is there a platform or cpython or windows or whatever package for the stuff that’s part of CPython 3.9 for Windows but not part of Python 3.9? What about the stackless package that’s built into Stackless and PyPy? What about packages that are standard but optional, like curses, which is theoretically available on every platform but in practice Windows users never have it and only CPython and PyPy include it?) What happens with custom builds of Python, or custom embedding apps, where parts of the stdlib and third-party libs might both be statically linked into the binary, bootstrapped into the importer, and/or stored together in a python39.zip? (You wouldn’t want to build something that works, then package it for distribution with py2app or pyInstaller and have it stop working because stuff that used to be local is now system, and I don’t think you want py2app to have to edit code on the fly while packaging it to fix that, but maybe having it set up a list of packages in a single place would work or something?) There would also be some new questions from extending the idea. For example, are the user and system-wide site packages treated differently or merged together? What about (non-isolated) virtual environment vs. underlying installation? If someone ever updates Jython or Iron to 3.x, what namespace do Java or .NET packages live in? While we’re at it, this might also be an opportunity to solve the problem where running script.py means the same code is available as two separate modules script and __main__, but I’m not sure. Anyway, I think a big part of the problem last time around was that the OP didn’t have a clear idea of what the std package was trying to accomplish, so the answers weren’t obvious, while you seem to have a clear idea of exactly what you want to solve, and are interested in std specifically to solve that problem, so maybe it would be different. There’s a good chance if you went back and read over that thread, you could preemptively answer all of those questions and turn this into a proposal that covers everything but the implementation details (with maybe a few narrow open questions).
I have been bitten by this too. It seems to me like the import system could raise an error in these cases. This seems like it should solve these problems and avoid the need for new syntax. I guess there's a performance problem with this though, since the import system then can't early return?
Hi All I think we can learn from Unix/Linux here. Like python, it has a PATH variable for finding executables. It also has a command 'which'. According to the man page <BEGIN> DESCRIPTION which returns the pathnames of the files (or links) which would be executed in the current environment, had its arguments been given as commands in a strictly POSIX-conformant shell. It does this by searching the PATH for executable files matching the names of the arguments. It does not canonicalize path names. OPTIONS -a print all matching pathnames of each argument <END> Suppose we had a similar 'which' command in Python. It wouldn't by itself stop things going wrong. But when they do go wrong, it might help diagnose the problem. -- Jonathan
On 11/8/19 10:40 AM, Jonathan Fine wrote:
Python already has a "-v" option: -v Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded. When given twice, print a message for each file that is checked for when searching for a module. Also provides information on module cleanup at exit.
On 11/8/19 11:28 AM, Anders Hovmöller wrote:
On 8 Nov 2019, at 18:06, Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
On 11/8/19 10:40 AM, Jonathan Fine wrote:
I think we can learn from Unix/Linux here. Like python, it has a PATH variable for finding executables. It also has a command 'which'.
If I don't understand the problem, then I won't know to use a "which" command, either (or even to think that such a thing exists in the first place). All I know is that I get weird errors. Then again, why would I look for Python's "-v" option? ;-) This is an old problem: https://groups.google.com/forum/#!topic/comp.lang.python/Gy3YVSqr8DM Disallowing a shadowing import seems to be out of the question; a warning might be the right compromise.
throwing this idea out there, no idea if it is practical but it might be pretty nice/easily understood syntax. could a context manager be created such that anything imported under it is guaranteed to be imported from the standard library, and produce an error otherwise? perhaps by adding a level keyword argument to the __import__ built in. something like: with __import__(level="std"): # imports guaranteed to fail of not in the standard library from pathlib import Path from sys import argv with __import__(level="package"): # imports guaranteed to fail of not in the current package import mod1 import mod2 with __import__(level="local"): # imports guaranteed to fail of not in the local directory import mod1 import mod2 with __import__(level="site"): # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy as np
On Fri, Nov 8, 2019 at 10:01 AM Ricky Teachey <ricky@teachey.org> wrote:
__import__ already has a 'level' argument.
Not without frame inspection to know what import statements are in the context manager's block. This can all be done with code which calls importlib.import_module() and checks __spec__.origin to see where the module came from. Basically if you're willing to give up the syntax support of 'import' statements (which are just calls to __import__ with some assignments afterwards to bind things to names) you can have this protection today without adding syntax (which is always a massive ask).
__import__ already has a 'level' argument.
doh! maybe "context" is better, then.
Not without frame inspection to know what import statements are in the context manager's block.
I don't doubt you know what you're talking about, so this is a learning question: why couldn't you do it by simply adding an "inside_context_manager" flag, and producing an appropriate globals and locals based on the flag value? this is really lame, but maybe like this: class __import__: def __enter__(self): self.inside_context_manager = True return None def __exit__(self, *args): self.inside_context_manager = False def __call__(self, name, globals=None, locals=None, fromlist=(), level=0, context=None): am_i_outside_context_manager = not self.inside_context_manager if am_i_outside_context_manager: do_normal_import(name, globals, locals, fromlist, level) else: # inside context manager if context is None: raise ValueError("invalid import context") cglobals = contextualized_globals(globals, context) clocals = contextualized_locals(locals, context) do_special_import(name, cglobals, clocals, fromlist, level)
On Fri, Nov 8, 2019 at 11:07 AM Ricky Teachey <ricky@teachey.org> wrote:
Not thread-safe (we have a custom import lock to avoid this sort of problem) and the instant you start mucking with globals() and locals() you have (a) become a bit magical, and (b) made things extremely slow for PyPy. My opinion on this idea is while I appreciate the motivation behind it, this is proposing a lot of churn and change in Python to fix a single issue that only some people hit and most people learn to avoid the first time to realize what they have done. If it was a problem people ran into constantly throughout their life using Python then maybe I would personally be more receptive, but since it's a one-time mistake for vast majority of people I think I don't feel comfortable making every book on Python be outdated overnight over this. -Brett
On 2019-11-08 18:09, Brett Cannon wrote:
If we were to add syntax, the best I think I can come up with is: # imports guaranteed to fail of not in the standard library from pathlib in std import Path from sys in std import argv # imports guaranteed to fail of not in the current package import mod1 in package import mod2 in package # imports guaranteed to fail of not in the local directory import mod1 in local import mod2 in local # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy in site as np
From: Brett Cannon <brett@python.org> Sent: 08 November 2019 18:10 To: Ricky Teachey <ricky@teachey.org> Cc: Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com>; python-ideas <python-ideas@python.org> Subject: [Python-ideas] Re: Suggest having a mechanism to distinguish import sources On Fri, Nov 8, 2019 at 10:01 AM Ricky Teachey <ricky@teachey.org<mailto:ricky@teachey.org>> wrote: throwing this idea out there, no idea if it is practical but it might be pretty nice/easily understood syntax. could a context manager be created such that anything imported under it is guaranteed to be imported from the standard library, and produce an error otherwise? perhaps by adding a level keyword argument to the __import__ built in. __import__ already has a 'level' argument. [Steve Barnes] To be honest this is the first that I have heard of __import__(level) but looking at the docs and help for 3.8 doesn’t read like what you describe below – maybe it is an un/under-documented feature – it certainly wasn’t easy to find in the documentation (https://docs.python.org/3/library/functions.html#__import__)! level specifies whether to use absolute or relative imports. 0 (the default) means only perform absolute imports. Positive values for level indicate the number of parent directories to search relative to the directory of the module calling __import__()<https://docs.python.org/3/library/functions.html#__import__> (see PEP 328<https://www.python.org/dev/peps/pep-0328> for the details). PEP 328 doesn’t seem to mention any of the names detailed below. something like: with __import__(level="std"): # imports guaranteed to fail of not in the standard library from pathlib import Path from sys import argv with __import__(level="package"): # imports guaranteed to fail of not in the current package import mod1 import mod2 with __import__(level="local"): # imports guaranteed to fail of not in the local directory import mod1 import mod2 with __import__(level="site"): # imports guaranteed to fail if not in site-packages, or some other definition that makes sense import numpy as np Not without frame inspection to know what import statements are in the context manager's block. This can all be done with code which calls importlib.import_module() and checks __spec__.origin to see where the module came from. Basically if you're willing to give up the syntax support of 'import' statements (which are just calls to __import__ with some assignments afterwards to bind things to names) you can have this protection today without adding syntax (which is always a massive ask).
On Sat, Nov 9, 2019 at 10:57 AM Steve Barnes <GadgetSteve@live.co.uk> wrote:
It has to do with handling relative imports. It doesn't have anything to do with the idea being proposed beyond being a parameter name that already exists.
I strongly advise reading PEPs as documentation once their work has landed. At that point they are mostly historical documents and will not be kept up to date going forward. IOW do not read any import-related PEPs for help; everything will be in the stdlib docs for importlib or in the language reference for imports. -Brett
Jonathan Fine wrote:
This thread got me thinking about something exactly along these lines. It looks like the needed information is provided by, e.g., inspect.getfile, which on quick testing works for both modules and for objects imported from modules. For builtins, a TypeError is raised:
For nonbuiltins, the full path to the relevant .py is returned:
inspect.getfile(inspect) 'C:\\...\\Python\\Python37\\Lib\\inspect.py'
I'm sure there are other ways of inferring this, too. So, I guess the first question is whether it's worth actually implementing something into Python to make surfacing this information easier, or more automatic, or whatever -- or if it's really just a marketing campaign to make people more aware of, e.g., `inspect.getfile()` as a handy import debugging tool. From there is the thornier question of whether implementing a feature to enable active filtering of imports based on location/category is a good idea or not. It seems like there's enough variability in installation locations/contexts/etc. that coming up with a consistent set of filtering rules would be a piece of a nightmare....
On Nov 8, 2019, at 18:14, Brian Skinn <brian.skinn@gmail.com> wrote:
What does it do for zipimported modules, bootstrapped modules, modules distributed as a .pyc file without the .py source, etc., plain old .py filed but that are found in an unusual way (a 3.x equivalent of Apple’s 2.7 Extras directory, or even a custom importlib finder), whatever Pythonista does on iOS, etc., much less modules that use a custom loader? Also, even for normal .py imports on a stock CPython on Windows or Mac installation, can a non-sophisticated user (or automated software) reliably distinguish system, site, added-by-old-school-egg, user, venv, script-dir, and custom based on the pathname?
Andrew Barnert wrote:
All excellent questions, of which I had not thought. Research required!
The way I see it, if someone is trying to diagnose a bug coming from a wrong-location import, in order to succeed they are inevitably going to have to become rather *un*-non-sophisticated about the import path cascade in the process. And no, I suspect it would be *quite* hard to write automated software able to reliably assign a given module's path into those categories. It raises basically the same set of problems as writing an automated mechanism for _constraining_ imports only to certain of those category/ies; it "... would be a piece of a nightmare...."
On Nov 8, 2019, at 18:59, Brian Skinn <brian.skinn@gmail.com> wrote:
And no, I suspect it would be *quite* hard to write automated software able to reliably assign a given module's path into those categories. It raises basically the same set of problems as writing an automated mechanism for _constraining_ imports only to certain of those category/ies; it "... would be a piece of a nightmare...."
I’m not sure it would be that hard. The standard install directories have to get set up at Python install time, and the installer knows which of those directories are “stdlib” vs. “site”, is storing that information somewhere is probably not a huge deal. Custom Python installers (like Anaconda) and packaging tools (like py2app) would have to change as well, but again, it shouldn’t be too hard for them. And note that this works fine for zipimports, or custom “paths” that are handled by custom importers (although I suspect you’d want to leave that functionality up to custom finders to do whatever they wanted), because it doesn’t have to be a literal directory path; it can be a zipfile path, or an arbitrary key or whatever. Anything added by the package site is additional “site” packages. The empty entry that the interpreter adds is “local”. Anything else is “other”, but tools/packagers/etc. that want to do something different with their extra directories can have a way to specify one of them. This doesn’t cover the current package directory, but if we can already handle that for relative imports, if we need it here too, we can presumably do it here. This all assumes that we actually want the distinction to be between groups of directories/finders (as opposed to, say, “everything that’s documented as the Python standard library is std but nothing else”), but if that’s what we want, I think it’s doable without an implementation nightmare.
On Nov 8, 2019, at 07:50, Steve Barnes <GadgetSteve@live.co.uk> wrote:
I hope that his hasn’t been discussed to death in the past but I quite often encounter cases where developers have unwittingly created a file or directory that has a name clash with either a system import or a pip installed library import. This can be very confusing to new users! And, of course, as the number of core libraries and the installable ecosystem grow this potential problem is growing all of the time.
One issue I didn’t think of at first: Sometimes you deliberately create a module with the same name as a stdlib module _deliberately_. And we want to be careful not to gratuitously break any legit uses of that. Usually, packages that exist in Python 3.x but are on PyPI for 3.(x-1) and earlier have different names (statistics vs. stats, enum34 vs. enum, etc.), so the user code has to explicitly try one and except import the fallback. But when the fallback is inside your own application, sometimes you just use be same name and inject it into the sys.path if needed. I don’t think the most obvious ways of solving or problem would break things (and if brand-new code starts using try from std import except from local import instead of screwing with sys.path, that’s not a problem, it’s a good thing). But it’s something to keep in mind.
participants (10)
-
Anders Hovmöller -
Andrew Barnert -
Brett Cannon -
Brian Skinn -
Dan Sommers -
Eric V. Smith -
Jonathan Fine -
MRAB -
Ricky Teachey -
Steve Barnes