Beginners accidentally shadow module names due to different expectation of the resolution order for module imports
Hello everyone:
The scenario/observation: Beginners sometimes create scripts with the name of a package e.g. pandas.py or seaborn.py in order to test something. Later, they create and run another script or Jupyter Notebook in the same folder which contains e.g. "import seaborn as sns" where the import itself (most often) succeeds. However - and you see that coming already - when they try to use attributes of the module, this fails with an AttributeError and the poor souls don't know what happened and are not familiar and confident enough to walk the traceback to find out that the import was resolved by the local seaborn.py file instead of the installed seaborn module/library. They are blocked until they have access to a senior Pythonista who helps them out with that or they understand the Stackoverflow advice even though they think the solution should come from a different direction. Either way, *it does not make Python seem seamless and intuitive to them*. (And the next time around, they might have forgotten about this idiosyncrasy and go through the loop again - especially if they don't code on a daily basis which - in my experience - is the fact for a significant number of casual Python users.)
The problem: *Their internal mental model / expectation is that this import name should (first) relate to the installed library - rather than the local file that happens to live in the (often messy) same folder.*
Having said that, I guess that this has been discussed already in the past and I am curious about any pointers to discussions of the past. (I searched for "import order modules" in the archive and could not find something similar in the first 10 results.)
Have a great day, Florian
PS: I guess, one potential solution would be to change the import resolution order. But of course, this is a breaking change (at least in implicit semantics - not sure if it will actually lead to too many real-world problems - but probably there will be some) and might only be feasible in a safe way in a new major version? (Or even in a minor version but with other flags or imports from "the past" instead of "future" that restore the old behavior if that is actually important for some users?)
On Wed, Nov 3, 2021 at 1:39 AM Florian Wetschoreck florian.wetschoreck@gmail.com wrote:
I guess, one potential solution would be to change the import resolution order. But of course, this is a breaking change (at least in implicit semantics - not sure if it will actually lead to too many real-world problems - but probably there will be some) and might only be feasible in a safe way in a new major version? (Or even in a minor version but with other flags or imports from "the past" instead of "future" that restore the old behavior if that is actually important for some users?)
Yes, that would be a breaking change. But here's a potential migration path to consider: Make the script directory into an automatic package. That way, when you specifically *want* to import a nearby file, you can write a relative import:
# demo.py print("Hello, world")
# example.py from . import demo
I'm not sure what the consequences would be on sys.modules; maybe they'd be keyed as, eg, __main__.demo ?
ChrisA
On 11/2/21 6:03 AM, Florian Wetschoreck wrote:
Hello everyone:
The scenario/observation: Beginners sometimes create scripts with the name of a package e.g. pandas.py or seaborn.py in order to test something.
There's a previous discussion about preventing importing a file into itself. It's a common beginner mistake: "I want to try out the random module, I will name my test program random.py". Then they get a baffling error: "AttributeError: module 'random' has no attribute 'shuffle'". If we make the actual error more apparent ("ImportError: module 'random' imported itself"), we might nip this error in the bud.
Thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/XJIEQQ7...
--Ned.
Hey Ned,
Thank you for mentioning the old and related issue.
In order to prevent confusion, I want to point out that the primary scenario that I meant is not that the file imports itself but another file in the same directory with the name of a module that is also installed in site-packages. Not sure if I made that clear in my original post?
On Wed, Nov 3, 2021 at 4:30 PM Ned Batchelder ned@nedbatchelder.com wrote:
On 11/2/21 6:03 AM, Florian Wetschoreck wrote:
Hello everyone:
The scenario/observation: Beginners sometimes create scripts with the name of a package e.g. pandas.py or seaborn.py in order to test something.
There's a previous discussion about preventing importing a file into itself. It's a common beginner mistake: "I want to try out the random module, I will name my test program random.py". Then they get a baffling error: "AttributeError: module 'random' has no attribute 'shuffle'". If we make the actual error more apparent ("ImportError: module 'random' imported itself"), we might nip this error in the bud.
Thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/XJIEQQ7...
--Ned.
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VVQRV5... Code of Conduct: http://python.org/psf/codeofconduct/
On 11/3/21 12:21 PM, Florian Wetschoreck wrote:
In order to prevent confusion, I want to point out that the primary scenario that I meant is not that the file imports itself but another file in the same directory with the name of a module that is also installed in site-packages. Not sure if I made that clear in my original post?
Maybe if the module-level AttributeError included the complete path/filename of the module? It would then be much more easily discernible that the imported "pandas" is not the correct one.
-- ~Ethan~
On Thu, Nov 4, 2021 at 6:37 AM Ethan Furman ethan@stoneleaf.us wrote:
On 11/3/21 12:21 PM, Florian Wetschoreck wrote:
In order to prevent confusion, I want to point out that the primary scenario that I meant is not that the file imports itself but another file in the same directory with the name of a module that is also installed in site-packages. Not sure if I made that clear in my original post?
Maybe if the module-level AttributeError included the complete path/filename of the module? It would then be much more easily discernible that the imported "pandas" is not the correct one.
I like this idea. It already happens with a from-import. A cursory look at Objects/moduleobject.c suggests that it'd be one more lookup in the error handling in module_getattro, pretty doable.
ChrisA
On Wed, Nov 3, 2021, 5:41 PM Chris Angelico rosuav@gmail.com wrote:
On Thu, Nov 4, 2021 at 6:37 AM Ethan Furman ethan@stoneleaf.us wrote:
On 11/3/21 12:21 PM, Florian Wetschoreck wrote:
In order to prevent confusion, I want to point out that the primary
scenario that I meant is not that the file imports
itself but another file in the same directory with the name of a
module that is also installed in site-packages. Not
sure if I made that clear in my original post?
Maybe if the module-level AttributeError included the complete
path/filename of the module? It would then be much more
easily discernible that the imported "pandas" is not the correct one.
I like this idea. It already happens with a from-import. A cursory look at Objects/moduleobject.c suggests that it'd be one more lookup in the error handling in module_getattro, pretty doable.
ChrisA
Why not go one step further and let the user know there is/was another pandas waiting in line to be imported.... "maybe you intended to import module of the same name at these other locations".
This would be an expensive disk read if you did it at the time the exception was raised, but what if at import time python kept a reference around to the other candidate locations of a module name that would have been imported? Most of the time those locations would not have changed.
And furthermore to reduce any hit to performance you could have it only site this information for this helpful error message if the import came from the local directory and not somewhere like site packages. Though I don't know how people would feel about different error messages bubbling up depending on where a module was imported from....
participants (5)
-
Chris Angelico -
Ethan Furman -
Florian Wetschoreck -
Ned Batchelder -
Ricky Teachey