Mailman 3 Improve error message for trying to import itself - Python-ideas

newer
raise TypeError when giving wrong...

Improve error message for trying to import itself

Gustav O

21 Aug 2020 21 Aug '20

9:35 a.m.

If a file named "tkinter.py" is created and tkinter is imported and used within the file, the following exception is raised: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to a circular import)" I've spoken to multiple beginners who got stuck on this and just couldn't figure out what was happening. Just like the error message tells us, it's due to a "circular import". In other words, we're trying to import the very file that is doing the import. To make the message easier to understand, I propose that we change the message to something that mentions that the program is trying to import itself. I think the following would be an option: "ImportError: partially initialized module 'tkinter' can't import itself" I can't think of any reason to import itself in a way like this, but even if there is a valid reason, the error message could probably be improved. Of course, this applies to more than just tkinter.

Show replies by date

Chris Angelico

21 Aug 21 Aug

1:53 p.m.

On Fri, Aug 21, 2020 at 11:40 PM Gustav O wrote:

...

If a file named "tkinter.py" is created and tkinter is imported and used within the file, the following exception is raised: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to a circular import)"

I've spoken to multiple beginners who got stuck on this and just couldn't figure out what was happening. Just like the error message tells us, it's due to a "circular import". In other words, we're trying to import the very file that is doing the import. To make the message easier to understand, I propose that we change the message to something that mentions that the program is trying to import itself.

I think the following would be an option: "ImportError: partially initialized module 'tkinter' can't import itself"

I can't think of any reason to import itself in a way like this, but even if there is a valid reason, the error message could probably be improved. Of course, this applies to more than just tkinter.

The trouble is, a circular import might not be the module directly importing itself. If you create a file to try to do some arithmetic and call it "math.py", and in there, you "import statistics", you'll end up with a circular import: Traceback (most recent call last): File "/home/rosuav/tmp/aaaa/math.py", line 1, in <module> import statistics File "/usr/local/lib/python3.10/statistics.py", line 105, in <module> import random File "/usr/local/lib/python3.10/random.py", line 49, in <module> from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil ImportError: cannot import name 'log' from 'math' (/home/rosuav/tmp/aaaa/math.py) The current behaviour, as you see here, is to show the source of the failing module. Since the "math" module normally should be a built-in, there's a strong hint that this is where the problem is. It's more flexible than simply catching that a module is trying to import itself. By the way, this actually works: x = 1 from __main__ import x as y print(x, y) Is this the most ridiculous and convoluted way to write "y = x"? ChrisA

Greg Ewing

3:22 p.m.

Maybe check whether the module being imported from is shadowing another module further along the search path and warn about that? -- Greg

Serhiy Storchaka

6:43 p.m.

21.08.20 18:22, Greg Ewing пише:

...

Maybe check whether the module being imported from is shadowing another module further along the search path and warn about that?

It would have non-zero cost. There is a common idiom: try: from foo import bar except ImportError: def bar(): ... In this case you would need to try importing foo from other locations. And what if some improved version intentionally hides the old one? You could end with two imported modules with the same name.

Greg Ewing

11:52 p.m.

On 22/08/20 6:43 am, Serhiy Storchaka wrote:

...

It would have non-zero cost. There is a common idiom:

try: from foo import bar except ImportError: def bar(): ...

In this case you would need to try importing foo from other locations.

I wouldn't suggest going that far. The ImportError should always happen as before, just with a different error message. This case would still fail in an obscure way, but I don't think much can be done about that. -- Greg

Gustav O

3:41 p.m.

Great that you point that out. I have thought of are two possible solutions to this: 1. Make the current error message clearer, but make sure that it's correct for both cases 2. Create a special case for trying to import itself. I personally think the second option would be superior. If necessary, we could create an error that inherits from ImportError for this special case, but I think modifying the message will be enough. This message — or something similar — could be used for when the program is trying to import itself: "ImportError: partially initialized module 'tkinter' can't import itself" Since this isn't a too uncommon mistake, changing the error message for this case could both help aid the learning process and make people less scared of tracebacks, since the error message would actually give some valuable information — and possibly help them solve the error.

Serhiy Storchaka

6:35 p.m.

21.08.20 12:35, Gustav O пише:

...

If a file named "tkinter.py" is created and tkinter is imported and used within the file, the following exception is raised: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to a circular import)"

I've spoken to multiple beginners who got stuck on this and just couldn't figure out what was happening. Just like the error message tells us, it's due to a "circular import". In other words, we're trying to import the very file that is doing the import. To make the message easier to understand, I propose that we change the message to something that mentions that the program is trying to import itself.

I think the following would be an option: "ImportError: partially initialized module 'tkinter' can't import itself"

I can't think of any reason to import itself in a way like this, but even if there is a valid reason, the error message could probably be improved. Of course, this applies to more than just tkinter.

There is nothing wrong wrong in importing itself. Actually when you unpickle function or class defined in the same module you can import the same module. The problem is in using partially initialized modules. And the error message has been already improved. It contains all necessary information and a hint for most common error. And I do not think that it is a common issue. It is more likely to hide the stdlib module imported indirectly.

Gustav O

7:13 p.m.

I actually didn't know that there were use cases for importing itself — thanks for letting me know. The best way of implementing a better error message would probably be to change the message for when the user import itself, while it's still partially initialized. In the beginning of the programming journey, getting a message about circular imports when they were testing out tkinter in a file names "tkinter.py" would probably not be helpful. Getting a message that hints that you may have accidentally imported itself, however, will probably help significantly. Maybe an error like this would be better, even though the wording could be worked on: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to trying to import the file that is being executed)"

Random832

9:03 p.m.

On Fri, Aug 21, 2020, at 15:13, Gustav O wrote:

...

In the beginning of the programming journey, getting a message about circular imports when they were testing out tkinter in a file names "tkinter.py" would probably not be helpful. Getting a message that hints that you may have accidentally imported itself, however, will probably help significantly.

Maybe an error like this would be better, even though the wording could be worked on: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to trying to import the file that is being executed)"

Detecting being partially loaded won't be enough for the shadowed file case, consider this minimal example of a file named tkinter.py: import tkinter if name == '__main__': root = tkinter.Tk() This file is loaded twice, once as __main__ and once as tkinter. The one that is loaded as tkinter and imported is *not* partially initialized, it finishes initializing with no errors. It may be necessary to do more introspection (does the import machinery have a way to see if a module name exists twice on the path?) to determine when an AttributeError is caused by shadowing. It can also happen deep inside stdlib code, if one module whose name is not shadowed imports a shadowed module.

Steven D'Aprano

22 Aug 22 Aug

1:52 a.m.

On Fri, Aug 21, 2020 at 09:35:14AM -0000, Gustav O wrote:

...

If a file named "tkinter.py" is created and tkinter is imported and used within the file, the following exception is raised: "AttributeError: partially initialized module 'tkinter' has no attribute 'Tk' (most likely due to a circular import)"

...

I've spoken to multiple beginners who got stuck on this and just couldn't figure out what was happening.

Alas, I think that this is an error that won't be obvious to a beginner no matter how we word it. * It's unusual enough that googling for the error message comes up with lots of false positives and few correct hits. (I couldn't find anything relevant in the first half dozen results when using DuckDuckGo, although Google's first result gave the correct solution but the wrong error message!) * The concept of a "partially initialized module" requires a pretty sophisticated understanding of the difference between a module "spam.py" as a source file and a module `spam` initialised in memory. * It requires an understanding of circular imports and how they can go wrong. It's also tricky to word the error message in a way that leads beginners to a solution without hiding information or being misleading in other cases. The *actual error* here is that the imported module tkinter doesn't have a Tk attribute -- everything else is a guess as to what the problem *might* be: - a module importing itself is not necessarily wrong; - not all circular imports are modules importing themselves; - loading a partially imported module is a normal consequence of circular imports; - a module not containing an attribute doesn't necessarily mean you have shadowed the real module, it might just mean a typo or bug in your code. The other factor is that *beginners don't read error messages*. Obviously there are exceptions, but beginners tend to be really, really bad at reading error messages even when the messages are clear, succint, to the point, and tell them exactly what is wrong. So I'm reluctant to spend to much time or energy trying to craft the perfect error message to help beginners given that the ones who most need the help are the least likely to read it. Debugging code is a skill and there really is no substitute for time and experience to learn it. One of those lessons is first to learn that you must read the error message and second how to interpret it. Having said all that, I cannot help but feel that shadowing of modules is such an issue that maybe we need to make a special case of this. Not specifically tkinter, of course, but if a module attempts to directly import a module with the same name, and that import fails for any reason, we should include a message about shadowing in addition to the normal exception traceback.

...

I think the following would be an option: "ImportError: partially initialized module 'tkinter' can't import itself"

That's a regression. In this specific case it doesn't matter because you know that the error actually is shadowing (the user named their own file "tkinter.py") but in the general case we lose the information of what name was being looked up: AttributeError: partially initialized module 'spam' has no attribute 'egsg' (most likely due to a circular import) If the error wasn't due to shadowing or a circular import, then knowing that I had misspelled 'eggs' would be very useful. -- Steve

Random832

7:09 a.m.

On Fri, Aug 21, 2020, at 21:52, Steven D'Aprano wrote:

...

Having said all that, I cannot help but feel that shadowing of modules is such an issue that maybe we need to make a special case of this. Not specifically tkinter, of course, but if a module attempts to directly import a module with the same name, and that import fails for any reason, we should include a message about shadowing in addition to the normal exception traceback.

I don't think a majority of these cases result in the error happening during the import rather than at a subsequent attribute access. I think there are two possible paths here: either always print a warning [not an error] when importing a module from the current directory [the first entry in sys.path] that is also available in any other entry in sys.path, or check for that situation whenever a module attribute access fails, to add more information to the error message at that point. I do think circular imports are a bit of a red herring - it's often the case, but not always. I've seen this happen as a result of a chaotic project directory whereby one script they'd written - which didn't itself depend on the named system module - shadowed a module that was imported from a different program.

...

...
I think the following would be an option: "ImportError: partially initialized module 'tkinter' can't import itself"

That's a regression. In this specific case it doesn't matter because you know that the error actually is shadowing (the user named their own file "tkinter.py") but in the general case we lose the information of what name was being looked up:

AttributeError: partially initialized module 'spam' has no attribute 'egsg' (most likely due to a circular import)

If the error wasn't due to shadowing or a circular import, then knowing that I had misspelled 'eggs' would be very useful.

hmm what about AttributeError: module 'spam' from './spam.py' has no attribute

Random832

7:20 a.m.

On Sat, Aug 22, 2020, at 03:09, Random832 wrote:

...

...
AttributeError: partially initialized module 'spam' has no attribute 'egsg' (most likely due to a circular import)

If the error wasn't due to shadowing or a circular import, then knowing that I had misspelled 'eggs' would be very useful.

hmm

what about

AttributeError: module 'spam' from './spam.py' has no attribute

I prematurely hit send, I don't mean to suggest remove the attribute name, just to add the module path [maybe the full path, whatever's in __file__ would be easiest]. This way wouldn't even require detecting circular imports or shadowing specifically, though it would require a little bit more alertness on the part of the user to notice that the path is not the system module they expect. this could be done in combination with a warning or additional text detecting shadowing as suggested in my previous post.

Christopher Barker

6:41 p.m.

On Sat, Aug 22, 2020 at 12:11 AM Random832 wrote:

...

I think there are two possible paths here: either always print a warning [not an error] when importing a module from the current directory [the first entry in sys.path] that is also available in any other entry in sys.path,

This would be GREAT! This is a well known "gotcha" in Python -- as it happens, this happened to me just yesterday, and while I figure it out, I did waste some time on it, 'cause it was happening on the CI, which is configured differently than my development machine, and the person setting up the CI is not as familiar with Python as I am. In fact, once I'd figured it out, her response was "that seems like "gotcha" in Python" -- and indeed she's right. So getting a warning about it would be fabulous! -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Gustav O

23 Aug 23 Aug

12:11 a.m.

I agree — having a warning would be fantastic! Personally, I think the minor speed drawback of checking the full path would be worth it, in most cases. The fact that a module is being shadowed is really valuable information to have. If there’s enough of a demand for this, which I think there is, I’d like to add this to the issue tracker or write a PEP for it. Since this is my first time contributing, any guidance would be greatly appreciated. :) -- Gustav

Chris Angelico

12:17 a.m.

On Sun, Aug 23, 2020 at 10:14 AM Gustav O wrote:

...

I agree — having a warning would be fantastic! Personally, I think the minor speed drawback of checking the full path would be worth it, in most cases. The fact that a module is being shadowed is really valuable information to have.

A speed drawback on every import would almost certainly be too high a price to pay. Maybe this would make a useful excepthook? If an AttributeError is about to be printed to the console, and the failing object is a module, *then* look through the path and see if it was shadowed. ChrisA

Gustav O

12:59 a.m.

It seems like most of us agree that the speed drawbacks wouldn’t be worth it — which I find understandable. If my understanding of excepthooks is correct, and that it’s the place to put code that should be executed once an AttributeError is raised, it sounds like a great solution. However, AttributeErrors aren’t the only exceptions raised when a shadowing module is imported. A broad range of exceptions could be raised as a result of an unintended module being imported. One option would be to implement a similar search whenever any exception is raised and the conditions that you mentioned apply. Limiting it to AttributeError would be odd. -- Gustav

Chris Angelico

1:04 a.m.

On Sun, Aug 23, 2020 at 11:02 AM Gustav O wrote:

...

It seems like most of us agree that the speed drawbacks wouldn’t be worth it — which I find understandable.

If my understanding of excepthooks is correct, and that it’s the place to put code that should be executed once an AttributeError is raised, it sounds like a great solution.

Not quite - it happens if such an error isn't caught. It's the place where exceptions get printed to the console. That's the perfect place for this sort of thing.

...

However, AttributeErrors aren’t the only exceptions raised when a shadowing module is imported. A broad range of exceptions could be raised as a result of an unintended module being imported.

One option would be to implement a similar search whenever any exception is raised and the conditions that you mentioned apply. Limiting it to AttributeError would be odd.

Yep, you can handle whatever errors you like, although I think ImportError and AttributeError will cover pretty much everything you need. As an extra advantage, the exception hook is pure Python code, so you don't have to worry about messing around in C. You should be able to make use of importlib to do most of the work for you. ChrisA

Gustav O

1:17 a.m.

In your answer, it sounds like the code would be written by the person developing a program. Or they could possibly have the importhook a separate module, which can be imported whenever you’d like to have it. Maybe I misunderstood you, but this wouldn’t really be optimal. I think most people would like to get the part about shadowing imports without having to write any code or import a module that sends such error messages if a module might be shadowed. That removes the convenience of having something like that be a standard feature of the language itself. Would it be possible to make the exceptionhook a standard part of python, so it’s there by default?

Chris Angelico

1:22 a.m.

On Sun, Aug 23, 2020 at 11:19 AM Gustav O wrote:

...

In your answer, it sounds like the code would be written by the person developing a program. Or they could possibly have the importhook a separate module, which can be imported whenever you’d like to have it.

Maybe I misunderstood you, but this wouldn’t really be optimal. I think most people would like to get the part about shadowing imports without having to write any code or import a module that sends such error messages if a module might be shadowed. That removes the convenience of having something like that be a standard feature of the language itself.

Initially, it would be something you could add to sitecustomize.py, making it immediately available.

...

Would it be possible to make the exceptionhook a standard part of python, so it’s there by default?

Yes, it would. That would be a fully-debated proposal, but it is certainly possible for this sort of thing to be added to the core language. But if it's done as an excepthook, then even before that, it can be added by any sysadmin or any individual programmer. ChrisA

Gustav O

1:40 a.m.

Alright! I’ll sure take a look at that. I also think adding it to the core language would be greatly beneficial. Just like Christopher said, this is one of those "gotchas". I’m still new to this process, but do you think that writing a PEP for this would be a reasonable next step?

Chris Angelico

1:47 a.m.

On Sun, Aug 23, 2020 at 11:44 AM Gustav O wrote:

...

Alright! I’ll sure take a look at that.

I also think adding it to the core language would be greatly beneficial. Just like Christopher said, this is one of those "gotchas".

I’m still new to this process, but do you think that writing a PEP for this would be a reasonable next step?

I'd start by creating a working implementation, in this case. Since it doesn't require any changes to syntax or anything like that, it should be fairly straight-forward. Then open a new thread with the exact details of what you're proposing. That way, people can try it out, and decide whether they think it's worth adding to core, or worth mentioning in the docs, or whatever else from there. ChrisA

Gustav O

1:51 a.m.

Alright, thanks for the help. You’ll probably see an implementation in a new thread here soon again. :)

Christopher Barker

2:52 a.m.

On Sat, Aug 22, 2020 at 5:19 PM Chris Angelico wrote:

...

On Sun, Aug 23, 2020 at 10:14 AM Gustav O wrote:

...
...
I agree — having a warning would be fantastic! Personally, I think the minor speed drawback of

...

A speed drawback on every import would almost certainly be too high a

price to pay.

Yes, though if the extra check through the rest of the path were only done when the module is found in the cwd, it may not cost much at all. -CHB Maybe this would make a useful excepthook? If an

...

AttributeError is about to be printed to the console, and the failing

object is a module, *then* look through the path and see if it was

shadowed.

ChrisA

_______________________________________________

Python-ideas mailing list -- python-ideas@python.org

To unsubscribe send an email to python-ideas-leave@python.org

https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/S6IXKE...

Code of Conduct: http://python.org/psf/codeofconduct/

--

Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Random832

24 Aug 24 Aug

7:43 a.m.

On Sat, Aug 22, 2020, at 20:17, Chris Angelico wrote:

...

A speed drawback on every import would almost certainly be too high a price to pay.

My proposal would only add an extra check 1) on module load [not on import, so you only pay it once per module, and not at all for built-in modules] 2) only for modules loaded from the first entry in sys.path [script directory / current directory], so it shouldn't add much overhead at all to imports of stdlib or installed packages.

Gustav O

9:33 a.m.

It seems like we all agree that getting such a warning would be optimal. The only real question is how we would make sure that it doesn’t slow down imports more than it helps. I’d gladly see your proposed option become a part of the language. With the mentioned restrictions on when it actually conducts the search, I think the benefits would outweigh the costs. -- Gustav

Gustav O

22 Aug 22 Aug

9:38 a.m.

I think you have some really valid points in your answer, Steven. To me, it seems like we have two main options to improve the message for shadowing modules. I copied part of your and Random832’s answer here, since they summarize it quite nicely: 1. If a module attempts to directly import a module with the same name, and that import fails for any reason, we include a message about shadowing in addition to the normal exception traceback. 2. Always print a warning [not an error] when importing a module from the current directory [the first entry in sys.path] that is also available in any other entry in sys.path I think both methods would add significant information. The second option would help a lot in the cases where the program still works, but doesn’t throw an error (possibly because the accessed objects exist in both modules). Either way, giving a hint that a module might be shadowed by the running program would help significantly. -- Gustav

Steve Barnes

10:18 a.m.

Couldn’t the whole issue potentially be avoided with some meta-modules, e.g.: from STDLIB import sys from STDLIB.os import path from LIBS import pandas as pd from LOCALS import path # This should cause an error from LOCALS import path as mypath # this not This would leave the current import syntax unchanged but allow users to clarify where they are expecting to import from. There is an underused mechanism for doing the same in C/C++ where #include will always be from the system includes but #include “something.h” will not (if your compile includes have a blank path to distinguish). Might be worth some thinking! Steve Barnes Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10 From: Gustav Omailto:gustav@odinger.se Sent: 22 August 2020 10:41 To: python-ideas@python.orgmailto:python-ideas@python.org Subject: [Python-ideas] Re: Improve error message for trying to import itself I think you have some really valid points in your answer, Steven. To me, it seems like we have two main options to improve the message for shadowing modules. I copied part of your and Random832’s answer here, since they summarize it quite nicely: 1. If a module attempts to directly import a module with the same name, and that import fails for any reason, we include a message about shadowing in addition to the normal exception traceback. 2. Always print a warning [not an error] when importing a module from the current directory [the first entry in sys.path] that is also available in any other entry in sys.path I think both methods would add significant information. The second option would help a lot in the cases where the program still works, but doesn’t throw an error (possibly because the accessed objects exist in both modules). Either way, giving a hint that a module might be shadowed by the running program would help significantly. -- Gustav _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MQ3BEV... Code of Conduct: http://python.org/psf/codeofconduct/

1340

Age (days ago)

1343

Last active (days ago)

List overview

Download

26 comments

8 participants

participants (8)

Chris Angelico
Christopher Barker
Greg Ewing
Gustav O
Random832
Serhiy Storchaka
Steve Barnes
Steven D'Aprano

Improve error message for trying to import itself

tags

participants (8)