[Adding python-dev. I'm quoting the entire original message.]
On Thu, Mar 19, 2009 at 6:40 PM, Fredrik Lundh
wrote: PS. Is it just me, or is import broken in 3.0? Consider this:
$ more package\a.py print("in a")
import b
def a(): print("here")
def main(): b.b()
$ more package\b.py print("in b")
import a
def b(): a.a()
Under 2.X, this prints "in a" "in b" and "here", as expected. Under 3.0, using the "from . import" form for relative imports, it bombs out with a:
in a in b Traceback (most recent call last): File "main.py", line 1, in <module> from package import a File "package/a.py", line 3, in <module> from . import b File "package\b.py", line 3, in <module> from . import a ImportError: cannot import name a
Sure, it's a recursive import, but there's no recursive dependency here - nobody will access the module contents until the main program calls the library. What am I missing?
On Mon, Mar 30, 2009 at 5:44 PM, Guido van Rossum
I reproduced this, but it seems to have more to do with "from . import ..." than with the Python version. If I add the "from ." before each of the imports, "python -c 'import p.a' " fails with roughly the above traceback for any version of Python that supports this syntax, while without that it passes for any 2.x.
If I use the "from ." syntax in a.py but not in b.py, "import p.a" passes but "import p.b" fails.
I'll see if anyone present at the sprints has a clue.
Made some progress. Anything using "from <whatever> import b" (where <whatever> is either '.' or 'p') will fail when b's import is not completed. OTOH using "import p.b" works. I reduced it to: p/a.py == "from p import b" p/b.py == "import a" python -c "import p.b" The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful. At least, this is my hypothesis, thinking about it -- I might look at the code later. :-) The most portable solution is to avoid "from...import" and instead write something like import p.b as b -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, Mar 30, 2009 at 6:17 PM, Guido van Rossum
[Adding python-dev. I'm quoting the entire original message.]
On Thu, Mar 19, 2009 at 6:40 PM, Fredrik Lundh
wrote: PS. Is it just me, or is import broken in 3.0? Consider this:
$ more package\a.py print("in a")
import b
def a(): print("here")
def main(): b.b()
$ more package\b.py print("in b")
import a
def b(): a.a()
Under 2.X, this prints "in a" "in b" and "here", as expected. Under 3.0, using the "from . import" form for relative imports, it bombs out with a:
in a in b Traceback (most recent call last): File "main.py", line 1, in <module> from package import a File "package/a.py", line 3, in <module> from . import b File "package\b.py", line 3, in <module> from . import a ImportError: cannot import name a
Sure, it's a recursive import, but there's no recursive dependency here - nobody will access the module contents until the main program calls the library. What am I missing?
On Mon, Mar 30, 2009 at 5:44 PM, Guido van Rossum
wrote: I reproduced this, but it seems to have more to do with "from . import ..." than with the Python version. If I add the "from ." before each of the imports, "python -c 'import p.a' " fails with roughly the above traceback for any version of Python that supports this syntax, while without that it passes for any 2.x.
If I use the "from ." syntax in a.py but not in b.py, "import p.a" passes but "import p.b" fails.
I'll see if anyone present at the sprints has a clue.
Made some progress. Anything using "from <whatever> import b" (where <whatever> is either '.' or 'p') will fail when b's import is not completed. OTOH using "import p.b" works. I reduced it to:
p/a.py == "from p import b" p/b.py == "import a" python -c "import p.b"
The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful.
At least, this is my hypothesis, thinking about it -- I might look at the code later. :-)
The most portable solution is to avoid "from...import" and instead write something like
import p.b as b
So it turns out that "from X import Y" compiles into this bytecode: 0 LOAD_CONST 0 (-1) 3 LOAD_CONST 1 (('Y',)) 6 IMPORT_NAME 0 (X) 9 IMPORT_FROM 1 (Y) 12 STORE_NAME 1 (Y) 15 POP_TOP The first three opcodes (through IMPORT_NAME) call __import__('X', None, None, ('Y',)) and push the result on top of the stack; this result is the toplevel package X. The IMPORT_FROM opcode is essentially a getattr call that turns an AttributeError into an ImportError exception. I changed p/a.py into p = __import__('p', None, None, ['b']) print(p.b) and confirmed that it fails on the print() line in p.b. Does anyone feel that this ought to be fixed? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Mon, Mar 30, 2009 at 6:17 PM, Guido van Rossum
wrote: [Adding python-dev. I'm quoting the entire original message.]
On Thu, Mar 19, 2009 at 6:40 PM, Fredrik Lundh
wrote: PS. Is it just me, or is import broken in 3.0? Consider this: [snip]
Sure, it's a recursive import, but there's no recursive dependency here - nobody will access the module contents until the main program calls the library. What am I missing?
Problems with recursive imports are a perennial topic on Python list. A common suggestion is to refactor to avoid them.
On Mon, Mar 30, 2009 at 5:44 PM, Guido van Rossum
wrote: I reproduced this, but it seems to have more to do with "from . import ..." than with the Python version. If I add the "from ." before each of the imports, "python -c 'import p.a' " fails with roughly the above traceback for any version of Python that supports this syntax, while without that it passes for any 2.x.
If I use the "from ." syntax in a.py but not in b.py, "import p.a" passes but "import p.b" fails.
I'll see if anyone present at the sprints has a clue. Made some progress. Anything using "from <whatever> import b" (where <whatever> is either '.' or 'p') will fail when b's import is not completed. OTOH using "import p.b" works. I reduced it to:
p/a.py == "from p import b" p/b.py == "import a" python -c "import p.b"
The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful.
At least, this is my hypothesis, thinking about it -- I might look at the code later. :-)
The most portable solution is to avoid "from...import"
When doing recursive imports (it seems to work fine otherwise).
and instead write something like
import p.b as b
So it turns out that "from X import Y" compiles into this bytecode:
0 LOAD_CONST 0 (-1) 3 LOAD_CONST 1 (('Y',)) 6 IMPORT_NAME 0 (X) 9 IMPORT_FROM 1 (Y) 12 STORE_NAME 1 (Y) 15 POP_TOP
The first three opcodes (through IMPORT_NAME) call __import__('X', None, None, ('Y',)) and push the result on top of the stack; this result is the toplevel package X. The IMPORT_FROM opcode is essentially a getattr call that turns an AttributeError into an ImportError exception. I changed p/a.py into
p = __import__('p', None, None, ['b']) print(p.b)
and confirmed that it fails on the print() line in p.b.
If I understand, you are saying that from x import y is equivalent in effect to import x y = x.y del x except that the binding of 'x' never happens. This is pretty much what the (3.0.1) doc says: "The from form does not bind the module name: it goes through the list of identifiers, looks each one of them up in the module found in step (1), and binds the name in the local namespace to the object thus found. " where step 1 is the (completed) initialization of the module. So it seems to me that the behavior Fredrik noticed is implied by the doc. It could be main plainer though. I have not read Brett's proposed import doc yet.
Does anyone feel that this ought to be fixed?
What would be the new doc? Terry Jan Reedy
Terry Reedy wrote:
Guido van Rossum wrote:
The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful.
I remember a pydev discussion on this point.
Terry Reedy wrote:
Terry Reedy wrote:
Guido van Rossum wrote:
The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful.
I remember a pydev discussion on this point.
The quote from Fredrik in Guido's original message sounded familiar. Checking my nosy list on the tracker brought me to the following issue: http://bugs.python.org/issue992389 Jim Fulton's example in that tracker issue shows that with a bit of creativity you can provoke this behaviour *without* using a from-style import. Torsten Bronger later brought up the same issue that Fredrik did - it prevents some kinds of explicit relative import that look like they should be fine. It's starting to look to me like the "set in advance and delete on failure" approach taken with sys.modules may make sense for the attributes in the parent package after all. Given the way that import is defined, I suspect that is easier said than done though... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jim Fulton's example in that tracker issue shows that with a bit of creativity you can provoke this behaviour *without* using a from-style import. Torsten Bronger later brought up the same issue that Fredrik did - it prevents some kinds of explicit relative import that look like they should be fine.
I haven't been following this very closely, but if there's something that's making absolute and relative imports behave differently, I think it should be fixed. The only difference between an absolute and relative import of the same module should be the way you specify the module. -- Greg
Greg Ewing wrote:
Nick Coghlan wrote:
Jim Fulton's example in that tracker issue shows that with a bit of creativity you can provoke this behaviour *without* using a from-style import. Torsten Bronger later brought up the same issue that Fredrik did - it prevents some kinds of explicit relative import that look like they should be fine.
I haven't been following this very closely, but if there's something that's making absolute and relative imports behave differently, I think it should be fixed. The only difference between an absolute and relative import of the same module should be the way you specify the module.
That's exactly the problem though. Because of the difference in the way the target module is specified, the way it is looked up is different: 'import a.b.c' will look in sys.modules for "a.b.c", succeed and work, even if "a.b.c" is in the process of being imported. 'from a.b import c' (or 'from . import c' in a subpackage of "a.b") will only look in sys.modules for "a.b", and then look on that object for a "c" attribute. The cached "a.b.c' module in sys.modules is ignored. It doesn't appear to be an impossible problem to solve, but it probably isn't going to be easy to fix in a backwards compatible way. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
'import a.b.c' will look in sys.modules for "a.b.c", succeed and work, even if "a.b.c" is in the process of being imported.
'from a.b import c' (or 'from . import c' in a subpackage of "a.b") will only look in sys.modules for "a.b", and then look on that object for a "c" attribute. The cached "a.b.c' module in sys.modules is ignored.
Hasn't 'from a.b import c' always been that way, though? Is the problem just that relative imports make it easier to run into this behaviour, or has something about the way imports work changed? -- Greg
Greg Ewing wrote:
Nick Coghlan wrote:
'import a.b.c' will look in sys.modules for "a.b.c", succeed and work, even if "a.b.c" is in the process of being imported.
'from a.b import c' (or 'from . import c' in a subpackage of "a.b") will only look in sys.modules for "a.b", and then look on that object for a "c" attribute. The cached "a.b.c' module in sys.modules is ignored.
Hasn't 'from a.b import c' always been that way, though? Is the problem just that relative imports make it easier to run into this behaviour, or has something about the way imports work changed?
The former - while a few things have obviously changed in this area due to PEP 328 and PEP 366, I don't believe any of that affected this aspect of the semantics (the issue I linked dates from 2004!). Instead, I'm pretty sure implicit relative imports use the 'import a.b.c' rules and hence work in situations where explicit relative imports now fail. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Terry Reedy wrote:
Terry Reedy wrote:
Guido van Rossum wrote:
The reason seems to be that until the outermost import (in this case p.b) is completed, while sys.modules has the (incomplete) modules 'p', 'p.a' and 'p.b', the attributes p.a and p.b aren't added until after their import is completed. Which it isn't during recursive import. Apparently 'from <anything> import <something>' looks for the <something> attribute in the <parent> object. This is because "from...import" can also be used to import objects other than modules (e.g. "from M import C"). I'm guessing that setting the attribute is delayed until the import is totally complete, because upon a failed import we remove the half-imported module object from sys.modules, but apparently we didn 't want to be in the business of removing the attribute from the parent package, so that's only set after the import is deemed successful.
I remember a pydev discussion on this point.
A half-baked idea that I just posted to the tracker item: What if the import semantics were adjusted so that, as a last gasp effort before bailing out with an ImportError, the import process checked sys.modules again with the full module name? Would that be enough to eliminate this error? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (4)
-
Greg Ewing
-
Guido van Rossum
-
Nick Coghlan
-
Terry Reedy