Improve import mechanism for circular imports
Consider the following directory structure (Python 3): [test] main.py [tree] __init__.py # empty branch.py root.py main.py: print('main: Entered the module') print('main: from tree import root')from tree import root # this first finished importing module 'tree.root'# then creates local name 'root' which references that imported module# Why not creating local name 'root' at the same time when sys.modules['tree.root'] is created?# If import fails:# - either delete the attribute 'root'# - or leave it - how it affects the runtime?from tree import branch print('main: Creating a root and a leaf attached to it') _root = root.Root()_branch = branch.Branch(_root) branch.py: print('tree.branch: Entered the module') import tree, sysprint("tree.branch: 'root' in dir(tree) ->", 'root' in dir(tree))print("tree.branch: 'tree.root' in sys.modules ->", 'tree.root' in sys.modules) print('tree.branch: from tree import root')#from tree import root # ImportError: cannot import name root import tree.root # workaroundroot = sys.modules['tree.root'] # though name 'root' does not exist yet in module 'tree', sys.modules['root.tree'] already does print('tree.branch: Defining class branch.Branch()') class Branch(): def __init__(self, _parent): assert isinstance(_parent, root.Root), 'Pass a `Root` instance' root.py: print('tree.root: Entered the module') print('tree.root: from tree import branch')from tree import branch print('tree.root: Defining class Root()') class Root(): def attach(self, _branch): assert isinstance(_branch, branch.Branch), 'Pass a `Branch` instance' self.branch = _branch Running it: vic@wic:~/projects/test$ python3 main.py main: Entered the modulemain: from tree import roottree.root: Entered the moduletree.root: from tree import branchtree.branch: Entered the moduletree.branch: 'root' in dir(tree) -> Falsetree.branch: 'tree.root' in sys.modules -> Truetree.branch: from tree import roottree.branch: Defining class branch.Branch()tree.root: Defining class Root()main: Creating a root and a leaf attached to it So, There are circular imports in this example code. Currently, `root = sys.modules['tree.root']` hack in branch.py works. Wouldn't it be useful to create attribute `root` in `main` at the same time `sys.modules['tree.root']` is created when doing `from tree import root` in main? This would solve more complex cases with when circular imports are involved, without applying such hacks. Thank you -- *Victor *
Hi! On Tue, Apr 10, 2012 at 12:30:34PM +0300, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
My opinion is - restructure code to avoid circular import instead of hacking import machinery. Why does a submodule import the entire package instead of importing just root? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Apr 10, 2012 at 12:55 PM, Oleg Broytman <phd@phdru.name> wrote:
My opinion is - restructure code to avoid circular import instead of hacking import machinery.
It's not feasible sometimes. See: http://stackoverflow.com/questions/1556387/circular-import-dependency-in-pyt... Yes, they could be considered the same package. But if this results in a massively huge file then it's impractical. I agree that frequently, circular dependencies mean the design should be thought through again. But there ARE some design patterns where it's appropriate (and where merging the files together would result in a huge file) so I think it's dogmatic to say that the packages should either be combined or the design should be re-evaluated. – Matthew Lund Dec 1 '11 at 21:49
Why does a submodule import the entire package instead of importing just root?
import tree, sys print("tree.branch: 'root' in dir(tree) ->", 'root' in dir(tree)) print("tree.branch: 'tree.root' in sys.modules ->", 'tree.root' in sys.modules) Ignore this part - my fault. -- *Victor*
On Tue, Apr 10, 2012 at 02:24:18PM +0300, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
I don't see anything interesting there. Without deeper knowledge about the code I'd recommend to import b.d in a/__init__.py before importing c.
I didn't and do not recommend merging code into one huge module. Call me dogmatic but I recommend to refactor and move common parts to avoid circular import. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Apr 10, 2012 at 3:03 PM, Oleg Broytman <phd@phdru.name> wrote:
JFY here is an example how to make it in general case. http://codereview.appspot.com/5449109/
See: http://stackoverflow.com/questions/1556387/circular-import-dependency-in-pyt...
Do you really fighting with this specific case? There is a solution to that with delayed import.
Oleg Broytman schrieb am Tue, 10. Apr 2012, um 16:03:15 +0400:
I actually did ran into cases where this was not possible. (I just wrote a lengthy description of such a case, but I figured it wasn't too helpful, so it's not included here.) A point to consider is that there are cases of circular imports that used to work fine with implicit relative imports. When using explicit relative imports though, they would stop working -- see [1] for a minimal example demonstrating this problem. [1]: http://stackoverflow.com/questions/6351805/cyclic-module-dependencies-and-re... This issue can be easily overcome with function-level imports, but some people don't like function-level imports either. The same issue turned up when porting the Python Imaging Library to Python 3. PIL uses implicit relative, circular imports which have to be turned into function-level imports to work properly on Python 3, see [2] for details. [2]: https://github.com/sloonz/pil-py3k/pull/2 Cheers, Sven
On Tue, Apr 10, 2012 at 02:56:36PM +0100, Sven Marnach <sven@marnach.net> wrote:
This issue can be easily overcome with function-level imports, but some people don't like function-level imports either.
Can I say I doubt it's a good reason to change Python?
Was there any major problem in fixing that? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Oleg Broytman schrieb am Tue, 10. Apr 2012, um 18:11:05 +0400:
Sorry, I actuually didn't mean to argue in favour of any proposal. I just meant to point out issues relevant to this thread that some people probably are not aware of.
This was meant as an example that these issues arise in practice, even in libraries that can hardly be cosidered obscure. Cheers, Sven
On 10 April 2012 10:30, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
Why does "tree" even exist? Why not just have 2 top-level modules "root" and "branch". ("It's only an example" isn't a good answer - it isn't a good example if it doesn't demonstrate why "tree" is needed for your real use case). Can you explain the purpose of "tree" in your real code? Also, having things happen at import time (as opposed to simply defining classes, functions, etc) is not good form, precisely because partially-imported modules are in an odd state, and problems like this can easily arise if you don't know what you are doing. Instead of giving a made-up example, if you describe what you are trying to achieve, I'm fairly certain someone here (or more likely on python-list) could show you a better way to do it, without needing circular imports. Paul.
On Tue, Apr 10, 2012 at 9:57 PM, Paul Moore <p.f.moore@gmail.com> wrote:
There isn't actually a *strong* philosophical objection to improving the circular import support. It's just a sufficiently hard problem that the rote answer is "nobody has cared enough about the problem to come up with a fix that works properly, is backwards compatible and doesn't hurt the performance of regular imports". The relevant tracker issue is http://bugs.python.org/issue992389 (yes, that issue is approaching it's 8th birthday later this year) With import.c going away soon (courtesy of the migration to importlib as the main import implementation), it may become easier to devise a solution (or at least generate a better error message). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi! On Tue, Apr 10, 2012 at 12:30:34PM +0300, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
My opinion is - restructure code to avoid circular import instead of hacking import machinery. Why does a submodule import the entire package instead of importing just root? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Apr 10, 2012 at 12:55 PM, Oleg Broytman <phd@phdru.name> wrote:
My opinion is - restructure code to avoid circular import instead of hacking import machinery.
It's not feasible sometimes. See: http://stackoverflow.com/questions/1556387/circular-import-dependency-in-pyt... Yes, they could be considered the same package. But if this results in a massively huge file then it's impractical. I agree that frequently, circular dependencies mean the design should be thought through again. But there ARE some design patterns where it's appropriate (and where merging the files together would result in a huge file) so I think it's dogmatic to say that the packages should either be combined or the design should be re-evaluated. – Matthew Lund Dec 1 '11 at 21:49
Why does a submodule import the entire package instead of importing just root?
import tree, sys print("tree.branch: 'root' in dir(tree) ->", 'root' in dir(tree)) print("tree.branch: 'tree.root' in sys.modules ->", 'tree.root' in sys.modules) Ignore this part - my fault. -- *Victor*
On Tue, Apr 10, 2012 at 02:24:18PM +0300, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
I don't see anything interesting there. Without deeper knowledge about the code I'd recommend to import b.d in a/__init__.py before importing c.
I didn't and do not recommend merging code into one huge module. Call me dogmatic but I recommend to refactor and move common parts to avoid circular import. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Apr 10, 2012 at 3:03 PM, Oleg Broytman <phd@phdru.name> wrote:
JFY here is an example how to make it in general case. http://codereview.appspot.com/5449109/
See: http://stackoverflow.com/questions/1556387/circular-import-dependency-in-pyt...
Do you really fighting with this specific case? There is a solution to that with delayed import.
Oleg Broytman schrieb am Tue, 10. Apr 2012, um 16:03:15 +0400:
I actually did ran into cases where this was not possible. (I just wrote a lengthy description of such a case, but I figured it wasn't too helpful, so it's not included here.) A point to consider is that there are cases of circular imports that used to work fine with implicit relative imports. When using explicit relative imports though, they would stop working -- see [1] for a minimal example demonstrating this problem. [1]: http://stackoverflow.com/questions/6351805/cyclic-module-dependencies-and-re... This issue can be easily overcome with function-level imports, but some people don't like function-level imports either. The same issue turned up when porting the Python Imaging Library to Python 3. PIL uses implicit relative, circular imports which have to be turned into function-level imports to work properly on Python 3, see [2] for details. [2]: https://github.com/sloonz/pil-py3k/pull/2 Cheers, Sven
On Tue, Apr 10, 2012 at 02:56:36PM +0100, Sven Marnach <sven@marnach.net> wrote:
This issue can be easily overcome with function-level imports, but some people don't like function-level imports either.
Can I say I doubt it's a good reason to change Python?
Was there any major problem in fixing that? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Oleg Broytman schrieb am Tue, 10. Apr 2012, um 18:11:05 +0400:
Sorry, I actuually didn't mean to argue in favour of any proposal. I just meant to point out issues relevant to this thread that some people probably are not aware of.
This was meant as an example that these issues arise in practice, even in libraries that can hardly be cosidered obscure. Cheers, Sven
On 10 April 2012 10:30, Victor Varvariuc <victor.varvariuc@gmail.com> wrote:
Why does "tree" even exist? Why not just have 2 top-level modules "root" and "branch". ("It's only an example" isn't a good answer - it isn't a good example if it doesn't demonstrate why "tree" is needed for your real use case). Can you explain the purpose of "tree" in your real code? Also, having things happen at import time (as opposed to simply defining classes, functions, etc) is not good form, precisely because partially-imported modules are in an odd state, and problems like this can easily arise if you don't know what you are doing. Instead of giving a made-up example, if you describe what you are trying to achieve, I'm fairly certain someone here (or more likely on python-list) could show you a better way to do it, without needing circular imports. Paul.
On Tue, Apr 10, 2012 at 9:57 PM, Paul Moore <p.f.moore@gmail.com> wrote:
There isn't actually a *strong* philosophical objection to improving the circular import support. It's just a sufficiently hard problem that the rote answer is "nobody has cared enough about the problem to come up with a fix that works properly, is backwards compatible and doesn't hurt the performance of regular imports". The relevant tracker issue is http://bugs.python.org/issue992389 (yes, that issue is approaching it's 8th birthday later this year) With import.c going away soon (courtesy of the migration to importlib as the main import implementation), it may become easier to devise a solution (or at least generate a better error message). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (6)
-
anatoly techtonik -
Nick Coghlan -
Oleg Broytman -
Paul Moore -
Sven Marnach -
Victor Varvariuc