Allow 'import star' with namespaces
Hello, I posted this on python-dev, but was told that this is the more appropriate list. Currently if I do: $ import pkg Then all of the public subpackages/submodules are not automatically pulled into the 'pkg' namespace. I can do: $ from pkg import * To get all of the public subpackages/submodules, but that dumps them all into the current namespace. Why not allow: $ import pkg.* This would allow easier interactive use (by eliminating the need to import individual subpackages/submodules) while keeping the 'pkg' namespace around. Thanks, Brendan Moloney
On 06.05.2011 09:20, dag.odenhall@gmail.com wrote:
And that's for a reason: it's not easy (I think it's even impossible, because for example individual submodules can change __path__) to determine all importable submodules of a package. So ``import pkg.*`` would not have any behavior other than ``import pkg``. Georg
dag.odenhall@gmail.com wrote:
I like this idea, except it's inconsistent with from-import-star, the latter which does *not* get you sub-packages or modules.
Georg Brandl [g.brandl@gmx.net] wrote:
So ``import pkg.*`` would not have any behavior other than ``import pkg``.
When I said all _public_ sub-packages and modules I was referring to those listed in the __all__ attribute of 'pkg'. Thus it would behave in the exact same way as from-import-star except you don't pollute the current namespace. Brendan
On Fri, May 6, 2011 at 5:12 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
The idea is to be able to do operate witha single import when submodules would have to be implicited imported - like xml.etree.ElementTree : [gwidion@powerpuff ~]$ python Python 2.6.1 (r261:67515, Apr 12 2009, 04:14:16) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Ethan Furman [ethan@stoneleaf.us] wrote:
I'm not catching the vision -- could you put together a short example that would illustrate?
The motivation is really just for interactive usage (much like the current from-import-star). If 'pkg' contains a number of sub-packages/modules that take a while to import, it makes sense to not automatically import them into the 'pkg' namespace (in the pkg.__init__ module). Putting the sub-package/module names into the __all__ list gives interactive users the ability to import everything in one go using from-import-star. Unfortunately the from-import-star usage pollutes the current namespace, and thus its use is discouraged. So really the vision is that developers can make their packages convenient for interactive use (by setting the __all__ attribute) without requiring users to use a discouraged language feature or making regular import of the package slow. Brendan
On Fri, May 6, 2011 at 2:12 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
He's saying that the package would be imported like normal. Then all "public" sub-modules of the package would automatically imported and bound to the namespace of the object that resulted from the import of the package. The trickery is that __all__ in the __init__.py would change meaning somewhat, and, do you bind the submodules into the package's module object or something else? If you have a list of the submodules you want imported then you can already accomplish this: import parent for mod in parent.__all_submodules__: __import__("parent.{}".format(mod)) Of course, this does not bind the submodules to the namespace of the package module, but I suppose you could try that with one more step. I am not sure of the specific import mechanism with regards to name binding, but that would seem to be a conflict with the way imported names for submodules are bound. -eric ~Ethan~
On 6 May 2011 21:52, Eric Snow <ericsnowcurrently@gmail.com> wrote:
There is no means of determining what submodules of a package exist. Check PEP 302 for details - finders find modules ant they can do so any way they like - there's nothing in the protocol to enumerate subpackages, so you can't do it (if faced with a general PEP 302 finder). Paul.
On 7 May 2011 01:38, Guido van Rossum <guido@python.org> wrote:
The point is that the pkg should use __all__ to declare what submodules exist. That's what it was invented for!
Hmm, OK. I missed that. But how would that work? p1/__init__.py: __all__ = ['p2', 'foo'] def foo(): print "p1.foo" p1/p2/__init__.py: __all__ = ['foo'] def foo(): print "p1.foo" If I import p1, p1.__all__ shows me that p2 and foo are public. p1.foo exists and I can tell it's not a module. p1.p2 doesn't exist in the p1 namespace at the moment, so how do I tell that I need to import it? Just assume all nonexistent names are subpackages, and import them? That doesn't seem like a very robust approach. A proof of concept in the form of a Python implementation (as a function) would help me understand, I guess. (But I still doubt that even if it's implementable, the feature is much practical use...) Paul.
On Sat, May 7, 2011 at 1:58 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Do whatever "from pkg import *" does today. Though the recursive application is new. I think (if we do this) it should be recursive. The implementation is straightforward, though the consequences may not be (think cyclic imports).
It deviates from "import what you use" for sure. OTOH it is a better alternative to "from pkg import *" because it does not pollute the namespace. I believe Java users are used to this. -- --Guido van Rossum (python.org/~guido)
On Sat, May 7, 2011 at 6:52 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Of course, this does not bind the submodules to the namespace of the package module
It actually does, as binding the submodule name in the parent package namespace is part of the responsibility of __import__():
This is one of the reasons circular imports are such a pain - we pre-bind them in sys.modules, and remove them again if the import fails, but we don't currently do that in the parent package namespace, so circular imports sometimes work and sometime break depending not only on which names are accessed but also *how* they're accessed (e.g. in a/b/c.py, "import a.b.c" will work, "import a.b.c; c = a.b.c" will fail with AttributeError and "from a.b import c" will fail with ImportError).
Nope, it's basically the same as what happens automatically when the modules are imported normally. Indeed, as near as I can tell, this request amounts to asking for syntactic sugar that does something roughly along the lines of: def _subnames(pkg_name, subnames): for subname in subnames: yield ".".join(pkg_name, subname) def import_all(pkg): try: pkg_all = pkg.__all__ except AttributeError: pass else: names = list(_subnames(pkg.__name__, pkg_all)) for name in names: mod = importlib.import_module(name) try: mod_all = mod.__all__ except AttributeError: pass else: names.extend(_subnames(mod.__name__, mod_all) I can see a case being made to provide that as a function in pkgutil (or perhaps importlib itself), but I don't see any reason to give it dedicated syntax. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, May 10, 2011 at 3:02 AM, Guido van Rossum <guido@python.org> wrote:
The relevant bug is still open: http://bugs.python.org/issue992389 My recollection is that the division of responsibility between the core import code and PEP 302 loaders gets a little confused on this point (although I don't recall if that's a real confusion or just an artefact of the structure of the legacy import code). It will hopefully be a little easier to fix once importlib takes over from import.c and the pre-PEP 302 legacy stuff goes away. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, May 9, 2011 at 8:04 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Well, dang it. Not sure how I missed this before: $ python3
$ python3
So the sub-module name binding mechanism is simply to bind the package module and then bind the submodules to it. However, "import temp.mod as something_else" and "from temp import mod" don't do this, which makes sense. This is one of the reasons circular imports are such a pain - we
This works as long as __all__ only contains submodule names, right?
+1 -eric
On Tue, May 10, 2011 at 4:55 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Not quite - both of the latter options change the name binding behaviour in the *current* module, but temp.mod will be set to the imported module regardless. It's part of the import process, whereas the namebinding in the current module happens later (the underlying complexity of all this is why importlib.import_module() was added to replace direct invocation of __import__(). The latter has quite a weird signature in order to support the various incarnations of the import statement). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 May 2011 22:12, Ethan Furman <ethan@stoneleaf.us> wrote:
If you're going to require listing in __all__ anyway, you might as well use what already works: import the modules in the package, and you can then import the package and access the modules as attributes: pkg/__init__.py: from . import mod script.py: import pkg pkg.mod #=> pkg/mod.py
On 06.05.2011 09:20, dag.odenhall@gmail.com wrote:
And that's for a reason: it's not easy (I think it's even impossible, because for example individual submodules can change __path__) to determine all importable submodules of a package. So ``import pkg.*`` would not have any behavior other than ``import pkg``. Georg
dag.odenhall@gmail.com wrote:
I like this idea, except it's inconsistent with from-import-star, the latter which does *not* get you sub-packages or modules.
Georg Brandl [g.brandl@gmx.net] wrote:
So ``import pkg.*`` would not have any behavior other than ``import pkg``.
When I said all _public_ sub-packages and modules I was referring to those listed in the __all__ attribute of 'pkg'. Thus it would behave in the exact same way as from-import-star except you don't pollute the current namespace. Brendan
On Fri, May 6, 2011 at 5:12 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
The idea is to be able to do operate witha single import when submodules would have to be implicited imported - like xml.etree.ElementTree : [gwidion@powerpuff ~]$ python Python 2.6.1 (r261:67515, Apr 12 2009, 04:14:16) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Ethan Furman [ethan@stoneleaf.us] wrote:
I'm not catching the vision -- could you put together a short example that would illustrate?
The motivation is really just for interactive usage (much like the current from-import-star). If 'pkg' contains a number of sub-packages/modules that take a while to import, it makes sense to not automatically import them into the 'pkg' namespace (in the pkg.__init__ module). Putting the sub-package/module names into the __all__ list gives interactive users the ability to import everything in one go using from-import-star. Unfortunately the from-import-star usage pollutes the current namespace, and thus its use is discouraged. So really the vision is that developers can make their packages convenient for interactive use (by setting the __all__ attribute) without requiring users to use a discouraged language feature or making regular import of the package slow. Brendan
On Fri, May 6, 2011 at 2:12 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
He's saying that the package would be imported like normal. Then all "public" sub-modules of the package would automatically imported and bound to the namespace of the object that resulted from the import of the package. The trickery is that __all__ in the __init__.py would change meaning somewhat, and, do you bind the submodules into the package's module object or something else? If you have a list of the submodules you want imported then you can already accomplish this: import parent for mod in parent.__all_submodules__: __import__("parent.{}".format(mod)) Of course, this does not bind the submodules to the namespace of the package module, but I suppose you could try that with one more step. I am not sure of the specific import mechanism with regards to name binding, but that would seem to be a conflict with the way imported names for submodules are bound. -eric ~Ethan~
On 6 May 2011 21:52, Eric Snow <ericsnowcurrently@gmail.com> wrote:
There is no means of determining what submodules of a package exist. Check PEP 302 for details - finders find modules ant they can do so any way they like - there's nothing in the protocol to enumerate subpackages, so you can't do it (if faced with a general PEP 302 finder). Paul.
On 7 May 2011 01:38, Guido van Rossum <guido@python.org> wrote:
The point is that the pkg should use __all__ to declare what submodules exist. That's what it was invented for!
Hmm, OK. I missed that. But how would that work? p1/__init__.py: __all__ = ['p2', 'foo'] def foo(): print "p1.foo" p1/p2/__init__.py: __all__ = ['foo'] def foo(): print "p1.foo" If I import p1, p1.__all__ shows me that p2 and foo are public. p1.foo exists and I can tell it's not a module. p1.p2 doesn't exist in the p1 namespace at the moment, so how do I tell that I need to import it? Just assume all nonexistent names are subpackages, and import them? That doesn't seem like a very robust approach. A proof of concept in the form of a Python implementation (as a function) would help me understand, I guess. (But I still doubt that even if it's implementable, the feature is much practical use...) Paul.
On Sat, May 7, 2011 at 1:58 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Do whatever "from pkg import *" does today. Though the recursive application is new. I think (if we do this) it should be recursive. The implementation is straightforward, though the consequences may not be (think cyclic imports).
It deviates from "import what you use" for sure. OTOH it is a better alternative to "from pkg import *" because it does not pollute the namespace. I believe Java users are used to this. -- --Guido van Rossum (python.org/~guido)
On Sat, May 7, 2011 at 6:52 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Of course, this does not bind the submodules to the namespace of the package module
It actually does, as binding the submodule name in the parent package namespace is part of the responsibility of __import__():
This is one of the reasons circular imports are such a pain - we pre-bind them in sys.modules, and remove them again if the import fails, but we don't currently do that in the parent package namespace, so circular imports sometimes work and sometime break depending not only on which names are accessed but also *how* they're accessed (e.g. in a/b/c.py, "import a.b.c" will work, "import a.b.c; c = a.b.c" will fail with AttributeError and "from a.b import c" will fail with ImportError).
Nope, it's basically the same as what happens automatically when the modules are imported normally. Indeed, as near as I can tell, this request amounts to asking for syntactic sugar that does something roughly along the lines of: def _subnames(pkg_name, subnames): for subname in subnames: yield ".".join(pkg_name, subname) def import_all(pkg): try: pkg_all = pkg.__all__ except AttributeError: pass else: names = list(_subnames(pkg.__name__, pkg_all)) for name in names: mod = importlib.import_module(name) try: mod_all = mod.__all__ except AttributeError: pass else: names.extend(_subnames(mod.__name__, mod_all) I can see a case being made to provide that as a function in pkgutil (or perhaps importlib itself), but I don't see any reason to give it dedicated syntax. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, May 10, 2011 at 3:02 AM, Guido van Rossum <guido@python.org> wrote:
The relevant bug is still open: http://bugs.python.org/issue992389 My recollection is that the division of responsibility between the core import code and PEP 302 loaders gets a little confused on this point (although I don't recall if that's a real confusion or just an artefact of the structure of the legacy import code). It will hopefully be a little easier to fix once importlib takes over from import.c and the pre-PEP 302 legacy stuff goes away. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, May 9, 2011 at 8:04 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Well, dang it. Not sure how I missed this before: $ python3
$ python3
So the sub-module name binding mechanism is simply to bind the package module and then bind the submodules to it. However, "import temp.mod as something_else" and "from temp import mod" don't do this, which makes sense. This is one of the reasons circular imports are such a pain - we
This works as long as __all__ only contains submodule names, right?
+1 -eric
On Tue, May 10, 2011 at 4:55 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Not quite - both of the latter options change the name binding behaviour in the *current* module, but temp.mod will be set to the imported module regardless. It's part of the import process, whereas the namebinding in the current module happens later (the underlying complexity of all this is why importlib.import_module() was added to replace direct invocation of __import__(). The latter has quite a weird signature in order to support the various incarnations of the import statement). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 May 2011 22:12, Ethan Furman <ethan@stoneleaf.us> wrote:
If you're going to require listing in __all__ anyway, you might as well use what already works: import the modules in the package, and you can then import the package and access the modules as attributes: pkg/__init__.py: from . import mod script.py: import pkg pkg.mod #=> pkg/mod.py
participants (10)
-
Benjamin Peterson -
Brendan Moloney -
dag.odenhall＠gmail.com -
Eric Snow -
Ethan Furman -
Georg Brandl -
Guido van Rossum -
Joao S. O. Bueno -
Nick Coghlan -
Paul Moore