Skip modules by default in star-import
From the description of the import statement in the language reference [1]:
As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident. For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1] There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example: >>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree <module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'> >>> ElementPath <module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'> Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good. I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules. In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`. [1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem. One could propose to fix this by doing: 1. Imported symbols and locally declared symbols are put in two different dicts 2. Deprecation warning when you import a symbol from the import dict of another module 3. At some future point promote this to an error In this future re-exporting a symbol might be done with: from foo import bar bar = bar There are other details like being able to programmatically access the import dict for example. I'm +1 on this.
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`: from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder It is possible even to use a star import. So this change would break much more code.
I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module). And the length of the deprecation period is tweakable. We can set it for 30 years if we want. / Anders
On 7/26/2019 4:23 PM, Anders Hovmöller wrote:
I personally have dozens of packages where I do "from .submodule import *" in a __init__.py. The stdlib has about 20 such cases. It's a very common pattern when you had a module that became a package and was refactored over time.
And the length of the deprecation period is tweakable. We can set it for 30 years if we want.
I don't see the point of this change, then. I'm okay with making it a style guide issue, or catch it in a linter. I'm -1 on this proposal. There's not enough advantage given the churn it would cause. Eric
Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.) Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems. That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all. / Anders
On Fri, Jul 26, 2019 at 11:51 PM Anders Hovmöller <boxed@killingar.net> wrote:
But the fact that you tried to solve "other problems" means you changed the scope of the discussion in this thread.
Please start a separate thread if you want to discuss the much broader topic of disallowing exporting imported modules as this thread is specifically around `import *`.
26.07.19 23:53, Guido van Rossum пише:
I have tested this change on the stdlib, and it had very small effect. It broke `test_pkg` [1] which tested star-import of package and the `xml.parsers.expat` module [2] which is a thin wrapper around the `pyexpat` module, including its submodules `model` and `errors`. It is not easy to determine the effect on third-party code, but seems that well-known packages has defined `__all__` in they main modules. It may affect small projects which do not care much about linters and `__all__`. But taking to account that the effect on the stdlib (where many modules do not have `__all__`) is small, there is a hope that it is small for other projects too. It may be worth to mention that help() does not show imported module. It shows only submodules. `__all__` has been added to Tkinter recently. [3] Excluding imported modules was not the only reason, there was other problem which can be solved only by `__all__`. [1] https://github.com/python/cpython/blob/master/Lib/test/test_pkg.py [2] https://github.com/python/cpython/blob/master/Lib/xml/parsers/expat.py[3] https://bugs.python.org/issue29446
I think I agree with Serhiy, but I don't think Anders' suggestion is workable since it's very common in __init__.py to do exactly this (import * from local modules in order to re-export them), and you definitely don't want to explitly foo=foo for everything. That's the whole reason you "import *". For example, https://github.com/numpy/numpy/blob/master/numpy/linalg/__init__.py Best, Neil On Friday, July 26, 2019 at 2:53:23 PM UTC-4, Anders Hovmöller wrote:
On 2019-07-26 11:10, Anders Hovmöller wrote:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem. Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter. As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported. The actual problem is that people use star-importing when they shouldn't. I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
If you see many questions it's by definition a problem. Take the questions seriously. It's Confusing at best. We can fix it.
Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter.
I see fairly often and it creates weird cases where searching doesn't find stuff because it's imported from the wrong place. It is bad for readability for no good use imo.
As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported.
OK. How about enforcing that? This seems to be your suggestion?
The actual problem is that people use star-importing when they shouldn't.
That's OPs problem. But that's not mine. I only use import * interactively or in throw away scripts.
I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use.
I'm fine with keeping star import for interactive use only. *shrug* But this also a long deprecation process. / Anders
26.07.19 20:43, Serhiy Storchaka пише:
Opened an issue and a PR: https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263
On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
I like this — but think it’s way too late. It could break a fair bit of code. To the extent that import * is used at all ;-) I, at least, do a lot of importing of modules in __init__.py Simply to make the names available. I don’t encourage import * either, but that doesn’t mean my users don’t use it. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Just because du to the side tracking it is looking like "it might be a nice idea": IMHO, this is not. and I second Brendan's arguments - This fix too litle, and normally just for beginners who happen to use * imports in code that will selfon be put to critical use. (and even if it does, the extra names from "*" won't break it). On the other hand doing from `import submodule` just to make submodule available in a package `__init__` is a very nice practice, and it would make a lot of developers have to review a lot, really lot, of well written software just to add explicit `__all__` declarations. I am as "-1" on this as possible. js -><- On Wed, 18 Sep 2019 at 22:20, Christopher Barker <pythonchb@gmail.com> wrote:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem. One could propose to fix this by doing: 1. Imported symbols and locally declared symbols are put in two different dicts 2. Deprecation warning when you import a symbol from the import dict of another module 3. At some future point promote this to an error In this future re-exporting a symbol might be done with: from foo import bar bar = bar There are other details like being able to programmatically access the import dict for example. I'm +1 on this.
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`: from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder It is possible even to use a star import. So this change would break much more code.
I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module). And the length of the deprecation period is tweakable. We can set it for 30 years if we want. / Anders
On 7/26/2019 4:23 PM, Anders Hovmöller wrote:
I personally have dozens of packages where I do "from .submodule import *" in a __init__.py. The stdlib has about 20 such cases. It's a very common pattern when you had a module that became a package and was refactored over time.
And the length of the deprecation period is tweakable. We can set it for 30 years if we want.
I don't see the point of this change, then. I'm okay with making it a style guide issue, or catch it in a linter. I'm -1 on this proposal. There's not enough advantage given the churn it would cause. Eric
Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.) Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems. That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all. / Anders
On Fri, Jul 26, 2019 at 11:51 PM Anders Hovmöller <boxed@killingar.net> wrote:
But the fact that you tried to solve "other problems" means you changed the scope of the discussion in this thread.
Please start a separate thread if you want to discuss the much broader topic of disallowing exporting imported modules as this thread is specifically around `import *`.
26.07.19 23:53, Guido van Rossum пише:
I have tested this change on the stdlib, and it had very small effect. It broke `test_pkg` [1] which tested star-import of package and the `xml.parsers.expat` module [2] which is a thin wrapper around the `pyexpat` module, including its submodules `model` and `errors`. It is not easy to determine the effect on third-party code, but seems that well-known packages has defined `__all__` in they main modules. It may affect small projects which do not care much about linters and `__all__`. But taking to account that the effect on the stdlib (where many modules do not have `__all__`) is small, there is a hope that it is small for other projects too. It may be worth to mention that help() does not show imported module. It shows only submodules. `__all__` has been added to Tkinter recently. [3] Excluding imported modules was not the only reason, there was other problem which can be solved only by `__all__`. [1] https://github.com/python/cpython/blob/master/Lib/test/test_pkg.py [2] https://github.com/python/cpython/blob/master/Lib/xml/parsers/expat.py[3] https://bugs.python.org/issue29446
I think I agree with Serhiy, but I don't think Anders' suggestion is workable since it's very common in __init__.py to do exactly this (import * from local modules in order to re-export them), and you definitely don't want to explitly foo=foo for everything. That's the whole reason you "import *". For example, https://github.com/numpy/numpy/blob/master/numpy/linalg/__init__.py Best, Neil On Friday, July 26, 2019 at 2:53:23 PM UTC-4, Anders Hovmöller wrote:
On 2019-07-26 11:10, Anders Hovmöller wrote:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem. Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter. As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported. The actual problem is that people use star-importing when they shouldn't. I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
If you see many questions it's by definition a problem. Take the questions seriously. It's Confusing at best. We can fix it.
Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter.
I see fairly often and it creates weird cases where searching doesn't find stuff because it's imported from the wrong place. It is bad for readability for no good use imo.
As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported.
OK. How about enforcing that? This seems to be your suggestion?
The actual problem is that people use star-importing when they shouldn't.
That's OPs problem. But that's not mine. I only use import * interactively or in throw away scripts.
I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use.
I'm fine with keeping star import for interactive use only. *shrug* But this also a long deprecation process. / Anders
26.07.19 20:43, Serhiy Storchaka пише:
Opened an issue and a PR: https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263
On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
I like this — but think it’s way too late. It could break a fair bit of code. To the extent that import * is used at all ;-) I, at least, do a lot of importing of modules in __init__.py Simply to make the names available. I don’t encourage import * either, but that doesn’t mean my users don’t use it. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Just because du to the side tracking it is looking like "it might be a nice idea": IMHO, this is not. and I second Brendan's arguments - This fix too litle, and normally just for beginners who happen to use * imports in code that will selfon be put to critical use. (and even if it does, the extra names from "*" won't break it). On the other hand doing from `import submodule` just to make submodule available in a package `__init__` is a very nice practice, and it would make a lot of developers have to review a lot, really lot, of well written software just to add explicit `__all__` declarations. I am as "-1" on this as possible. js -><- On Wed, 18 Sep 2019 at 22:20, Christopher Barker <pythonchb@gmail.com> wrote:
participants (9)
-
Anders Hovmöller -
Brendan Barnwell -
Brett Cannon -
Christopher Barker -
Eric V. Smith -
Guido van Rossum -
Joao S. O. Bueno -
Neil Girdhar -
Serhiy Storchaka