Skip modules by default in star-import
From the description of the import statement in the language reference [1]:
The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident.
For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1]
There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example:
>>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree <module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'> >>> ElementPath <module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'>
Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good.
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.
In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`.
[1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
One could propose to fix this by doing:
1. Imported symbols and locally declared symbols are put in two different dicts 2. Deprecation warning when you import a symbol from the import dict of another module 3. At some future point promote this to an error
In this future re-exporting a symbol might be done with:
from foo import bar bar = bar
There are other details like being able to programmatically access the import dict for example.
I'm +1 on this.
On 26 Jul 2019, at 19:43, Serhiy Storchaka storchaka@gmail.com wrote:
From the description of the import statement in the language reference [1]:
The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident.
For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1]
There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example:
from xml.etree import * ElementTree
Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined
ElementPath
Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined
import xml.etree.ElementTree from xml.etree import * ElementTree
<module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'>
ElementPath
<module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'>
Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good.
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.
In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`.
[1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/AKWL7C... Code of Conduct: http://python.org/psf/codeofconduct/
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder
It is possible even to use a star import.
So this change would break much more code.
On 26 Jul 2019, at 20:34, Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder
It is possible even to use a star import.
So this change would break much more code.
I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly:
from foo import bar bar = bar
For "from x import *" you'd need to iterate over __import_dict__ (or whatever we call it) and set them all.
/ Anders
On 7/26/2019 2:50 PM, Anders Hovmöller wrote:
On 26 Jul 2019, at 20:34, Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoderIt is possible even to use a star import.
So this change would break much more code.
I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly:
from foo import bar bar = bar
I think breaking a whole lot of existing code is a bad idea. This just isn't a big problem in practice. We've typically said that the documentation is the final word on what's public, and that we can change anything else.
Eric
On 26 Jul 2019, at 20:58, Eric V. Smith eric@trueblade.com wrote:
On 7/26/2019 2:50 PM, Anders Hovmöller wrote:
On 26 Jul 2019, at 20:34, Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder
It is possible even to use a star import.
So this change would break much more code.
I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly: from foo import bar bar = bar
I think breaking a whole lot of existing code is a bad idea.
I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module).
And the length of the deprecation period is tweakable. We can set it for 30 years if we want.
/ Anders
On 7/26/2019 4:23 PM, Anders Hovmöller wrote:
On 26 Jul 2019, at 20:58, Eric V. Smith eric@trueblade.com wrote:
On 7/26/2019 2:50 PM, Anders Hovmöller wrote:
On 26 Jul 2019, at 20:34, Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoderIt is possible even to use a star import.
So this change would break much more code.
I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly: from foo import bar bar = bar
I think breaking a whole lot of existing code is a bad idea.
I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module).
I personally have dozens of packages where I do "from .submodule import *" in a __init__.py. The stdlib has about 20 such cases. It's a very common pattern when you had a module that became a package and was refactored over time.
And the length of the deprecation period is tweakable. We can set it for 30 years if we want.
I don't see the point of this change, then. I'm okay with making it a style guide issue, or catch it in a linter.
I'm -1 on this proposal. There's not enough advantage given the churn it would cause.
Eric
Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)
Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads.
On 26 Jul 2019, at 22:53, Guido van Rossum guido@python.org wrote:
Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)
Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads.
I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems. That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all.
/ Anders
On Fri, Jul 26, 2019 at 11:51 PM Anders Hovmöller boxed@killingar.net wrote:
On 26 Jul 2019, at 22:53, Guido van Rossum guido@python.org wrote:
Serhiy proposed a relatively minor change to the behavior of `import *`
in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)
Anders then stole Serhiy's thread to advocate fora much more radical
idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads.
I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems.
But the fact that you tried to solve "other problems" means you changed the scope of the discussion in this thread.
That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all.
Please start a separate thread if you want to discuss the much broader topic of disallowing exporting imported modules as this thread is specifically around `import *`.
26.07.19 23:53, Guido van Rossum пише:
Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)
I have tested this change on the stdlib, and it had very small effect. It broke `test_pkg` [1] which tested star-import of package and the `xml.parsers.expat` module [2] which is a thin wrapper around the `pyexpat` module, including its submodules `model` and `errors`.
It is not easy to determine the effect on third-party code, but seems that well-known packages has defined `__all__` in they main modules. It may affect small projects which do not care much about linters and `__all__`. But taking to account that the effect on the stdlib (where many modules do not have `__all__`) is small, there is a hope that it is small for other projects too.
It may be worth to mention that help() does not show imported module. It shows only submodules.
`__all__` has been added to Tkinter recently. [3] Excluding imported modules was not the only reason, there was other problem which can be solved only by `__all__`.
[1] https://github.com/python/cpython/blob/master/Lib/test/test_pkg.py [2] https://github.com/python/cpython/blob/master/Lib/xml/parsers/expat.py%5B3] https://bugs.python.org/issue29446
I think I agree with Serhiy, but I don't think Anders' suggestion is workable since it's very common in __init__.py to do exactly this (import * from local modules in order to re-export them), and you definitely don't want to explitly foo=foo for everything. That's the whole reason you "import *". For example, https://github.com/numpy/numpy/blob/master/numpy/linalg/__init__.py
Best,
Neil
On Friday, July 26, 2019 at 2:53:23 PM UTC-4, Anders Hovmöller wrote:
On 26 Jul 2019, at 20:34, Serhiy Storchaka <stor...@gmail.com
javascript:> wrote:
26.07.19 21:10, Anders Hovmöller пише:
This doesn't really solve the problem imo. Imported symbols shouldn't
be i portable elsewhere. Not by import * or explicitly. That's the problem.
I do not think that this is always a problem. It is common to refactor
the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:
from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder
It is possible even to use a star import.
So this change would break much more code.
I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly:
from foo import bar bar = bar
For "from x import *" you'd need to iterate over __import_dict__ (or whatever we call it) and set them all.
/ Anders
Python-ideas mailing list -- python...@python.org javascript: To unsubscribe send an email to python-id...@python.org javascript: https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EP3PK3... Code of Conduct: http://python.org/psf/codeofconduct/
On 2019-07-26 11:10, Anders Hovmöller wrote:
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem. Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter.
As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported.
The actual problem is that people use star-importing when they shouldn't. I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use.
On 26 Jul 2019, at 21:50, Brendan Barnwell brenbarn@brenbarn.net wrote:
On 2019-07-26 11:10, Anders Hovmöller wrote: This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.
I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem.
If you see many questions it's by definition a problem. Take the questions seriously. It's Confusing at best. We can fix it.
Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter.
I see fairly often and it creates weird cases where searching doesn't find stuff because it's imported from the wrong place. It is bad for readability for no good use imo.
As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported.
OK. How about enforcing that? This seems to be your suggestion?
The actual problem is that people use star-importing when they shouldn't.
That's OPs problem. But that's not mine. I only use import * interactively or in throw away scripts.
I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use.
I'm fine with keeping star import for interactive use only. *shrug* But this also a long deprecation process.
/ Anders
26.07.19 20:43, Serhiy Storchaka пише:
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.
Opened an issue and a PR:
https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263
On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 20:43, Serhiy Storchaka пише:
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.
I like this — but think it’s way too late. It could break a fair bit of code.
To the extent that import * is used at all ;-)
I, at least, do a lot of importing of modules in __init__.py Simply to make the names available.
I don’t encourage import * either, but that doesn’t mean my users don’t use it.
-CHB
Opened an issue and a PR:
https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263 _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CEC2N4... Code of Conduct: http://python.org/psf/codeofconduct/
Just because du to the side tracking it is looking like "it might be a nice idea": IMHO, this is not. and I second Brendan's arguments -
This fix too litle, and normally just for beginners who happen to use * imports in code that will selfon be put to critical use. (and even if it does, the extra names from "*" won't break it).
On the other hand doing from `import submodule` just to make submodule available in a package `__init__` is a very nice practice, and it would make a lot of developers have to review a lot, really lot, of well written software just to add explicit `__all__` declarations.
I am as "-1" on this as possible.
js -><-
On Wed, 18 Sep 2019 at 22:20, Christopher Barker pythonchb@gmail.com wrote:
On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka storchaka@gmail.com wrote:
26.07.19 20:43, Serhiy Storchaka пише:
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.
I like this — but think it’s way too late. It could break a fair bit of code.
To the extent that import * is used at all ;-)
I, at least, do a lot of importing of modules in __init__.py Simply to make the names available.
I don’t encourage import * either, but that doesn’t mean my users don’t use it.
-CHB
Opened an issue and a PR:
https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263 _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CEC2N4... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YZFB4T... Code of Conduct: http://python.org/psf/codeofconduct/
participants (9)
-
Anders Hovmöller -
Brendan Barnwell -
Brett Cannon -
Christopher Barker -
Eric V. Smith -
Guido van Rossum -
Joao S. O. Bueno -
Neil Girdhar -
Serhiy Storchaka