Mailman 3 Skip modules by default in star-import - Python-ideas

newer
Discussions-To PEP-0604:...

Skip modules by default in star-import

older
Simpler syntax for async generators

Serhiy Storchaka

26 Jul 2019 26 Jul '19

11:43 a.m.

From the description of the import statement in the language reference [1]:

...

The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).

As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident. For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1] There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example: >>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree >>> ElementPath Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good. I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules. In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`. [1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446

Show replies by date

Anders Hovmöller

26 Jul 26 Jul

12:10 p.m.

This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem. One could propose to fix this by doing: 1. Imported symbols and locally declared symbols are put in two different dicts 2. Deprecation warning when you import a symbol from the import dict of another module 3. At some future point promote this to an error In this future re-exporting a symbol might be done with: from foo import bar bar = bar There are other details like being able to programmatically access the import dict for example. I'm +1 on this.

...

On 26 Jul 2019, at 19:43, Serhiy Storchaka wrote:

From the description of the import statement in the language reference [1]:

...
The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).

As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident.

For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1]

There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example:

...
...
...
from xml.etree import * ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined import xml.etree.ElementTree from xml.etree import * ElementTree ElementPath

Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good.

I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.

In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`.

[1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/AKWL7C... Code of Conduct: http://python.org/psf/codeofconduct/

Serhiy Storchaka

12:34 p.m.

26.07.19 21:10, Anders Hovmöller пише:

...

This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`: from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder It is possible even to use a star import. So this change would break much more code.

Anders Hovmöller

12:50 p.m.

...

On 26 Jul 2019, at 20:34, Serhiy Storchaka wrote:

26.07.19 21:10, Anders Hovmöller пише:

...
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:

from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder

It is possible even to use a star import.

So this change would break much more code.

I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly: from foo import bar bar = bar For "from x import *" you'd need to iterate over __import_dict__ (or whatever we call it) and set them all. / Anders

Eric V. Smith

12:58 p.m.

On 7/26/2019 2:50 PM, Anders Hovmöller wrote:

...

...
On 26 Jul 2019, at 20:34, Serhiy Storchaka wrote:

26.07.19 21:10, Anders Hovmöller пише:

...
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:

from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder

It is possible even to use a star import.

So this change would break much more code.

I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly:

from foo import bar bar = bar

I think breaking a whole lot of existing code is a bad idea. This just isn't a big problem in practice. We've typically said that the documentation is the final word on what's public, and that we can change anything else. Eric

Anders Hovmöller

2:23 p.m.

...

On 26 Jul 2019, at 20:58, Eric V. Smith wrote:

On 7/26/2019 2:50 PM, Anders Hovmöller wrote:

...
...
On 26 Jul 2019, at 20:34, Serhiy Storchaka wrote:

26.07.19 21:10, Anders Hovmöller пише:

...
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:

from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder

It is possible even to use a star import.

So this change would break much more code. I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly: from foo import bar bar = bar

I think breaking a whole lot of existing code is a bad idea.

I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module). And the length of the deprecation period is tweakable. We can set it for 30 years if we want. / Anders

Eric V. Smith

2:34 p.m.

On 7/26/2019 4:23 PM, Anders Hovmöller wrote:

...

...
On 26 Jul 2019, at 20:58, Eric V. Smith wrote:

On 7/26/2019 2:50 PM, Anders Hovmöller wrote:

...
...
On 26 Jul 2019, at 20:34, Serhiy Storchaka wrote:

26.07.19 21:10, Anders Hovmöller пише:

...
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:

from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder

It is possible even to use a star import.

So this change would break much more code. I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly: from foo import bar bar = bar

I think breaking a whole lot of existing code is a bad idea.

I don't think it's that common that code would break for this. If it does the fixed code is very likely better (except in libs where some API is exposed but implented in another module).

I personally have dozens of packages where I do "from .submodule import *" in a __init__.py. The stdlib has about 20 such cases. It's a very common pattern when you had a module that became a package and was refactored over time.

...

And the length of the deprecation period is tweakable. We can set it for 30 years if we want.

I don't see the point of this change, then. I'm okay with making it a style guide issue, or catch it in a linter. I'm -1 on this proposal. There's not enough advantage given the churn it would cause. Eric

Guido van Rossum

2:53 p.m.

Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.) Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Anders Hovmöller

27 Jul 27 Jul

12:49 a.m.

...

On 26 Jul 2019, at 22:53, Guido van Rossum wrote:

Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)

Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads.

I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems. That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all. / Anders

Brett Cannon

29 Jul 29 Jul

12:59 p.m.

On Fri, Jul 26, 2019 at 11:51 PM Anders Hovmöller wrote:

...

...
On 26 Jul 2019, at 22:53, Guido van Rossum wrote:

Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)

Anders then stole Serhiy's thread to advocate fora much more radical idea. That's not good netiquette. That idea is going to break a lot more code than Serhiy's. But my key message here is to Anders: stay on topic or start a new thread. You're welcome to discuss your idea in a separate thread. But don't steal existing threads.

I respectfully disagree. Sure I advocated for a more radical idea, but one that solves the original problem and also other problems.

But the fact that you tried to solve "other problems" means you changed the scope of the discussion in this thread.

...

That doesn't seem like stealing a thread to me but offering another perspective on the problem at hand. Taking a step back and getting a bigger picture seems like a good thing. We're just throwing ideas around after all.

Please start a separate thread if you want to discuss the much broader topic of disallowing exporting imported modules as this thread is specifically around `import *`.

Serhiy Storchaka

27 Jul 27 Jul

2:21 a.m.

26.07.19 23:53, Guido van Rossum пише:

...

Serhiy proposed a relatively minor change to the behavior of `import *` in the absence of __all__. This sounds like an idea we could try though we should have a look at the implications for various well-known packages. It is also a relatively minor problem that he's trying to solve, so it's not worth breaking lots of stuff over. (We can just add an __all__ to Tkinter that contains the desired list.)

I have tested this change on the stdlib, and it had very small effect. It broke `test_pkg` [1] which tested star-import of package and the `xml.parsers.expat` module [2] which is a thin wrapper around the `pyexpat` module, including its submodules `model` and `errors`. It is not easy to determine the effect on third-party code, but seems that well-known packages has defined `__all__` in they main modules. It may affect small projects which do not care much about linters and `__all__`. But taking to account that the effect on the stdlib (where many modules do not have `__all__`) is small, there is a hope that it is small for other projects too. It may be worth to mention that help() does not show imported module. It shows only submodules. `__all__` has been added to Tkinter recently. [3] Excluding imported modules was not the only reason, there was other problem which can be solved only by `__all__`. [1] https://github.com/python/cpython/blob/master/Lib/test/test_pkg.py [2] https://github.com/python/cpython/blob/master/Lib/xml/parsers/expat.py[3] https://bugs.python.org/issue29446

Neil Girdhar

19 Sep 19 Sep

11:06 a.m.

I think I agree with Serhiy, but I don't think Anders' suggestion is workable since it's very common in __init__.py to do exactly this (import * from local modules in order to re-export them), and you definitely don't want to explitly foo=foo for everything. That's the whole reason you "import *". For example, https://github.com/numpy/numpy/blob/master/numpy/linalg/__init__.py Best, Neil On Friday, July 26, 2019 at 2:53:23 PM UTC-4, Anders Hovmöller wrote:

...

...
On 26 Jul 2019, at 20:34, Serhiy Storchaka javascript:> wrote:

26.07.19 21:10, Anders Hovmöller пише:

...
This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I do not think that this is always a problem. It is common to refactor the code by defining names in submodules and then importing them in the main module. For example, in `json/__init__.py`:

from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder

It is possible even to use a star import.

So this change would break much more code.

I believe I covered that in my last email. I'll repeat it here for clarity: if you indent to re-export you can do that explicitly:

from foo import bar bar = bar

For "from x import *" you'd need to iterate over __import_dict__ (or whatever we call it) and set them all.

/ Anders

_______________________________________________ Python-ideas mailing list -- python...@python.org javascript: To unsubscribe send an email to python-id...@python.org javascript: https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EP3PK3... Code of Conduct: http://python.org/psf/codeofconduct/

Brendan Barnwell

26 Jul 26 Jul

1:50 p.m.

On 2019-07-26 11:10, Anders Hovmöller wrote:

...

This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem. Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter. As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported. The actual problem is that people use star-importing when they shouldn't. I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

Anders Hovmöller

2:19 p.m.

...

On 26 Jul 2019, at 21:50, Brendan Barnwell wrote:

...
On 2019-07-26 11:10, Anders Hovmöller wrote: This doesn't really solve the problem imo. Imported symbols shouldn't be i portable elsewhere. Not by import * or explicitly. That's the problem.

I see a fair number of questions about this on StackOverflow and I just don't understand why people feel it's a problem.

If you see many questions it's by definition a problem. Take the questions seriously. It's Confusing at best. We can fix it.

...

Who cares what names are accessible in an imported module's namespace? If you do `import foo` and then `foo.blah` happens to be some oddball name you didn't expect to be in there, so what? Just don't use it and it doesn't matter.

I see fairly often and it creates weird cases where searching doesn't find stuff because it's imported from the wrong place. It is bad for readability for no good use imo.

...

As for star-import, it should NEVER be used except in cases where the module you're star-importing explicitly defines an __all__ and documents that it's okay to star-import it, in which case the issue is moot since the __all__ will prevent any unexpected names from being imported.

OK. How about enforcing that? This seems to be your suggestion?

...

The actual problem is that people use star-importing when they shouldn't.

That's OPs problem. But that's not mine. I only use import * interactively or in throw away scripts.

...

I don't think there's any need to change the language to make star imports easier or more convenient; if anything we should be trying to move away from their use.

I'm fine with keeping star import for interactive use only. *shrug* But this also a long deprecation process. / Anders

Serhiy Storchaka

18 Sep 18 Sep

10:03 a.m.

26.07.19 20:43, Serhiy Storchaka пише:

...

I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.

Opened an issue and a PR: https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263

Christopher Barker

7:18 p.m.

On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka wrote:

...

26.07.19 20:43, Serhiy Storchaka пише:

...
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.

I like this — but think it’s way too late. It could break a fair bit of code. To the extent that import * is used at all ;-) I, at least, do a lot of importing of modules in __init__.py Simply to make the names available. I don’t encourage import * either, but that doesn’t mean my users don’t use it. -CHB

...

Opened an issue and a PR:

https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263 _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CEC2N4... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Joao S. O. Bueno

19 Sep 19 Sep

10:24 a.m.

Just because du to the side tracking it is looking like "it might be a nice idea": IMHO, this is not. and I second Brendan's arguments - This fix too litle, and normally just for beginners who happen to use * imports in code that will selfon be put to critical use. (and even if it does, the extra names from "*" won't break it). On the other hand doing from `import submodule` just to make submodule available in a package `__init__` is a very nice practice, and it would make a lot of developers have to review a lot, really lot, of well written software just to add explicit `__all__` declarations. I am as "-1" on this as possible. js -><- On Wed, 18 Sep 2019 at 22:20, Christopher Barker wrote:

...

On Wed, Sep 18, 2019 at 9:04 AM Serhiy Storchaka wrote:

...
26.07.19 20:43, Serhiy Storchaka пише:

...
I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules.

I like this — but think it’s way too late. It could break a fair bit of code.

To the extent that import * is used at all ;-)

I, at least, do a lot of importing of modules in __init__.py Simply to make the names available.

I don’t encourage import * either, but that doesn’t mean my users don’t use it.

-CHB

...
...
Opened an issue and a PR:

https://bugs.python.org/issue38215 https://github.com/python/cpython/pull/16263 _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CEC2N4... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD

Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YZFB4T... Code of Conduct: http://python.org/psf/codeofconduct/

1680

Age (days ago)

1735

Last active (days ago)

List overview

Download

16 comments

9 participants

participants (9)

Anders Hovmöller
Brendan Barnwell
Brett Cannon
Christopher Barker
Eric V. Smith
Guido van Rossum
Joao S. O. Bueno
Neil Girdhar
Serhiy Storchaka

Skip modules by default in star-import

tags

participants (9)