Add an __exclude_all__ complement to __all__
Hi, Python has an __all__ variable that can be defined in a module to restrict which members of the module should be included in a call `from foo import *`. However specifying which members of the module should be excluded is more difficult. There are three workarounds that I see: 1) Defining __all__ to specify most of the members of the module, except the few that should be excluded. 2) Naming members to be excluded with a leading underscore. e.g. as done here in ctypes: https://github.com/python/cpython/blob/master/Lib/ctypes/__init__.py#L3 3) Manually deleting members from the module. e.g. as done here in numpy: https://github.com/numpy/numpy/blob/master/numpy/core/__init__.py#L52 I think it might be cleaner to also allow an __exclude_all__ variable which will exclude some members from being imported. If both were defined I would imagine __exclude_all__ being applied second. Best, George Harding
On 3/3/21 12:55 PM, George Harding wrote:
Python has an __all__ variable that can be defined in a module to restrict which members of the module should be included in a call `from foo import *`.
The primary purpose these days for `__all__` is to codify a module's API. The *-import is just a happy accident.
However specifying which members of the module should be excluded is more difficult.
And unnecessary -- specify `__all__` so your users know which classes, functions, variables, etc., can be safely used. If it should be excluded, don't put it in `__all__`. -- ~Ethan~
Hi Ethan, I'm not convinced that __all__ is responsible for codifying the api. The contents of __all__, does not stop anyone from importing anything in the module using `from foo import bar`. __all__ is specified in the tutorial as being included for the `from foo import *` case: https://docs.python.org/3/tutorial/modules.html And similar the first result for __all__ on google, is a stack overflow post stating that it is for from foo import *: https://stackoverflow.com/questions/44834/can-someone-explain-all-in-python Best, George On Wed, Mar 3, 2021 at 9:47 PM Ethan Furman <ethan@stoneleaf.us> wrote:
Hi Ethan, I'm sorry, I take that back, that convention was codified in PEP8. https://www.python.org/dev/peps/pep-0008/#id50 Best, George On Wed, Mar 3, 2021 at 10:18 PM George Harding < george.winton.harding@gmail.com> wrote:
On 3/3/21 2:18 PM, George Harding wrote:
__all__ does not stop anyone from importing anything in the module using `from foo import bar`.
That is true, but that is also Python. If you import something not included in `__all__` then you do so at your own risk as it could change or vanish in the next release. -- ~Ethan~
On Thu, Mar 4, 2021 at 7:57 AM George Harding <george.winton.harding@gmail.com> wrote:
I think it might be cleaner to also allow an __exclude_all__ variable which will exclude some members from being imported.
If both were defined I would imagine __exclude_all__ being applied second.
You could implement that yourself: __all__ = {n for n in globals() if not n.startswith("_")} - __exclude_all__ ChrisA
On 2021-03-03 14:06, Chris Angelico wrote:
You could implement that yourself:
__all__ = {n for n in globals() if not n.startswith("_")} - __exclude_all__
Sort of. That will only work if you define __all__ at the end of the file. But usually you want to define it at the beginning as a sort of documentation aid ("this is the public API"). I do think something like __exclude_all__ would be handy. It can be annoying to have to define __all__ just to exclude a few things. But it's not a huge issue. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Thu, Mar 4, 2021 at 9:57 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
Given that it's meant to be program-readable, that shouldn't be too big a deal. You can always put your human-readable public API description in the docstring (where there's room for actual descriptions, not just a list of names).
We are programmers. When there is a problem, we (should!) seek a solution that involves getting the computer to do the work for us. ChrisA
On 3/3/21 2:17 PM, Brendan Barnwell wrote:
[...] usually you want to define [__all__] at the beginning as a sort of documentation aid ("this is the public API").
That is its purpose. :-)
I do think something like __exclude_all__ would be handy. It can be annoying to have to define __all__ just to exclude a few things.
Exclude a few things from what? *-imports? If you're using *-imports that often you are already courting confusion (where did that name come from? why is this name not that function?). I maintain a handful of libraries, all with `__all__`, and only one them is designed to be used with `from ... import *`. The presence or absence of `__all__` has no bearing on named imports. `from blahblah import _xyz` will work in either case. -- ~Ethan~
If you are using __all__ to define your API, then you'll end up needing `from foo import *` if you want to combine modules, e.g.: https://github.com/python/cpython/blob/master/Lib/asyncio/__init__.py So the two need to coexist. *-import does have valid uses. The majority of these cases are within a library, rather than between libraries. Best, George
On 3/3/21 3:38 PM, George Harding wrote:
If you are using __all__ to define your API, then you'll end up needing `from foo import *` if you want to combine modules, e.g.:
https://github.com/python/cpython/blob/master/Lib/asyncio/__init__.py
Absolutely!
So the two need to coexist. *-import does have valid uses.
Agreed. -- ~Ethan~
On Wed, 3 Mar 2021 at 23:59, Brendan Barnwell <brenbarn@brenbarn.net> wrote:
But usually you want to define it at the beginning as a sort of documentation aid ("this is the public API").
This is a little off-topic, but I'm curious, since usually, for public functions and classes, I do __all__ = (Class.__name__, func.__name__, ...) So I have to put it at the end of the module. I do this because if I change the class or function name and I forget to change it in __all__, I get an exception. Furthermore, if there's a module composed by submodules, I usually do from .a import * from .b import * __all__ = a.__all__ + b.__all__ In your opinion, these are good or bad practices?
I certainly will not claim to be the arbitrator of good and bad practices but that seems reasonable enough. It is worth pointing out that it's pretty easy to unit test `__all__` ``` # module/__init__.py __all__ = 'Foo', 'Bar' class Foo: pass ``` ``` # tests/test_imports.py def test_star(): # will raise `AttributeError: module 'module' has not attribute 'Bar'` from module import * ```
from .a import * from .b import *
__all__ = a.__all__ + b.__all__
Assuming you mean: ``` from . import a from . import b from .a import * from .b import * __all__ = a.__all__ + b.__all__ ``` It is fairly unnecessary if you aren't adding more names. but definitely isn't harmful. On Fri, Mar 5, 2021 at 12:11 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On 3/3/21 12:55 PM, George Harding wrote:
Python has an __all__ variable that can be defined in a module to restrict which members of the module should be included in a call `from foo import *`.
The primary purpose these days for `__all__` is to codify a module's API. The *-import is just a happy accident.
However specifying which members of the module should be excluded is more difficult.
And unnecessary -- specify `__all__` so your users know which classes, functions, variables, etc., can be safely used. If it should be excluded, don't put it in `__all__`. -- ~Ethan~
Hi Ethan, I'm not convinced that __all__ is responsible for codifying the api. The contents of __all__, does not stop anyone from importing anything in the module using `from foo import bar`. __all__ is specified in the tutorial as being included for the `from foo import *` case: https://docs.python.org/3/tutorial/modules.html And similar the first result for __all__ on google, is a stack overflow post stating that it is for from foo import *: https://stackoverflow.com/questions/44834/can-someone-explain-all-in-python Best, George On Wed, Mar 3, 2021 at 9:47 PM Ethan Furman <ethan@stoneleaf.us> wrote:
Hi Ethan, I'm sorry, I take that back, that convention was codified in PEP8. https://www.python.org/dev/peps/pep-0008/#id50 Best, George On Wed, Mar 3, 2021 at 10:18 PM George Harding < george.winton.harding@gmail.com> wrote:
On 3/3/21 2:18 PM, George Harding wrote:
__all__ does not stop anyone from importing anything in the module using `from foo import bar`.
That is true, but that is also Python. If you import something not included in `__all__` then you do so at your own risk as it could change or vanish in the next release. -- ~Ethan~
On Thu, Mar 4, 2021 at 7:57 AM George Harding <george.winton.harding@gmail.com> wrote:
I think it might be cleaner to also allow an __exclude_all__ variable which will exclude some members from being imported.
If both were defined I would imagine __exclude_all__ being applied second.
You could implement that yourself: __all__ = {n for n in globals() if not n.startswith("_")} - __exclude_all__ ChrisA
On 2021-03-03 14:06, Chris Angelico wrote:
You could implement that yourself:
__all__ = {n for n in globals() if not n.startswith("_")} - __exclude_all__
Sort of. That will only work if you define __all__ at the end of the file. But usually you want to define it at the beginning as a sort of documentation aid ("this is the public API"). I do think something like __exclude_all__ would be handy. It can be annoying to have to define __all__ just to exclude a few things. But it's not a huge issue. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Thu, Mar 4, 2021 at 9:57 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
Given that it's meant to be program-readable, that shouldn't be too big a deal. You can always put your human-readable public API description in the docstring (where there's room for actual descriptions, not just a list of names).
We are programmers. When there is a problem, we (should!) seek a solution that involves getting the computer to do the work for us. ChrisA
On 3/3/21 2:17 PM, Brendan Barnwell wrote:
[...] usually you want to define [__all__] at the beginning as a sort of documentation aid ("this is the public API").
That is its purpose. :-)
I do think something like __exclude_all__ would be handy. It can be annoying to have to define __all__ just to exclude a few things.
Exclude a few things from what? *-imports? If you're using *-imports that often you are already courting confusion (where did that name come from? why is this name not that function?). I maintain a handful of libraries, all with `__all__`, and only one them is designed to be used with `from ... import *`. The presence or absence of `__all__` has no bearing on named imports. `from blahblah import _xyz` will work in either case. -- ~Ethan~
If you are using __all__ to define your API, then you'll end up needing `from foo import *` if you want to combine modules, e.g.: https://github.com/python/cpython/blob/master/Lib/asyncio/__init__.py So the two need to coexist. *-import does have valid uses. The majority of these cases are within a library, rather than between libraries. Best, George
On 3/3/21 3:38 PM, George Harding wrote:
If you are using __all__ to define your API, then you'll end up needing `from foo import *` if you want to combine modules, e.g.:
https://github.com/python/cpython/blob/master/Lib/asyncio/__init__.py
Absolutely!
So the two need to coexist. *-import does have valid uses.
Agreed. -- ~Ethan~
On Wed, 3 Mar 2021 at 23:59, Brendan Barnwell <brenbarn@brenbarn.net> wrote:
But usually you want to define it at the beginning as a sort of documentation aid ("this is the public API").
This is a little off-topic, but I'm curious, since usually, for public functions and classes, I do __all__ = (Class.__name__, func.__name__, ...) So I have to put it at the end of the module. I do this because if I change the class or function name and I forget to change it in __all__, I get an exception. Furthermore, if there's a module composed by submodules, I usually do from .a import * from .b import * __all__ = a.__all__ + b.__all__ In your opinion, these are good or bad practices?
I certainly will not claim to be the arbitrator of good and bad practices but that seems reasonable enough. It is worth pointing out that it's pretty easy to unit test `__all__` ``` # module/__init__.py __all__ = 'Foo', 'Bar' class Foo: pass ``` ``` # tests/test_imports.py def test_star(): # will raise `AttributeError: module 'module' has not attribute 'Bar'` from module import * ```
from .a import * from .b import *
__all__ = a.__all__ + b.__all__
Assuming you mean: ``` from . import a from . import b from .a import * from .b import * __all__ = a.__all__ + b.__all__ ``` It is fairly unnecessary if you aren't adding more names. but definitely isn't harmful. On Fri, Mar 5, 2021 at 12:11 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
participants (7)
-
Abe Dillon -
Brendan Barnwell -
Caleb Donovick -
Chris Angelico -
Ethan Furman -
George Harding -
Marco Sulla