After discussion on python-ideas, it looks this PEP moves towards a favorable decision. For a recent discussion see https://mail.python.org/pipermail/python-ideas/2017-November/047806.html. The PEP is available at https://www.python.org/dev/peps/pep-0562/ The most important recent change is the addition of __dir__, as proposed by Guido.
Here is the full text:
+++++++++++++++++++++
PEP: 562 Title: Module __getattr__ and __dir__ Author: Ivan Levkivskyi levkivskyi@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017
Abstract ========
It is proposed to support a ``__getattr__`` and ``__dir__`` functions defined +on modules to provide basic customization of module attribute access.
Rationale =========
It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or replacing the ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example::
# lib.py
from warnings import warn
deprecated_names = ["old_function", ...]
def _deprecated_old_function(arg, other): ...
def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}")
# main.py
from lib import old_function # Works, but emits the warning
Another widespread use case for ``__getattr__`` would be lazy submodule imports. Consider a simple example::
# lib/__init__.py
import importlib
__all__ = ['submod', ...]
def __getattr__(name): if name in __all__: return importlib.import_module("." + name, __name__) raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# lib/submod.py
print("Submodule loaded") class HeavyClass: ...
# main.py
import lib lib.submodule.HeavyClass # prints "Submodule loaded"
There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization. An additional motivation for this proposal is that PEP 484 already defines the use of module ``__getattr__`` for this purpose in Python stub files, see [1]_.
In addition, to allow modifying result of a ``dir()`` call on a module to show deprecated and other dynamically generated attributes, it is proposed to support module level ``__dir__`` function. For example::
# lib.py
deprecated_names = ["old_function", ...] __all__ = ["new_function_one", "new_function_two", ...]
def new_function_one(arg, other): ... def new_function_two(arg, other): ...
def __dir__(): return sorted(__all__ + deprecated_names)
# main.py
import lib
dir(lib) # prints ["new_function_one", "new_function_two", "old_function", ...]
Specification =============
The ``__getattr__`` function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
The ``__dir__`` function should accept no arguments, and return a list of strings that represents the names accessible on module::
def __dir__() -> List[str]: ...
If present, this function overrides the standard ``dir()`` search on a module.
The reference implementation for this PEP can be found in [2]_.
Backwards compatibility and impact on performance =================================================
This PEP may break code that uses module level (global) names ``__getattr__`` and ``__dir__``. The performance implications of this PEP are minimal, since ``__getattr__`` is called only for missing attributes.
Discussion ==========
Note that the use of module ``__getattr__`` requires care to keep the referred objects pickleable. For example, the ``__name__`` attribute of a function should correspond to the name with which it is accessible via ``__getattr__``::
def keep_pickleable(func): func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
One should be also careful to avoid recursion as one would do with a class level ``__getattr__``.
References ==========
.. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files)
.. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files)
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
14.11.17 22:34, Ivan Levkivskyi пише:
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Backwards compatibility and impact on performance
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
def keep_pickleable(func): func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
On 15 November 2017 at 08:43, Serhiy Storchaka storchaka@gmail.com wrote:
14.11.17 22:34, Ivan Levkivskyi пише:
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Good point!
Backwards compatibility and impact on performance
=================================================
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it. I will add a note to the PEP about this.
def keep_pickleable(func):
func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func@keep_pickleable def _deprecated_old_function(arg, other): ...
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
But in which module these helpers should live?
-- Ivan
15.11.17 12:53, Ivan Levkivskyi пише:
On 15 November 2017 at 08:43, Serhiy Storchaka <storchaka@gmail.com mailto:storchaka@gmail.com> wrote:
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.Good point!
And please document idiomatic way of using a module global with triggering __getattr__. For example if you want to use a lazy loaded submodule.
sys.modules[__name__].foobar
or
from . import foobar
The difference between them that the latter sets the module attribute, thus __getattr__ will be called only once.
Backwards compatibility and impact on performance ================================================= What is affect on pydoc, word completion, inspect, pkgutil, unittest?This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it.. I will add a note to the PEP about this.
This problem is not new, since it was possible to replace a module with a module subclass with overridden __getattr__ and __dir__ before, but now this problem can occur more often.
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.But in which module these helpers should live?
Good question. lazy_import() could be added in importlib (or importlib.util?). The helper that just adds deprecation on importing a name, could be added in importlib too. But I think that it would be better if the deprecated() helper will also create a wrapper that raises a deprecation warning on the use of deprecated function. It could be added in the warnings or functools modules.
I would add also a more general lazy_initialized(). It is something like cached module property. Executes the specified code on first use, and cache the result as a module attribute.
In all these cases the final __getattr__ method should be automatically constructed from different chunks. At the end it could call a user supplied __getattr__. Or maybe the module method __getattr__ should look first at special registry before calling the instance attribute __getattr__()?
def ModuleType.__getattr__(self, name): if name in self.__properties__: call self.__properties__[name]() elif '__getattr__' in self.__dict__: call self.__dict__['__getattr__'](name) else: raise AttributeError
I'm wondering if the __set_name__ mechanism can be extended to modules. What if call the __set_name__() method for all items in a module dict after finishing importing the module?
I think it's reasonable for the PEP to include some examples, consequences and best practices. I don't think it's reasonable for the PEP to also define the API and implementation of helper functions that might be added once the mechanisms are in place. Those are better developed as 3rd party packages first.
On Wed, Nov 15, 2017 at 3:59 AM, Serhiy Storchaka storchaka@gmail.com wrote:
15.11.17 12:53, Ivan Levkivskyi пише:
On 15 November 2017 at 08:43, Serhiy Storchaka <storchaka@gmail.com mailto:storchaka@gmail.com> wrote:
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.Good point!
And please document idiomatic way of using a module global with triggering __getattr__. For example if you want to use a lazy loaded submodule.
sys.modules[__name__].foobaror
from . import foobarThe difference between them that the latter sets the module attribute, thus __getattr__ will be called only once.
Backwards compatibility and impact on performance================================================= What is affect on pydoc, word completion, inspect, pkgutil, unittest?This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it.. I will add a note to the PEP about this.
This problem is not new, since it was possible to replace a module with a module subclass with overridden __getattr__ and __dir__ before, but now this problem can occur more often.
I would create more standard helpers (for deprecation, for lazyimporting). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.But in which module these helpers should live?
Good question. lazy_import() could be added in importlib (or importlib.util?). The helper that just adds deprecation on importing a name, could be added in importlib too. But I think that it would be better if the deprecated() helper will also create a wrapper that raises a deprecation warning on the use of deprecated function. It could be added in the warnings or functools modules.
I would add also a more general lazy_initialized(). It is something like cached module property. Executes the specified code on first use, and cache the result as a module attribute.
In all these cases the final __getattr__ method should be automatically constructed from different chunks. At the end it could call a user supplied __getattr__. Or maybe the module method __getattr__ should look first at special registry before calling the instance attribute __getattr__()?
def ModuleType.__getattr__(self, name): if name in self.__properties__: call self.__properties__[name]() elif '__getattr__' in self.__dict__: call self.__dict__['__getattr__'](name) else: raise AttributeErrorI'm wondering if the __set_name__ mechanism can be extended to modules. What if call the __set_name__() method for all items in a module dict after finishing importing the module?
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi levkivskyi@gmail.com wrote: [..]
Rationale
It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or replacing the ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example::
# lib.py
from warnings import warn
deprecated_names = ["old_function", ...]
def _deprecated_old_function(arg, other): ...
def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}")
# main.py
from lib import old_function # Works, but emits the warning
Deprecating functions is already possible, so I assume the reason for this would be performance? If so, are you sure this would help for performance? Deprecating module attributes / globals is indeed difficult to do at present. This PEP would allow deprecation warnings for accessing attributes, which is nice! However, as thread-unsafe as it is, many modules use module attributes to configure the state of the module. In that case, the user is more likely to *set* the attribute that to *get* it. Is this outside the scope of the PEP?
[..]
There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization.
I'm not surprised that the comparison is in favor of this PEP ;-).
[..]
Specification
The ``__getattr__`` function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this.
[..]
Discussion
Note that the use of module ``__getattr__`` requires care to keep the referred objects pickleable. For example, the ``__name__`` attribute of a function should correspond to the name with which it is accessible via ``__getattr__``::
def keep_pickleable(func): func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
One should be also careful to avoid recursion as one would do with a class level ``__getattr__``.
Off-topic: In some sense, I'm happy to hear something about pickleability. But in some sense not.
I think there are three kinds of people regarding pickleability:
1. Those who don't care about anything being pickleable
2. Those who care about some things being picklable
3. Those who care about all things being picklable
Personally, I'd like to belong to group 3, but because group 3 cannot even attempt to coexist with groups 1 and 2, I actually belong to group 1 most of the time.
––Koos
References
.. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files)
.. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files)
Copyright
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ k7hoven%40gmail.com
On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:
Rationale
[...] It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method
[...]
Specification
The ``__getattr__`` function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this.
I don't see how. This is exactly the way normal __getattr__ works.
-- ~Ethan~
On Wed, Nov 15, 2017 at 8:02 PM, Ethan Furman ethan@stoneleaf.us wrote:
On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:
Rationale
=========
[...] It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method
[...]
Specification
=============
The ``__getattr__`` function at the module level should accept one
argument
which is the name of an attribute and return the computed value or raise
an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this.
I don't see how. This is exactly the way normal __getattr__ works.
Oh sorry, I think I put this email together too quickly. I was writing down a bunch of thoughts I had earlier but hadn't written down. I think I was mixing this up in my head with overriding __getitem__ for the module namespace dict and __class_getitem__ from PEP 560, which only gets called if the metaclass doesn't implement __getitem__ (IIRC).
But I did have another thought related to this. I was wondering whether the lack of passing the module to the methods as `self` would harm future attempts to generalize these ideas.
-- Koos
participants (5)
-
Ethan Furman -
Guido van Rossum -
Ivan Levkivskyi -
Koos Zevenhoven -
Serhiy Storchaka