After discussion on python-ideas, it looks this PEP moves towards a
favorable decision. For a recent discussion see
https://mail.python.org/pipermail/python-ideas/2017-November/047806.html.
The PEP is available at https://www.python.org/dev/peps/pep-0562/
The most important recent change is the addition of __dir__, as proposed by
Guido.
Here is the full text:
+++++++++++++++++++++
PEP: 562
Title: Module __getattr__ and __dir__
Author: Ivan Levkivskyi
14.11.17 22:34, Ivan Levkivskyi пише:
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Backwards compatibility and impact on performance =================================================
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
def keep_pickleable(func): func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
On 15 November 2017 at 08:43, Serhiy Storchaka
14.11.17 22:34, Ivan Levkivskyi пише:
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Good point!
Backwards compatibility and impact on performance
=================================================
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it. I will add a note to the PEP about this.
def keep_pickleable(func):
func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
But in which module these helpers should live? -- Ivan
15.11.17 12:53, Ivan Levkivskyi пише:
On 15 November 2017 at 08:43, Serhiy Storchaka
mailto:storchaka@gmail.com> wrote: It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Good point!
And please document idiomatic way of using a module global with triggering __getattr__. For example if you want to use a lazy loaded submodule. sys.modules[__name__].foobar or from . import foobar The difference between them that the latter sets the module attribute, thus __getattr__ will be called only once.
Backwards compatibility and impact on performance =================================================
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it.. I will add a note to the PEP about this.
This problem is not new, since it was possible to replace a module with a module subclass with overridden __getattr__ and __dir__ before, but now this problem can occur more often.
I would create more standard helpers (for deprecation, for lazy importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
But in which module these helpers should live?
Good question. lazy_import() could be added in importlib (or importlib.util?). The helper that just adds deprecation on importing a name, could be added in importlib too. But I think that it would be better if the deprecated() helper will also create a wrapper that raises a deprecation warning on the use of deprecated function. It could be added in the warnings or functools modules. I would add also a more general lazy_initialized(). It is something like cached module property. Executes the specified code on first use, and cache the result as a module attribute. In all these cases the final __getattr__ method should be automatically constructed from different chunks. At the end it could call a user supplied __getattr__. Or maybe the module method __getattr__ should look first at special registry before calling the instance attribute __getattr__()? def ModuleType.__getattr__(self, name): if name in self.__properties__: call self.__properties__[name]() elif '__getattr__' in self.__dict__: call self.__dict__['__getattr__'](name) else: raise AttributeError I'm wondering if the __set_name__ mechanism can be extended to modules. What if call the __set_name__() method for all items in a module dict after finishing importing the module?
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi
Rationale =========
It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or replacing the ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example::
# lib.py
from warnings import warn
deprecated_names = ["old_function", ...]
def _deprecated_old_function(arg, other): ...
def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}")
# main.py
from lib import old_function # Works, but emits the warning
Deprecating functions is already possible, so I assume the reason for this would be performance? If so, are you sure this would help for performance? Deprecating module attributes / globals is indeed difficult to do at present. This PEP would allow deprecation warnings for accessing attributes, which is nice! However, as thread-unsafe as it is, many modules use module attributes to configure the state of the module. In that case, the user is more likely to *set* the attribute that to *get* it. Is this outside the scope of the PEP? [..]
There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization.
I'm not surprised that the comparison is in favor of this PEP ;-). [..]
Specification =============
The ``__getattr__`` function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this. [..]
Discussion ==========
Note that the use of module ``__getattr__`` requires care to keep the referred objects pickleable. For example, the ``__name__`` attribute of a function should correspond to the name with which it is accessible via ``__getattr__``::
def keep_pickleable(func): func.__name__ = func.__name__.replace('_deprecated_', '') func.__qualname__ = func.__qualname__.replace('_deprecated_', '') return func
@keep_pickleable def _deprecated_old_function(arg, other): ...
One should be also careful to avoid recursion as one would do with a class level ``__getattr__``.
Off-topic: In some sense, I'm happy to hear something about pickleability. But in some sense not. I think there are three kinds of people regarding pickleability: 1. Those who don't care about anything being pickleable 2. Those who care about some things being picklable 3. Those who care about all things being picklable Personally, I'd like to belong to group 3, but because group 3 cannot even attempt to coexist with groups 1 and 2, I actually belong to group 1 most of the time. ––Koos
References ==========
.. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files)
.. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files)
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ k7hoven%40gmail.com
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
I think it's reasonable for the PEP to include some examples, consequences
and best practices. I don't think it's reasonable for the PEP to also
define the API and implementation of helper functions that might be added
once the mechanisms are in place. Those are better developed as 3rd party
packages first.
On Wed, Nov 15, 2017 at 3:59 AM, Serhiy Storchaka
15.11.17 12:53, Ivan Levkivskyi пише:
On 15 November 2017 at 08:43, Serhiy Storchaka
mailto:storchaka@gmail.com> wrote: It is worth to mention that using name as a module global will bypass __getattr__. And this is intentional, otherwise calling __getattr__ for builtins will harm a performance.
Good point!
And please document idiomatic way of using a module global with triggering __getattr__. For example if you want to use a lazy loaded submodule.
sys.modules[__name__].foobar
or
from . import foobar
The difference between them that the latter sets the module attribute, thus __getattr__ will be called only once.
Backwards compatibility and impact on performance
=================================================
What is affect on pydoc, word completion, inspect, pkgutil, unittest?
This is rather gray area. I am not sure that we need to update them in any way, just the people who use __getattr__ should be aware that some tools might not yet expect it.. I will add a note to the PEP about this.
This problem is not new, since it was possible to replace a module with a module subclass with overridden __getattr__ and __dir__ before, but now this problem can occur more often.
I would create more standard helpers (for deprecation, for lazy
importing). This feature is helpful not by itself, but because it will be used for implementing new features. Using __getattr__ directly will need to write a boilerplate code. Maybe when implementing these helper you will discover that this PEP needs some additions.
But in which module these helpers should live?
Good question. lazy_import() could be added in importlib (or importlib.util?). The helper that just adds deprecation on importing a name, could be added in importlib too. But I think that it would be better if the deprecated() helper will also create a wrapper that raises a deprecation warning on the use of deprecated function. It could be added in the warnings or functools modules.
I would add also a more general lazy_initialized(). It is something like cached module property. Executes the specified code on first use, and cache the result as a module attribute.
In all these cases the final __getattr__ method should be automatically constructed from different chunks. At the end it could call a user supplied __getattr__. Or maybe the module method __getattr__ should look first at special registry before calling the instance attribute __getattr__()?
def ModuleType.__getattr__(self, name): if name in self.__properties__: call self.__properties__[name]() elif '__getattr__' in self.__dict__: call self.__dict__['__getattr__'](name) else: raise AttributeError
I'm wondering if the __set_name__ mechanism can be extended to modules. What if call the __set_name__() method for all items in a module dict after finishing importing the module?
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org
-- --Guido van Rossum (python.org/~guido)
On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:
Rationale =========
[...] It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method
[...]
Specification =============
The ``__getattr__`` function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this.
I don't see how. This is exactly the way normal __getattr__ works. -- ~Ethan~
On Wed, Nov 15, 2017 at 8:02 PM, Ethan Furman
On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:
Rationale
=========
[...] It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method
[...]
Specification
=============
The ``__getattr__`` function at the module level should accept one argument
which is the name of an attribute and return the computed value or raise
an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.
The Rationale (quoted in the beginning of this email) easily leaves a different impression of this.
I don't see how. This is exactly the way normal __getattr__ works.
Oh sorry, I think I put this email together too quickly. I was writing down a bunch of thoughts I had earlier but hadn't written down. I think I was mixing this up in my head with overriding __getitem__ for the module namespace dict and __class_getitem__ from PEP 560, which only gets called if the metaclass doesn't implement __getitem__ (IIRC). But I did have another thought related to this. I was wondering whether the lack of passing the module to the methods as `self` would harm future attempts to generalize these ideas. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
participants (5)
-
Ethan Furman
-
Guido van Rossum
-
Ivan Levkivskyi
-
Koos Zevenhoven
-
Serhiy Storchaka