So we've seen a real use case for __class__ assignment: deprecating things on access. That use case could also be solved if modules natively supported defining __getattr__ (with the same "only used if attribute not found otherwise" semantics as it has on classes), but it couldn't be solved using @property (or at least it would be quite hacky). Is there a real use case for @property? Otherwise, if we're going to mess with module's getattro, it makes more sense to add __getattr__, which would have made Nathaniel's use case somewhat simpler. (Except for the __dir__ thing -- what else might we need?) On Tue, Sep 5, 2017 at 3:52 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Tue, Sep 5, 2017 at 3:03 PM, Larry Hastings <larry@hastings.org> wrote:

...

I've written a PEP proposing a language change:

https://www.python.org/dev/peps/pep-0549/

The TL;DR summary: add support for property objects to modules. I've already posted a prototype.

Interesting idea! It's definitely less arcane than the __class__ assignment support that was added in 3.5. I guess the question is whether to add another language feature here, or to provide better documentation/helpers for the existing feature.

If anyone's curious what the __class__ trick looks like in practice, here's some simple deprecation machinery: https://github.com/njsmith/trio/blob/ee8d909e34a2b28d55b5c6137707e8 861eee3234/trio/_deprecate.py#L102-L138

And here's what it looks like in use: https://github.com/njsmith/trio/blob/ee8d909e34a2b28d55b5c6137707e8 861eee3234/trio/__init__.py#L91-L115

Advantages of PEP 549: - easier to explain and use

Advantages of the __class__ trick: - faster (no need to add an extra step to the normal attribute lookup fast path); only those who need the feature pay for it

- exposes the full power of Python's class model. Notice that the above code overrides __getattr__ but not __dir__, so the attributes are accessible via direct lookup but not listed in dir(mod). This is on purpose, for two reasons: (a) tab completion shouldn't be suggesting deprecated attributes, (b) when I did expose them in __dir__, I had trouble with test runners that iterated through dir(mod) looking for tests, and ended up spewing tons of spurious deprecation warnings. (This is especially bad when you have a policy of running your tests with DeprecationWarnings converted to errors.) I don't think there's any way to do this with PEP 549.

- already supported in CPython 3.5+ and PyPy3, and with a bit of care can be faked on all older CPython releases (including, crucially, CPython 2). PEP 549 OTOH AFAICT will only be usable for packages that have 3.7 as their minimum supported version.

I don't imagine that I would actually use PEP 549 any time in the foreseeable future, due to the inability to override __dir__ and the minimum version requirement. If you only need to support CPython 3.5+ and PyPy3 5.9+, then you can effectively get PEP 549's functionality at the cost of 3 lines of code and a block indent:

import sys, types class _MyModuleType(types.ModuleType): @property def ...

@property def ... sys.modules[__name__].__class__ = _MyModuleType

It's definitely true though that they're not the most obvious lines of code :-)

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On Wed, Sep 6, 2017 at 9:15 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 6 September 2017 at 17:26, Guido van Rossum <guido@python.org> wrote:

...

Is there a real use case for @property? Otherwise, if we're going to mess with module's getattro, it makes more sense to add __getattr__, which would have made Nathaniel's use case somewhat simpler. (Except for the __dir__ thing -- what else might we need?)

One additional (IMO quite strong) argument in favor of module level __getattr__ is that this is already used by PEP 484 for stub files and is supported by mypy, see https://github.com/python/mypy/pull/3647

So we're looking for a competing PEP here. Shouldn't be long, just summarize the discussion about use cases and generality here. -- --Guido van Rossum (python.org/~guido)

On 09/07/2017 03:49 PM, Larry Hastings wrote:

Excluding Lib/test, there are 375 uses of "@property" in the stdlib in trunk, 60 uses of __getattr__, and 34 of __getattribute__.

I spent a minute looking at the output and realized there were a bunch of hits inside pydoc_data/topics.py, aka documentation. And then I realized I was only interested in definitions, not docs or calls or other hackery. Removing pydoc_data/topics.py and changing the search terms results in: "@property" 375 hits "def __getattr__" 28 hits "def __getattribute__(" 2 hits Unless the average use of __getattr__ serve in excess of 13 members each, property is the most popular technique of the three. //arry/

2017-09-11 19:00 GMT+02:00 Chris Barker <chris.barker@noaa.gov>:

I wish there were a property feature available almost very time I encounter a "get*" method in the stdlib (or any where):

There are a heck of a lot in the os module:

In [4]: [s for s in dir(os) if s.startswith('get')] Out[4]:

['get_blocking',

This one is not a good example: it takes a parameter. You cannot convert it to a property.

'getcwd',

And just yesterday I was getting annoyed by some in sysconfig:

In [6]: [s for s in dir(sysconfig) if s.startswith('get')] Out[6]:

['get_config_h_filename', 'get_config_var', 'get_config_vars', 'get_makefile_filename', 'get_path', 'get_path_names', 'get_paths', 'get_platform', 'get_python_version', 'get_scheme_names']

When designing an API, when I have to choose between property and function/method, I prefer function/method over a property when the code is slow, especially at the first call. I prefer to "warn" users than a call (like the first one which fills a cache) can be slow. Well, it's not a strong rule, sometimes I use a property even if the first call has to fill a cache :-) Here the sysconfig has to build an internal cache at the first call ... if I recall correctly. Victor

On Wed, Sep 6, 2017 at 10:26 AM, Guido van Rossum <guido@python.org> wrote:

So we've seen a real use case for __class__ assignment: deprecating things on access. That use case could also be solved if modules natively supported defining __getattr__ (with the same "only used if attribute not found otherwise" semantics as it has on classes), but it couldn't be solved using @property (or at least it would be quite hacky).

Is there a real use case for @property? Otherwise, if we're going to mess with module's getattro, it makes more sense to add __getattr__, which would have made Nathaniel's use case somewhat simpler. (Except for the __dir__ thing -- what else might we need?) -- --Guido van Rossum (python.org/~guido)

I think a more natural way for the __dir__ problem would be to update module_dir() in moduleobject.c to check if __all__ is defined and then just return that list if it is defined. I think that would be a friendlier default for __dir__ anyway. Cody

On 09/06/2017 08:26 AM, Guido van Rossum wrote:

So we've seen a real use case for __class__ assignment: deprecating things on access. That use case could also be solved if modules natively supported defining __getattr__ (with the same "only used if attribute not found otherwise" semantics as it has on classes), but it couldn't be solved using @property (or at least it would be quite hacky).

Is there a real use case for @property? Otherwise, if we're going to mess with module's getattro, it makes more sense to add __getattr__, which would have made Nathaniel's use case somewhat simpler. (Except for the __dir__ thing -- what else might we need?)

Doesn't assigning a module's __class__ make it so we can already write properties, descriptors, __getattr__, etc. ? Are these other things so common that we need /more/ syntax for them? -- ~Ethan~

On Sun, Sep 10, 2017 at 8:52 PM, Larry Hastings <larry@hastings.org> wrote:

On 09/06/2017 02:13 PM, Ronald Oussoren wrote:

To be honest this sounds like a fairly crude hack. Updating the __class__ of a module object feels dirty, but at least you get normal behavior w.r.t. properties.

Okay. Obviously I disagree, I think it's reasonable. But I'll assume you're -1.

I'm still poindering this. I am far from decided about whether it's better to just add @property support, or whether we should add __getattr__ instead (it can do everything @property can do, plus other things, but the @property support would be a little clumsier), or everything that a class can do with attributes (e.g. __getattribute__, __dir__, __setattr__), or leave things alone (__class__ assignment can solve everything). I worry that in the end @property isn't general enough and the major use cases end up still having to use __class__ assignment, and then we'd have a fairly useless feature that we cant withdraw, ever.

Why is there no mechanism to add new descriptors that can work in this context?

I've updated the prototype to add one. I added it as "collections.abc.InstanceDescriptor"; that's a base class you can inherit from, and then your descriptor will work in a module. Bikeshedding the name is fine.

I don't understand the question, or the answer. (And finding the prototype is taking longer than writing this email.)

BTW. The interaction with import is interesting… Module properties only work as naive users expect when accessing them as attributes of the module object, in particular importing the name using “from module import prop” would only call the property getter once and that may not be the intended behavior.

I spent a fair amount of time thinking about this. The short answer is: we *could* fix it. We *could* make it so that "from x import y", when x.y is an instance descriptor, ensures that y honors the descriptor protocol when referenced. We'd have to do it for three contexts:

- global scope (aka module scope) - class scope - function scope

The first two are pretty similar; create a proxy object that retains the module instance so it remains "bound" to that. The third one is trickier; it'd mean a new bytecode (LOAD_FAST_DESCRIPTOR), but it wouldn't slow down people who didn't use it.

Anyway, long story short, I think this would be worse than simply having "from x import y" only calling the getter once. As the Zen says: special cases aren't special enough to break the rules.

I agree -- let's please not fix this, it's way too complicated for something that started out as a nice localized tweak. When we write "from x import y" we already voluntarily give up seeing any later assignments to x.y. If this prevents a certain use case for lazy import, then so be it -- I don't think lazy import can ever be made so transparent that everything will work lazily without understanding the difference between lazy and eager import. (People already have to understand many subtleties of import, e.g. the difference between import and loading, and how circular imports work.) -- --Guido van Rossum (python.org/~guido)

On Mon, Sep 11, 2017 at 6:04 PM, Larry Hastings <larry@hastings.org> wrote:

On 09/11/2017 08:44 AM, Guido van Rossum wrote:

I worry that in the end @property isn't general enough and the major use cases end up still having to use __class__ assignment, and then we'd have a fairly useless feature that we cant withdraw, ever.

What can I say--I don't have that worry ;-)

As previously mentioned in this thread, I counted up uses of property, __getattr__, and __getattribute__ in 3.7/Lib. I grepped for the following strings, ignored pydoc_data/topics.py, and got these totals:

"@property" 375 hits "def __getattr__" 28 hits "def __getattribute__(" 2 hits

@property seems pretty popular.

I saw that the first time but it doesn't address my worry. (I don't feel like explaining the worry again though, I feel I've done all I can.)

Why is there no mechanism to add new descriptors that can work in this

...

context?

I've updated the prototype to add one. I added it as "collections.abc.InstanceDescriptor"; that's a base class you can inherit from, and then your descriptor will work in a module. Bikeshedding the name is fine.

I don't understand the question, or the answer. (And finding the prototype is taking longer than writing this email.)

Ronald was basically asking: what about user classes? The first revision of the prototype didn't provide a way to write your own instance descriptors. The only thing that could be a instance descriptor was property. So, I updated the prototype and added collections.abc.InstanceDescriptor, a base class user classes can inherit from that lets them be instance descriptors.

I still don't follow. How does one use InstanceDescriptor? Is this still about modules, or is it a generalization of properties for non-module instances? (If it is specific to modules, why doesn't it have "module" in its name? Or if it's not, why is it in this discussion?)

The prototype is linked to from the PEP; for your convenience here's a link:

https://github.com/larryhastings/cpython/tree/module-properties

I found that link in the PEP, but it's just a branch of a fork of cpython. It would be easier to review the prototype as a PR to upstream cpython.

-- --Guido van Rossum (python.org/~guido)

On 09/12/2017 12:38 AM, Larry Hastings wrote:

On 09/11/2017 07:22 PM, Guido van Rossum wrote:

...

The prototype is linked to from the PEP; for your convenience here's a link:

https://github.com/larryhastings/cpython/tree/module-properties <https://github.com/larryhastings/cpython/tree/module-properties>

I found that link in the PEP, but it's just a branch of a fork of cpython. It would be easier to review the prototype as a PR to upstream cpython.

Okay, I'll make a PR tomorrow and post a reply here (and update the PEP).

PR is here: https://github.com/python/cpython/pull/3534 Cheers, //arry/

Why not adding both? Properties do have their uses as does __getattr__.

In that case I would just add __getattr__ to module.c, and add a recipe or perhaps a utility module that implements a __getattr__ you can put into your module if you want @property support. That way you can have both but you only need a little bit of code in module.c to check for __getattr__ and call it when you'd otherwise raise AttributeError. -- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)

On Wed, Sep 13, 2017 at 2:00 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Wed, Sep 13, 2017 at 11:49 AM, Guido van Rossum <guido@python.org> wrote:

...

...

Why not adding both? Properties do have their uses as does __getattr__.

In that case I would just add __getattr__ to module.c, and add a recipe or perhaps a utility module that implements a __getattr__ you can put into your module if you want @property support. That way you can have both but you only need a little bit of code in module.c to check for __getattr__ and call it when you'd otherwise raise AttributeError.

Unfortunately I don't think this works. If there's a @property object present in the module's instance dict, then __getattribute__ will return it directly instead of calling __getattr__.

Hm, that's a good point. One would have to introduce some kind of convention where you can write properties with a leading _: @property def _foo(): return 42 and then a utility __getattr__ like this: def __getattr__(name): g = globals() name = '_' + name if name in g: return g[name]() raise AttributeError(...)

(I guess for full property emulation you'd also need to override __setattr__ and __dir__, but I don't know how important that is.)

At that point maybe __class__ assignment is better.

We could consider letting modules overload __getattribute__ instead of __getattr__, but I don't think this is viable either -- a key feature of __getattr__ is that it doesn't add overhead to normal lookups. If you implement deprecation warnings by overloading __getattribute__, then it makes all your users slower, even the ones who never touch the deprecated attributes. __getattr__ is much better than __getattribute__ for this purpose.

Agreed. Alternatively we can have a recipe that implements @property support

using __class__ assignment and overriding __getattribute__/__setattr__/__dir__, so instead of 'from module_helper.property_emulation import __getattr__' it'd be 'from module_helper import enable_property_emulation; enable_property_emulation(__name__)'. Still has the slowdown problem but it would work.

The emulation could do something less drastic than __class__ assignment -- it could look for globals that are properties, move them into some other dict (e.g. __properties__), and install a __getattr__ that looks things up in that dict and calls them. def __getattr__(name): if name in __properties__: return __properties__[name]() raise AttributeError(...) Still, proposals for sys.py notwithstanding, I'm worried that all of this is a solution looking for a problem. Yes, it's a cute hack. But is it art? -- --Guido van Rossum (python.org/~guido)

On Wed, Sep 13, 2017 at 3:01 PM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

@Guido

...

One would have to introduce some kind of convention where you can write properties with a leading _

One doesn't even need the @property decorator in this case. For example:

def __getattr__(name): g = globals() name = '_' + name if name in g: return g[name]() raise AttributeError(...)

def _attr(): "do something" return actual_object

One can even write a decorator that would change the name automatically (but this will require a call at the end of module to unbind original names). The space for play is infinite. The question is what exactly is needed. I would say that already basic customisation (like __getattr__) will be enough. IIUC, this will be actually a bit faster than setting __class__ since only "customized" attributes will be affected.

That last sentence is a key observation. Do we even know whether there are (non-toy) things that you can do *in principle* with __class__ assignment but which are too slow *in practice* to bother? And if yes, is __getattr__ fast enough? @property? IMO we're still looking for applications. -- --Guido van Rossum (python.org/~guido)

(sorry for obvious mistakes in the example in previous e-mail) On 14 September 2017 at 21:08, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 14 September 2017 at 01:13, Guido van Rossum <guido@python.org> wrote:

...

That last sentence is a key observation. Do we even know whether there are (non-toy) things that you can do *in principle* with __class__ assignment but which are too slow *in practice* to bother? And if yes, is __getattr__ fast enough? @property?

I myself have never implemented deprecation warnings management nor lazy loading, so it is hard to say if __class__ assignment is fast enough. For me it is more combination of three factors:

* modest performance improvement * some people might find __getattr__ clearer than __class__ assignment * this would be consistent with how stubs work

...

IMO we're still looking for applications.

How about this

def allow_forward_references(*allowed): caller_globals = sys._getframe().__globals__ def typing_getattr(name): if name in allowed: return name raise AttributeError(...) caller_globals.__getattr__ = typing_getattr

from typing_extensions import allow_forward_references allow_forward_references('Vertex', 'Edge')

T = TypeVar('T', bound=Edge)

class Vertex(List[Edge]): def copy(self: T) -> T: ...

class Edge: ends: Tuple[Vertex, Vertex] ...

Look mum, no quotes! :-)

-- Ivan

On 14 September 2017 at 22:07, Ethan Furman <ethan@stoneleaf.us> wrote:

For comparison's sake, what would the above look like using __class__ assignment? And what is the performance difference?

Actually I tried but I can't implement this without module __getattr__ so that one can just write: from typing_extensions import allow_forward_references allow_forward_references('Vertex', 'Edge') Maybe I am missing something, but either it is impossible in principle or highly non-trivial. -- Ivan

On 14 September 2017 at 22:07, Ethan Furman <ethan@stoneleaf.us> wrote:

For comparison's sake, what would the above look like using __class__ assignment? And what is the performance difference?

FWIW I found a different solution: # file mod.py from typing_extensions import allow_forward_references allow_forward_references() from mod import Vertex, Edge # the import is from this same module. It works both with __class__ assignment and with __getattr__ -- Ivan

On Thu, 14 Sep 2017 at 12:09 Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 14 September 2017 at 01:13, Guido van Rossum <guido@python.org> wrote:

...

That last sentence is a key observation. Do we even know whether there are (non-toy) things that you can do *in principle* with __class__ assignment but which are too slow *in practice* to bother? And if yes, is __getattr__ fast enough? @property?

I myself have never implemented deprecation warnings management nor lazy loading, so it is hard to say if __class__ assignment is fast enough.

For lazy loading it is, but that's because after the lazy load, module.__class__ is set back to its original value, so any performance penalty is paid only once upon triggering __getattribute__ on the lazy module. IOW no one has complained and I know plenty of places using the lazy loading mechanism in importlib at scale. :) -Brett

For me it is more combination of three factors:

* modest performance improvement * some people might find __getattr__ clearer than __class__ assignment * this would be consistent with how stubs work

...

IMO we're still looking for applications.

How about this

def allow_forward_references(*allowed): caller_globals = sys._getframe().__globals__ def typing_getattr(name): if name in allowed: return name raise AttributeError(...) caller_globals.__getattr__ = typing_getattr

from typing_extensions import allow_forward_references allow_forward_references('Vertex', 'Edge')

T = TypeVar('T', bound=Edge)

class Vertex(List[Edge]): def copy(self: T) -> T: ...

class Edge: ends: Tuple[Vertex, Vertex] ...

Look mum, no quotes! :-)

-- Ivan

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org