In general, directly accessing dunders is a bit of a code smell. (I exclude writing dunder methods in your classes, of course.) There's usually a built-in or similar to do the job for you, e.g. instead of iterator.__next__() we should use next(iterator). One of the lesser-known ones is vars(obj), which should be used in place of obj.__dict__. Unfortunately, vars() is less useful than it might be, since not all objects have a __dict__. Some objects have __slots__ instead, or even both. That is considered an implementation detail of the object. Proposal: enhance vars() to return a proxy to the object namespace, regardless of whether said namespace is __dict__ itself, or a number of __slots__, or both. Here is a woefully incompete and untested prototype: class VarsProxy(object): def __init__(self, obj): if not (hasattr(obj, '__dict__') or hasattr(obj, '__slots__')): raise TypeError('object has no namespace') self._obj = obj def __getitem__(self, key): slots = getattr(type(self), '__slots__', None) # see inspect.getattr__static for a more correct implementation if slots is not None and key in slots: # return the content of the slot, without any inheritance. return getattr(self._obj, key) else: return self._obj.__dict__[key] def __setitem__(self, key, value): ... def __delitem__(self, key): ... One complication: it is possible for the slot and the __dict__ to both contain the key. In 3.5 that ambiguity is resolved in favour of the slot: py> class X: ... __slots__ = ['spam', '__dict__'] ... def __init__(self): ... self.spam = 'slot' ... self.__dict__['spam'] = 'dict' ... py> x = X() py> x.spam 'slot' Although __slots__ are uncommon, this would clearly distinguish vars(obj) from obj.__dict__ and strongly encourage the use of vars() over direct access to the dunder attribute. Thoughts? -- Steve
On Mon, Dec 12, 2016 at 6:45 PM, Steven D'Aprano <steve@pearwood.info> wrote:
How do you propose dealing with classes defined in C? Their objects don't have __slots__. One possibility is to use __dir__ or dir(), but those can return anything and in the past developers were encouraged to put only "useful" attributes in __dir__.
I'm +1. This bites me far too often.
in the past developers were encouraged to put only "useful" attributes in __dir__.
Good. If I'm getting vars() I really only want the useful ones. If I need interesting/secret ones then I'll getattr for them. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Alexander Belopolsky" <alexander.belopolsky@gmail.com> Sent: 12/12/2016 19:47 To: "Steven D'Aprano" <steve@pearwood.info> Cc: "python-ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Enhancing vars() On Mon, Dec 12, 2016 at 6:45 PM, Steven D'Aprano <steve@pearwood.info> wrote: Proposal: enhance vars() to return a proxy to the object namespace, regardless of whether said namespace is __dict__ itself, or a number of __slots__, or both. How do you propose dealing with classes defined in C? Their objects don't have __slots__. One possibility is to use __dir__ or dir(), but those can return anything and in the past developers were encouraged to put only "useful" attributes in __dir__.
On Mon, Dec 12, 2016 at 10:45:39PM -0500, Alexander Belopolsky wrote:
I don't see any clean way to do so. Maybe we should have a convention that such objects provide a __slots__ attribute listing public attributes, but I'm not too concerned. Let vars(weird_c_object) raise TypeError, just as it does now.
Indeed. -- Steve
On 13/12/2016 00:45, Steven D'Aprano wrote:
+1. Would it be possible in the future (Py4?) to change the name `vars` to a more meaningful name? Maybe `namespace`, or something more appropriate. -- Marco Buttu INAF-Osservatorio Astronomico di Cagliari Via della Scienza n. 5, 09047 Selargius (CA) Phone: 070 711 80 217 Email: mbuttu@oa-cagliari.inaf.it
On Tue, Dec 13, 2016 at 10:29:38AM +0100, Marco Buttu wrote:
+1. Would it be possible in the future (Py4?) to change the name `vars` to a more meaningful name? Maybe `namespace`, or something more appropriate.
I'm not keen on the name vars() either, but it does make a certain sense: short for "variables", where "variable" here refers to attributes of an instance rather than local or global variables. I'm not sure that namespace is a better name: namespace, it seems to me, it likely to be used as the name of the target: namespace = vars(obj) But if there is a lot of popular demand for a name change, then I suppose it could happen. Ask again around Python 3.9 :-) -- Steve
On 13 December 2016 at 20:02, Steven D'Aprano <steve@pearwood.info> wrote:
It also refers to local and global variables, as vars() is effectively an alias for locals() if you don't pass an argument, and locals() is effectively an alias for globals() at module level: >>> locals() is globals() True >>> vars() is globals() True >>> def f(): return vars() is locals() ... >>> f() True To be honest, rather than an enhanced vars(), I'd prefer to see a couple more alternate dict constructors: dict.fromattrs(obj, attributes) dict.fromitems(obj, keys) (With the lack of an underscore being due to the precedent set by dict.fromkeys()) Armed with those, the "give me all the attributes from __dir__" command would be: attrs = dict.from_attrs(obj, dir(obj)) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 13 December 2016 at 11:28, Nick Coghlan <ncoghlan@gmail.com> wrote:
Which of course can already be spelled as attrs = { attr: getattr(obj, attr) for attr in dir(obj) } There's obviously a speed-up from avoiding repeated getattr calls, but is speed the key here? The advantage of an "enhanced vars" is more likely to be ease of discoverability, and I'm not sure dict.fromattrs gives us that benefit. Also, the dict constructor gives a *copy* of the namespace, where the proposal was for the proxy returned by vars() to provide update capability (if I understand the proposal correctly). Paul
<snip>
I was actually hoping that someone might propose to deprecate this behavior. Why do we need a cascade of namespaces looked up by `vars` (and `locals`)?. It seems to me that this just complicates what these functions do and I don't see a clear reason for it. Having a function do only one thing might also help promote it's use in the cases where we want it to be used (e.g. instead of accessing `__dict__` directly). Of course, that might be too big a change to make any time soon... but a guy can hope can't he? :-) -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt@getpattern.com // P: 603.892.7736 We’re looking for beta testers. Go here <https://www.getpattern.com/meetpattern> to sign up!
On 13.12.16 01:45, Steven D'Aprano wrote:
http://bugs.python.org/issue13290 http://mail.python.org/pipermail/python-dev/2012-October/122011.html
On Wed, Dec 14, 2016 at 12:58:16AM +0200, Serhiy Storchaka wrote:
Thanks Serhiy! Glad to see I'm not the only one with this idea. I think: - the behaviour of locals() (and vars() when given no argument, where it returns locals()) is anomalous and should not be copied unless we really need to. - Other Python implementations don't always emulate the weird behaviour of locals(), for example I think IronPython locals() is writeable, and the local variables do change. steve@orac:~$ ipy IronPython 2.6 Beta 2 DEBUG (2.6.0.20) on .NET 2.0.50727.1433 Type "help", "copyright", "credits" or "license" for more information.
CPython will print 1 instead. So CPython locals() is an implementation detail and we shouldn't feel the need to copy it's weird behaviour. When given an object, vars(obj) should return a dict-like object which is a read/write proxy to the object's namespace. If the object has a __dict__ but no __slots__, then there's no need to change anything: it can keep the current behaviour and just return the dict itself: assert vars(obj) is obj.__dict__ But if the object has __slots__, with or without a __dict__, then vars should return a proxy which direct reads and writes to the correct slot or dict. It might be helpful to have a slotsproxy object which provides a dict-like interface to an object with __slots__ but no __dict__, and build support for both __slots__ and a __dict__ on top of that. If the objects has *neither* __slots__ nor __dict__, vars can probably raise a TypeError. -- Steve
That might be a bit tricky, for example, it's possible that a class has a `foo` slot *and* a `foo` instance attribute (by virtue of subclasses). What would you do in that case? Or what if there's a slot that doesn't have any value (i.e. raises AttributeError on access, but exists on the class nonetheless), but an instance attribute with the same name exists? And so on.
If the objects has *neither* __slots__ nor __dict__, vars can probably raise a TypeError.
Is that even possible in pure Python? The only object I know that can do this is `object`, but some other C objects might do that too. -Emanuel
On Wed, Dec 14, 2016 at 12:12:39AM +0000, Emanuel Barry wrote:
vars() shouldn't need to care about inheritance: it only cares about the object's own individual namespace, not attributes inherited from the class or superclasses. That's how vars() works now: py> class C: ... cheese = 1 ... py> obj = C() py> ns = vars(obj) py> 'cheese' in ns False The only difference here is that if the direct parent class has __slots__, the instance will use them instead of (or in addition to) a __dict__. We don't need to care about superclass __slots__, because they aren't inherited.
Only the *list of slot names* exists on the class. The slots themselves are part of the instance. Nevertheless, you are right: a slot can be defined, but not assigned to. That has to be treated as if the slot didn't exist: py> class D: ... __slots__ = ['spam'] ... py> d = D() py> hasattr(d, 'spam') False So I would expect that 'spam' in vars(d) should likewise return False, until such time that d.spam is assigned too. The same applies even if the object has a __dict__. The slot always takes precedence, even if the slot isn't filled in. py> class E: ... __slots__ = ['spam', '__dict__'] ... py> e = E() py> e.__dict__['spam'] = 1 py> hasattr(e, 'spam') False
I don't think pure Python classes can do this, at least not without some metaclass trickery, but certainly `object` itself lacks both __slots__ and instance __dict__, and C objects can do the same (so I'm told). -- Steve
On 14 December 2016 at 10:40, Steven D'Aprano <steve@pearwood.info> wrote:
If folks genuinely want an attrproxy that provides a dict-like view over an instance, that's essentially: from collections import MutableMapping class AttrProxy(MutableMapping): def __init__(self, obj): self._obj = obj def __len__(self): return len(dir(self._obj)) def __iter__(self): for attr in dir(self._obj): yield attr def __contains__(self, attr): return hasattr(self._obj, attr) def __getitem__(self, attr): return getattr(self._obj, attr) def __setitem__(self, attr, value): setattr(self._obj, attr, value) def __delitem__(self, attr): delattr(self._obj, attr) >>> class C: ... a = 1 ... b = 2 ... c = 3 ... >>> ns = AttrProxy(C) >>> list(ns.keys()) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'a', 'b', 'c'] >>> ns["d"] = 4 >>> C.d 4 >>> C.c 3 >>> del ns["c"] >>> C.c Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: type object 'C' has no attribute 'c' >>> ns["a"] = 5 >>> C.a 5 Replacing the calls to `dir(self._obj)` with a method that filters out dunder-methods and updating the other methods to reject them as keys is also relatively straightforward if people want that behaviour. Indirecting through dir(), hasattr(), getattr(), setattr() and delattr() this way means you don't have to worry about the vagaries of the descriptor protocol or inheritance or instance attributes vs class attributes or anything else like that, while inheriting from MutableMapping automatically gives you view-based keys(), values() and items() implementations. I wouldn't have any real objection to providing an API that behaves like this (in simple cases its functionally equivalent to manipulating __dict__ directly, while in more complex cases, the attrproxy approach is likely to just work, whereas __dict__ manipulation may fail). (Re-using vars() likely wouldn't be appropriate in that case though, due to the change in the way inheritance is handled) I *would* object to a new proxy type that duplicated descriptor logic that's already programmatically accessible in other builtins, or only selectively supported certain descriptors (like those created for __slots__) while ignoring others. Pseudo-lookups like inspect.getattr_static() exist to help out IDEs, debuggers and other code analysers, rather than as something we want people to be doing as part of their normal application execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Dec 12, 2016 at 6:45 PM, Steven D'Aprano <steve@pearwood.info> wrote:
How do you propose dealing with classes defined in C? Their objects don't have __slots__. One possibility is to use __dir__ or dir(), but those can return anything and in the past developers were encouraged to put only "useful" attributes in __dir__.
I'm +1. This bites me far too often.
in the past developers were encouraged to put only "useful" attributes in __dir__.
Good. If I'm getting vars() I really only want the useful ones. If I need interesting/secret ones then I'll getattr for them. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Alexander Belopolsky" <alexander.belopolsky@gmail.com> Sent: 12/12/2016 19:47 To: "Steven D'Aprano" <steve@pearwood.info> Cc: "python-ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Enhancing vars() On Mon, Dec 12, 2016 at 6:45 PM, Steven D'Aprano <steve@pearwood.info> wrote: Proposal: enhance vars() to return a proxy to the object namespace, regardless of whether said namespace is __dict__ itself, or a number of __slots__, or both. How do you propose dealing with classes defined in C? Their objects don't have __slots__. One possibility is to use __dir__ or dir(), but those can return anything and in the past developers were encouraged to put only "useful" attributes in __dir__.
On Mon, Dec 12, 2016 at 10:45:39PM -0500, Alexander Belopolsky wrote:
I don't see any clean way to do so. Maybe we should have a convention that such objects provide a __slots__ attribute listing public attributes, but I'm not too concerned. Let vars(weird_c_object) raise TypeError, just as it does now.
Indeed. -- Steve
On 13/12/2016 00:45, Steven D'Aprano wrote:
+1. Would it be possible in the future (Py4?) to change the name `vars` to a more meaningful name? Maybe `namespace`, or something more appropriate. -- Marco Buttu INAF-Osservatorio Astronomico di Cagliari Via della Scienza n. 5, 09047 Selargius (CA) Phone: 070 711 80 217 Email: mbuttu@oa-cagliari.inaf.it
On Tue, Dec 13, 2016 at 10:29:38AM +0100, Marco Buttu wrote:
+1. Would it be possible in the future (Py4?) to change the name `vars` to a more meaningful name? Maybe `namespace`, or something more appropriate.
I'm not keen on the name vars() either, but it does make a certain sense: short for "variables", where "variable" here refers to attributes of an instance rather than local or global variables. I'm not sure that namespace is a better name: namespace, it seems to me, it likely to be used as the name of the target: namespace = vars(obj) But if there is a lot of popular demand for a name change, then I suppose it could happen. Ask again around Python 3.9 :-) -- Steve
On 13 December 2016 at 20:02, Steven D'Aprano <steve@pearwood.info> wrote:
It also refers to local and global variables, as vars() is effectively an alias for locals() if you don't pass an argument, and locals() is effectively an alias for globals() at module level: >>> locals() is globals() True >>> vars() is globals() True >>> def f(): return vars() is locals() ... >>> f() True To be honest, rather than an enhanced vars(), I'd prefer to see a couple more alternate dict constructors: dict.fromattrs(obj, attributes) dict.fromitems(obj, keys) (With the lack of an underscore being due to the precedent set by dict.fromkeys()) Armed with those, the "give me all the attributes from __dir__" command would be: attrs = dict.from_attrs(obj, dir(obj)) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 13 December 2016 at 11:28, Nick Coghlan <ncoghlan@gmail.com> wrote:
Which of course can already be spelled as attrs = { attr: getattr(obj, attr) for attr in dir(obj) } There's obviously a speed-up from avoiding repeated getattr calls, but is speed the key here? The advantage of an "enhanced vars" is more likely to be ease of discoverability, and I'm not sure dict.fromattrs gives us that benefit. Also, the dict constructor gives a *copy* of the namespace, where the proposal was for the proxy returned by vars() to provide update capability (if I understand the proposal correctly). Paul
<snip>
I was actually hoping that someone might propose to deprecate this behavior. Why do we need a cascade of namespaces looked up by `vars` (and `locals`)?. It seems to me that this just complicates what these functions do and I don't see a clear reason for it. Having a function do only one thing might also help promote it's use in the cases where we want it to be used (e.g. instead of accessing `__dict__` directly). Of course, that might be too big a change to make any time soon... but a guy can hope can't he? :-) -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt@getpattern.com // P: 603.892.7736 We’re looking for beta testers. Go here <https://www.getpattern.com/meetpattern> to sign up!
On 13.12.16 01:45, Steven D'Aprano wrote:
http://bugs.python.org/issue13290 http://mail.python.org/pipermail/python-dev/2012-October/122011.html
On Wed, Dec 14, 2016 at 12:58:16AM +0200, Serhiy Storchaka wrote:
Thanks Serhiy! Glad to see I'm not the only one with this idea. I think: - the behaviour of locals() (and vars() when given no argument, where it returns locals()) is anomalous and should not be copied unless we really need to. - Other Python implementations don't always emulate the weird behaviour of locals(), for example I think IronPython locals() is writeable, and the local variables do change. steve@orac:~$ ipy IronPython 2.6 Beta 2 DEBUG (2.6.0.20) on .NET 2.0.50727.1433 Type "help", "copyright", "credits" or "license" for more information.
CPython will print 1 instead. So CPython locals() is an implementation detail and we shouldn't feel the need to copy it's weird behaviour. When given an object, vars(obj) should return a dict-like object which is a read/write proxy to the object's namespace. If the object has a __dict__ but no __slots__, then there's no need to change anything: it can keep the current behaviour and just return the dict itself: assert vars(obj) is obj.__dict__ But if the object has __slots__, with or without a __dict__, then vars should return a proxy which direct reads and writes to the correct slot or dict. It might be helpful to have a slotsproxy object which provides a dict-like interface to an object with __slots__ but no __dict__, and build support for both __slots__ and a __dict__ on top of that. If the objects has *neither* __slots__ nor __dict__, vars can probably raise a TypeError. -- Steve
That might be a bit tricky, for example, it's possible that a class has a `foo` slot *and* a `foo` instance attribute (by virtue of subclasses). What would you do in that case? Or what if there's a slot that doesn't have any value (i.e. raises AttributeError on access, but exists on the class nonetheless), but an instance attribute with the same name exists? And so on.
If the objects has *neither* __slots__ nor __dict__, vars can probably raise a TypeError.
Is that even possible in pure Python? The only object I know that can do this is `object`, but some other C objects might do that too. -Emanuel
On Wed, Dec 14, 2016 at 12:12:39AM +0000, Emanuel Barry wrote:
vars() shouldn't need to care about inheritance: it only cares about the object's own individual namespace, not attributes inherited from the class or superclasses. That's how vars() works now: py> class C: ... cheese = 1 ... py> obj = C() py> ns = vars(obj) py> 'cheese' in ns False The only difference here is that if the direct parent class has __slots__, the instance will use them instead of (or in addition to) a __dict__. We don't need to care about superclass __slots__, because they aren't inherited.
Only the *list of slot names* exists on the class. The slots themselves are part of the instance. Nevertheless, you are right: a slot can be defined, but not assigned to. That has to be treated as if the slot didn't exist: py> class D: ... __slots__ = ['spam'] ... py> d = D() py> hasattr(d, 'spam') False So I would expect that 'spam' in vars(d) should likewise return False, until such time that d.spam is assigned too. The same applies even if the object has a __dict__. The slot always takes precedence, even if the slot isn't filled in. py> class E: ... __slots__ = ['spam', '__dict__'] ... py> e = E() py> e.__dict__['spam'] = 1 py> hasattr(e, 'spam') False
I don't think pure Python classes can do this, at least not without some metaclass trickery, but certainly `object` itself lacks both __slots__ and instance __dict__, and C objects can do the same (so I'm told). -- Steve
On 14 December 2016 at 10:40, Steven D'Aprano <steve@pearwood.info> wrote:
If folks genuinely want an attrproxy that provides a dict-like view over an instance, that's essentially: from collections import MutableMapping class AttrProxy(MutableMapping): def __init__(self, obj): self._obj = obj def __len__(self): return len(dir(self._obj)) def __iter__(self): for attr in dir(self._obj): yield attr def __contains__(self, attr): return hasattr(self._obj, attr) def __getitem__(self, attr): return getattr(self._obj, attr) def __setitem__(self, attr, value): setattr(self._obj, attr, value) def __delitem__(self, attr): delattr(self._obj, attr) >>> class C: ... a = 1 ... b = 2 ... c = 3 ... >>> ns = AttrProxy(C) >>> list(ns.keys()) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'a', 'b', 'c'] >>> ns["d"] = 4 >>> C.d 4 >>> C.c 3 >>> del ns["c"] >>> C.c Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: type object 'C' has no attribute 'c' >>> ns["a"] = 5 >>> C.a 5 Replacing the calls to `dir(self._obj)` with a method that filters out dunder-methods and updating the other methods to reject them as keys is also relatively straightforward if people want that behaviour. Indirecting through dir(), hasattr(), getattr(), setattr() and delattr() this way means you don't have to worry about the vagaries of the descriptor protocol or inheritance or instance attributes vs class attributes or anything else like that, while inheriting from MutableMapping automatically gives you view-based keys(), values() and items() implementations. I wouldn't have any real objection to providing an API that behaves like this (in simple cases its functionally equivalent to manipulating __dict__ directly, while in more complex cases, the attrproxy approach is likely to just work, whereas __dict__ manipulation may fail). (Re-using vars() likely wouldn't be appropriate in that case though, due to the change in the way inheritance is handled) I *would* object to a new proxy type that duplicated descriptor logic that's already programmatically accessible in other builtins, or only selectively supported certain descriptors (like those created for __slots__) while ignoring others. Pseudo-lookups like inspect.getattr_static() exist to help out IDEs, debuggers and other code analysers, rather than as something we want people to be doing as part of their normal application execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (12)
-
Alexander Belopolsky -
Emanuel Barry -
Ethan Furman -
Marco Buttu -
Matt Gilson -
Nick Coghlan -
Paul Moore -
Serhiy Storchaka -
Stephen J. Turnbull -
Steve Dower -
Steven D'Aprano -
Terry Reedy