Change in evaluation order in new object model
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made. I was comfortable with the way it worked. Is there an advantage to the change?. In the classic python model the interpreter looked in the instance dictionary and if the name wasn't there it looked in the class dictionary. The following illustrates this evaluation order.
class C: def __init__(self): self.a = 4
c = C() c.a 4 C.a = 6 c.a 4 c.a = 8 c.a 8
With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
class B(object): __slots__ = ['a','b','c']
b = B() b.a = 4 b.a 4 B.a = 6 b.a 6 b.a = 8 Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
Michael McLay wrote:
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made. I was comfortable with the way it worked. Is there an advantage to the change?.
In the classic python model the interpreter looked in the instance dictionary and if the name wasn't there it looked in the class dictionary. The following illustrates this evaluation order. ...
With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
class B(object): __slots__ = ['a','b','c']
b = B() b.a = 4 b.a 4 B.a = 6 b.a 6 b.a = 8 Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
Could someone please first explain what these slots are used for in the first place :-? There must be some difference to standard class attributes... which is probably also the reason for the above behaviour (even though it does look like a bug to me). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
Could someone please first explain what these slots are used for in the first place :-? There must be some difference to standard class attributes... which is probably also the reason for the above behaviour (even though it does look like a bug to me).
My understanding is that the slots define the set of attributes-like things that instances of that class can have. It 'robs' the instances of such classes from having a __dict__ (at least conceptually), and provides much lighter weight attributes. Instances only need to store the _values_ in a struct/array, while the set of slot names and their order (in memory) is kept at the class level. This sort of thing is helpful from a memory point of view when you're dealing with e.g. large numbers of similar objects which all share the same attribute set (coordinates, Tk labels, whatnot). It doesn't explain (to me at least) why the lookup order is different. --david
On Friday 02 November 2001 11:22 am, M.-A. Lemburg wrote:
Michael McLay wrote:
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made. I was comfortable with the way it worked. Is there an advantage to the change?.
In the classic python model the interpreter looked in the instance dictionary and if the name wasn't there it looked in the class dictionary. The following illustrates this evaluation order. ...
With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
class B(object):
__slots__ = ['a','b','c']
b = B() b.a = 4 b.a
4
B.a = 6 b.a
6
b.a = 8
Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
Could someone please first explain what these slots are used for in the first place :-? There must be some difference to standard class attributes... which is probably also the reason for the above behaviour (even though it does look like a bug to me).
Classes defined with slots do not use a __dict__ to hold the content of the class. Instead an instance of a type includes a table with one entry per slot name. The "member descriptor" for each slot defines the offset into the table for that slot. The size of each instances is reduced by the elimination of the __dict__. The lookup of a member is also faster because it uses a lookup of an offset instead of a dictionary lookup. The following example shows some of the characteristics of slots.
class B(object): __slots__ = ['a','b','c']
b.__dict__ Traceback (most recent call last): File "<pyshell#85>", line 1, in ? b.__dict__ AttributeError: 'B' object has no attribute '__dict__' dir(b) ['__class__', '__delattr__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__repr__', '__setattr__', '__slots__', '__str__', 'a', 'b', 'c']
The slot returns a None value if it is referenced before it is initialized
b = B() b.a b.a = 3 b.a 3
If a reference is made to an undefined slot an AttributeError is generated.
b.d = 5 Traceback (most recent call last): File "<pyshell#100>", line 1, in ? b.d = 5 AttributeError: 'B' object has no attribute 'd'
The declaration of slots using __slots__ is somewhat awkward and Guido's comment about the syntax indicates this may change. One immediate problem with the syntax is that it doesn't support associating doc strings with slot names. Andrew described an example of how the descriptor[1] capability will allow Python to be extended in interesting ways. It turns out to be relatively easy to extend the descriptor capability. I submitted a patch yesterday that adds support for doc strings and optional type checking to the members defined by slots. The syntax to add the new capabilities is similar to the __slots__, but uses dictionaries to assign the values. class B(object): __slots__ = ['a','b','c'] __slot_docs__ = {'a':"doc string for a", 'b' : "doc string for b"} __slot_types__ = {'a':(int,str), 'c':int, } [1] http://www.amk.ca/python/2.2/index.html#SECTION000320000000000000000
Andrew described an example of how the descriptor[1] capability will allow Python to be extended in interesting ways. It turns out to be relatively easy to extend the descriptor capability. I submitted a patch yesterday that adds support for doc strings and optional type checking to the members defined by slots. The syntax to add the new capabilities is similar to the __slots__, but uses dictionaries to assign the values.
class B(object): __slots__ = ['a','b','c'] __slot_docs__ = {'a':"doc string for a", 'b' : "doc string for b"} __slot_types__ = {'a':(int,str), 'c':int, }
This is very interesting - I will definitely look at your patch. This can also be achieved by implementing custom attribute descriptors, without changes to the core. You just have to equip the class with them manually. Currently I'm working on something similar (see http://lpfw.sf.net/, use the link titled 'Accessing and manipulating C data types'). Thomas
[Michael McLay]
... The lookup of a member is also faster because it uses a lookup of an offset instead of a dictionary lookup.
There's still a dict lookup: when you do obj.a where a is a __slot__ attribute of obj.__class__, obj.__class__.__dict__['a'] is looked up in order to get the descriptor for attribute 'a'. The fixed set of __slot__ attributes leaves a door open for future optimizations, though (e.g., if Python could *know* obj.__class__ at compile-time, and know that runtime code won't overwrite the 'a' descriptor in obj.__class__.__dict__, it could map obj.a directly to its storage offset (from the base of obj) at compile-time).
Tim Peters wrote:
[Michael McLay]
... The lookup of a member is also faster because it uses a lookup of an offset instead of a dictionary lookup.
There's still a dict lookup: when you do obj.a where a is a __slot__ attribute of obj.__class__, obj.__class__.__dict__['a'] is looked up in order to get the descriptor for attribute 'a'. The fixed set of __slot__ attributes leaves a door open for future optimizations, though (e.g., if Python could *know* obj.__class__ at compile-time, and know that runtime code won't overwrite the 'a' descriptor in obj.__class__.__dict__, it could map obj.a directly to its storage offset (from the base of obj) at compile-time).
That'd be cool :-) Say, would it also be possible to use __slots__ for methods ? Or even make all methods defined in the class automagically become __slots__ members ? (As I understood your explanations, __slots__ would not interfere with class attributes, only instance attributes, so this should be possible, right ?) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/
From: "M.-A. Lemburg" <mal@lemburg.com>
Michael McLay wrote:
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made. I was comfortable with the way it worked. Is there an advantage to the change?.
In the classic python model the interpreter looked in the instance dictionary and if the name wasn't there it looked in the class dictionary. The following illustrates this evaluation order. ...
With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
class B(object): __slots__ = ['a','b','c']
b = B() b.a = 4 b.a 4 B.a = 6 b.a 6 b.a = 8 Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
Could someone please first explain what these slots are used for in the first place :-? There must be some difference to standard class attributes... which is probably also the reason for the above behaviour (even though it does look like a bug to me).
The lookup order (instance dict, then class dict) hasn't changed, just there is npo instance dict any longer. B.a is an attribute descriptor which retrieves the 'a' attribute from the instance. B.a = 6 doesn't change anything in the instance, it just masks the slot so that it isn't accessible any longer. Is this a bug, a feature or an implementation detail? No idea... Thomas
[Michael McLay]
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made.
Read PEP 252, paying special attention to the section containing: When a dynamic attribute (one defined in a regular object's __dict__) has the same name as a static attribute (one defined by a meta-object in the inheritance graph rooted at the regular object's __class__), the static attribute has precedence if it is a descriptor that defines a __set__ method (see below); otherwise (if there is no __set__ method) the dynamic attribute has precedence. In other words, for data attributes (those with a __set__ method), the static definition overrides the dynamic definition, but for other attributes, dynamic overrides static. Rationale: we can't have a simple rule like "static overrides dynamic" or "dynamic overrides static", because ...
... With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
I should hope so! The *point* of __slots__ (which is what you're really talking about, not the general concept of "slots") is that the class, not the object, is responsible for doing the attribute name->storage_address mapping, and in intended use an object of a class with __slots__ doesn't even have a __dict__ (each __slot__ attribute is allocated at a fixed offset from the start of the object, saving tons of storage).
class C(object): ... __slots__ = ['a']
Now objects of type C don't have a dict: storage for one attribute 'a' is allocated directly in C objects.
c = C() c.__dict__ Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'C' object has no attribute '__dict__'
You can set and get 'a':
c.a = 12 c.a 12
C.a is a special beast:
C.a <member 'a' of 'C' objects>
What makes it special isn't that it came from __slots__, though, but that it has a __set__ method (reread the quoted text above until your eyes bleed <wink>: this is a deadly simple protocol, so simple that it can be hard to understand at first (shades of the metaclass hook and continuations, there)):
dir(C.a) ['__class__', '__delattr__', '__doc__', '__get__', '__getattribute__', ^^^^^^^ C.a.__get__ is called when c.a is referenced.
'__hash__', '__init__', '__name__', '__new__', '__objclass__', '__reduce__', '__repr__', '__set__', '__setattr__', '__str__'] ^^^^^^^ C.a.__set__ is called when c.a is bound or del'ed. Objects of C type can't grow new attributes:
c.b =12 Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'C' object has no attribute 'b'
class B(object): __slots__ = ['a','b','c']
b = B() b.a = 4 b.a 4 B.a = 6
Here you overwrote the descriptor that allows b.a to mean something sensible (you nuked the <member 'a' of 'B' objects> thingie that maps 'a' to its storage address). Now B.a is an ordinary class attribute, and remember that b doesn't have a __dict__ (which you asked for, by using __slots__; you're not required to use __slots__).
b.a 6 b.a = 8 Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
I agree it's an odd msg, but I'm not sure it can do better easily: by overwriting B.a (which was nuts -- you're exploring pathologies here, not intended usage), you've left b as an object with an 'a' attribute inherited from its class, but also as an object that can't grow new attributes of its own. Python looks at "hmm, I *can't* set 'a', but I do *have* an 'a'", and comes up with "read-only". Try your example again without using __slots__ (you do *not* want __slots__ if you intend an object's namespace to be dynamic -- __slots__ announces that you guarantee the set of object attributes is fixed at class creation time):
class B(object): pass ... b = B() b.a = 4 B.a = 6 b.a 4 b.a = 8 b.a 8 B.a 6
IOW, don't use new features if you don't want new semantics, and things look much the same. If you want __slots__, though, there was no way to get its effect prior to 2.2 short of writing an ExtensionClass in C.
On Friday 02 November 2001 01:32 pm, Tim Peters wrote:
[Michael McLay]
I was suprised by a change to the order of evaluation of members in the new object type. I haven't found an explanation for why the change was made.
Read PEP 252, paying special attention to the section containing:
When a dynamic attribute (one defined in a regular object's __dict__) has the same name as a static attribute (one defined by a meta-object in the inheritance graph rooted at the regular object's __class__), the static attribute has precedence if it is a descriptor that defines a __set__ method (see below); otherwise (if there is no __set__ method) the dynamic attribute has precedence. In other words, for data attributes (those with a __set__ method), the static definition overrides the dynamic definition, but for other attributes, dynamic overrides static.
Rationale: we can't have a simple rule like "static overrides dynamic" or "dynamic overrides static", because ...
... With the new slots mechanism the order has been reversed. The class level dictionary is searched and then the slots are evaluated.
I should hope so! The *point* of __slots__ (which is what you're really talking about, not the general concept of "slots") is that the class, not the object, is responsible for doing the attribute name->storage_address mapping, and in intended use an object of a class with __slots__ doesn't even have a __dict__ (each __slot__ attribute is allocated at a fixed offset from the start of the object, saving tons of storage).
Yes, that is one of the reasons I want to use them. I've read the paragraph and my eyes are bleeding. I'm still trying to understand why the definition in that pargraph cased the order to be reverse. Why idoes the dynamic class attribute overriding an instance attribute? The attributes added by __slots__ have __set__ and __get__ methods so it should take precedence according to the definition. The __slots__ names could have been used prior to the names in the dictionary in the class, just like the __dict__ in an old class instance object was searched prior to the __dict__ in the class definition.
b = B() b.a = 4 b.a
4
B.a = 6
Here you overwrote the descriptor that allows b.a to mean something sensible (you nuked the <member 'a' of 'B' objects> thingie that maps 'a' to its storage address). Now B.a is an ordinary class attribute, and remember that b doesn't have a __dict__ (which you asked for, by using __slots__; you're not required to use __slots__).
Do you consider it a good thing that a class attribute can be introduced outside of the class definition? Given the nature of the changes being introduced with the new type system I would think the ability to add names to a class dictionary from outside the class would be turned off for the new type classes. If not for all types then at least for cases where the __slot__ mechanism is being used to define members. The slots mechanism also allows the same name to be used twice in the definition of a class.
class B(object): __slots__ = ['a','b','c','c']
I suspect this is going to cause a slot to be created for the first 'c' and then covered up by the second 'c'.
class B(object): __slots__ = ['a','b','c']
class C(B): __slots__ = ['d','a']
In this example the same thing will happen when names are redefined in a subclass.
b.a = 8
Traceback (most recent call last): File "<pyshell#61>", line 1, in ? b.a = 8 AttributeError: 'B' object attribute 'a' is read-only
I agree it's an odd msg, but I'm not sure it can do better easily: by overwriting B.a (which was nuts -- you're exploring pathologies here, not intended usage),
Why not make the overwriting of B.a generate an error message?
you've left b as an object with an 'a' attribute inherited from its class, but also as an object that can't grow new attributes of its own. Python looks at "hmm, I *can't* set 'a', but I do *have* an 'a'", and comes up with "read-only".
Try your example again without using __slots__ (you do *not* want __slots__ if you intend an object's namespace to be dynamic -- __slots__ announces that you guarantee the set of object attributes is fixed at class creation
If I don't want the object's namespace to be dynamic then the class in which it is defined should not be dynamic either.
time):
class B(object): pass
...
b = B() b.a = 4 B.a = 6 b.a
4
b.a = 8 b.a
8
B.a
6
IOW, don't use new features if you don't want new semantics, and things look much the same. If you want __slots__, though, there was no way to get its effect prior to 2.2 short of writing an ExtensionClass in C.
I was looking for ways in which people could mess data iin objects from outside of those objects. The current semantics will allow people to do dumb thinks that will break code silently rather than raising an error. I need and like the new semantics. I have extending them to allow optional type checking. This will eliminate piles of ugly classes that wrap attribute access with isinstance type checks if they are added to the member descriptor and handled automatically by the set and get functions.
participants (5)
-
David Ascher -
M.-A. Lemburg -
Michael McLay -
Thomas Heller -
Tim Peters