Re: [Python-Dev] Meta-reflections
I fully intend to provide a reference implementation of some of these ideas. In fact, its likely to be a fairly small patch. However, I still don't know what the ideal semantics are. I would very much value your input on the matter, even if on a purely theoretical level. So, lets start with the premise that __attrs__ is a declaration like __slots__ except that:
It seems relevant to me that your choice of name ("attrs") indicates a relationship with attributes - which your ideas seem to deny. It is important to make the distinction entirely clear, otherwise your points are going to get obscured. I've been having a hard time keeping track of precisely what you are suggesting (on which basis, thanks for this summary...)
1) the namespace of these 'attrs' is flat -- repeated names in descendant classes either results in an error or silently re-using the existing slot. This maintains the traditional flat instance namespace of attributes.
FWIW, I disagree with this completely. I would expect slots with a particular name in derived classes to hide the same-named slots in base classes. Whether or not the base class slot is available via some sort of "super" shenannigans is less relevant. But hiding semantics is critical. How do you expect to reliably make it possible to derive from a class with slots otherwise? Perl gets into this sort of mess with its implementation of objects.
2) A complete and immutable list of slots is available as a member of each type to allow for easy and efficient reflection. (though I am also in favor of working on better formal reflection APIs)
Agreed - up to a point. I don't see a need for a way to distinguish between slots and "normal" attributes, personally. But I don't do anything fancy here, so my experience isn't very indicative. I'm more or less happy with dir() as a start, although I agree that a better formal reflection API would be helpful. I suspect that such a thing could be built in Python on top of the existing facilities, however...
3) These 'attrs' are to otherwise have the same semantics as normal __dict__ instance attributes. e.g., they should be automatically picklable, they can have properties assigned to them, etc.
I think I agree here. However, if you want slots to behave like normal attributes, except for the flat namespace, I see no value. Why have the exception at all? Hmm, this raises the question of why we have slots at all. If they act exactly like attributes, why have them? As a user, I perceive them as an efficiency thing - they save the memory associated with a dictionary, and are probably faster as well. There can be tradeoffs which you pay for that efficiency, but that's all. No semantic difference. Actually, that's pretty much how the descrintro document describes slots. Strange that... I guess I am saying that I'm happy with slots as designed (documented in descrintro) - modulo some implementation bugs such as not getting automatically pickled. Paul.
Paul, thanks for the very constructive feedback! On Wed, 20 Feb 2002, Moore, Paul wrote:
I fully intend to provide a reference implementation of some of these ideas. In fact, its likely to be a fairly small patch. However, I still don't know what the ideal semantics are. I would very much value your input on the matter, even if on a purely theoretical level. So, lets start with the premise that __attrs__ is a declaration like __slots__ except that:
It seems relevant to me that your choice of name ("attrs") indicates a relationship with attributes - which your ideas seem to deny.
You are right -- lets call them 'slotattrs', since they should ideally have virtually the same semantics as attributes, except they are allocated like slots.
1) the namespace of these 'attrs' is flat -- repeated names in descendant classes either results in an error or silently re-using the existing slot. This maintains the traditional flat instance namespace of attributes.
FWIW, I disagree with this completely. I would expect slots with a particular name in derived classes to hide the same-named slots in base classes. Whether or not the base class slot is available via some sort of "super" shenannigans is less relevant. But hiding semantics is critical. How do you expect to reliably make it possible to derive from a class with slots otherwise? Perl gets into this sort of mess with its implementation of objects.
Attributes currently have a flat namespace, and the construct that I feel is most natural would maintain that characteristic. e.g.: class Base: def __init__(self): self.foo = 1 class Derived(Base): def __init__(self): Base.__init__(self) self.foo = 2 # this is the same foo as in Base Python already implements a form data hiding semantics in a different way, so I'm not sure it is a good idea to add another ad-hoc method to do the same thing. The current way to implement data hiding is by using the namespace munging trick by prefixing symbols with '__', which is then munged by prepending an underscore and the name of the class. class Foo: __var = 1 dir(Foo)
['_Foo__var', '__doc__', '__module__']
2) A complete and immutable list of slots is available as a member of each type to allow for easy and efficient reflection. (though I am also in favor of working on better formal reflection APIs)
Agreed - up to a point. I don't see a need for a way to distinguish between slots and "normal" attributes, personally. But I don't do anything fancy here, so my experience isn't very indicative.
Without a more formal reflection API, the traditional way to get all normal dictionary attributes is by using instance.__dict__.keys(). All I'm proposing is that instance.__slotattrs__ (or possibly instance.__class__.__slotattrs__) returns a list of objects that reveal the name of all slots in the instance (including those declared in base classes). I am not sure what that list should look like, though here are the current suggestions: 1) __slotattrs__ = ('a','b') 2) # slot descriptor objects -- the repr is shown here __slotattrs__ = ('<member 'a' of 'Baz' objects>', <member 'b' of 'Baz' objects>') The only issue that concerns me is that I am not sure if the slot to slot name mapping should be fixed. The intrinsic definition of a slot is a type and the offset of the slot in the type. The name is just a binding to a slot descriptor, so it "feels" unnecessary to make that immutable. It either case, it is not a big issue for me.
3) These 'attrs' are to otherwise have the same semantics as normal __dict__ instance attributes. e.g., they should be automatically picklable, they can have properties assigned to them, etc.
I think I agree here. However, if you want slots to behave like normal attributes, except for the flat namespace, I see no value. Why have the exception at all?
Attributes currently have a flat namespace? I must not have been clear -- I _do_ want my slotattrs to be allocated like slots, mainly for efficiency reasons.
Hmm, this raises the question of why we have slots at all. If they act exactly like attributes, why have them? As a user, I perceive them as an efficiency thing - they save the memory associated with a dictionary, and are probably faster as well. There can be tradeoffs which you pay for that efficiency, but that's all. No semantic difference. Actually, that's pretty much how the descrintro document describes slots. Strange that...
EXACTLY! I want to use slots (or slotattrs, or whatever you call them) to be solely an allocation declaration. For small objects that you expect to create many, many instance of, they make a huge difference. I have some results that I measured on various implementations of a class to represent database rows. The purpose of this class is to give simple dictionary-like attribute access semantics to tuples returned as a result of default DB-API relational database queries. The trick is to add the ability to access fields by name (instead of by only by index) without incurring the overhead of allocating a dictionary for every instance. Below are results of a benchmark that compare various ways of implementing this class: time SIZE (sec) Bytes/row -------- ------ --------- overhead: 4,744KB 0.56 - tuple: 18,948KB 2.49 73 C extension w/ slots: 18,924KB 4.85 73 native dict*: 117MB 13.50 589 Python class w/ slots: 18,960KB 17.23 73 Python class w/o slots: 117MB 24.09 589 * the native dict implementation does not allow indexed access, and is only included as a reference point. [For more details and discussion of this specific application, please see this URL: http://opensource.theopalgroup.com/ ] Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
On 20 Feb 2002 at 8:30, Kevin Jacobs wrote:
Attributes currently have a flat namespace,
Instance attributes do, but that's a tautology.
and the construct that I feel is most natural would maintain that characteristic. e.g.:
class Base: def __init__(self): self.foo = 1
class Derived(Base): def __init__(self): Base.__init__(self) self.foo = 2 # this is the same foo as in Base
But these aren't: class Base foo = 1 class Derived(Base): foo = 2 -- Gordon http://www.mcmillan-inc.com/
On Wed, 20 Feb 2002, Gordon McMillan wrote:
On 20 Feb 2002 at 8:30, Kevin Jacobs wrote:
Attributes currently have a flat namespace,
Instance attributes do, but that's a tautology.
2 o.__class__.__base__.foo = 3
2
Yes, though one implication of the new slots mechanism in Python 2.2 is that we now have have non-flat per-instance namespaces for slots. i.e., we would have per-instance slots that would hide other per-instance slots of the same name from ancestor classes: class Base(object): __slots__ = ['foo'] def __init__(self): self.foo = 1 # which slot this sets depends on type(self) # if type(self) == Base, then the slot is # described by Base.foo. # else if type(self) == Derived, then the # slot is described by Derived.foo class Derived(Base): __slots__ = ['foo'] def __init__(self): Base.__init__(self) self.foo = 2 # this is NOT the same foo as in Base o = Derived() print o.foo print o.foo print o.__class__.__base__.foo
3
So slots, as currently implemented, do not act like attributes, and this whole discussion revolves around whether they should or should not. Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs wrote:
Yes, though one implication of the new slots mechanism in Python 2.2 is that we now have have non-flat per-instance namespaces for slots. i.e., we would have per-instance slots that would hide other per-instance slots of the same name from ancestor classes:
class Base(object): __slots__ = ['foo'] def __init__(self): self.foo = 1 # which slot this sets depends on type(self) # if type(self) == Base, then the slot is # described by Base.foo. # else if type(self) == Derived, then the # slot is described by Derived.foo
class Derived(Base): __slots__ = ['foo'] def __init__(self): Base.__init__(self) self.foo = 2 # this is NOT the same foo as in Base
2 o.__class__.__base__.foo = 3
2
o = Derived() print o.foo print o.foo print o.__class__.__base__.foo
3
So slots, as currently implemented, do not act like attributes, and this whole discussion revolves around whether they should or should not.
This example is not a great example of that, since the code above does exactly the same thing if you delete the lines defining __slots__. You're modifying class attributes in that case, but I think it's important to keep the examples which illustrate the problem "pure" and "realistic". My take on this thread is that I think it's simply not going to happen that slots are going to act 100% like attributes except for performance/memory considerations. It would be nice, but if that had been possible, then they'd simply be an internal optimization and would have no syntactic impact. There are much more shallow ways in which slots aren't like attributes:
class A(object): ... __slots__ = ('a',) ... a = A() a.a = 123 # set a slot on a A.a = 555 # set a slot on A a.a # Way 1: A's slot overrides a's 555 b = A() b.a 555 del A.a a.a # Way 2: deleting the class slot # did not uncover the instance slot AttributeError: 'A' object has no attribute 'a'
--david
On Wed, 20 Feb 2002, David Ascher wrote:
Kevin Jacobs wrote: This example is not a great example of that, since the code above does exactly the same thing if you delete the lines defining __slots__.
True, which is why the current implementation (IMHO) isn't broken; just flawed. There is, in effect, a flat slot namespace, only by virtue of the fact that there is no simple and explicit slot resolution syntax. This basically means that most arguments for using the current implementation of slots as a data-hiding mechanism over inheritance are very weak unless significant additional syntax is created.
You're modifying class attributes in that case, but I think it's important to keep the examples which illustrate the problem "pure" and "realistic".
Nope -- they aren't class attributes at all, they are per-instance slots with class-level descriptors (with which you expose another bug below).
My take on this thread is that I think it's simply not going to happen that slots are going to act 100% like attributes except for performance/memory considerations. It would be nice, but if that had been possible, then they'd simply be an internal optimization and would have no syntactic impact.
I'd like to know why else you think that? I'm fairly confident that I can submit a patch that accomplishes this (and even fix the following issue).
There are much more shallow ways in which slots aren't like attributes:
class A(object): ... __slots__ = ('a',) ... a = A() a.a = 123 # set a slot on a A.a = 555 # set a slot on A a.a # Way 1: A's slot overrides a's 555 b = A() b.a 555 del A.a a.a # Way 2: deleting the class slot # did not uncover the instance slot AttributeError: 'A' object has no attribute 'a'
Ouch! You've discovered another problem with the current implementation. You have effectively removed the slot descriptor from class A and replaced it with a class attribute. In fact, I don't think you can ever re-create the slot descriptor! This is actually the best form of data hiding in pure Python I've seen to date. The fix is to make slot descriptors read-only, like the rest of the immutible class attributes. Sigh, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin:
My take on this thread is that I think it's simply not going to happen that slots are going to act 100% like attributes except for performance/memory considerations. It would be nice, but if that had been possible, then they'd simply be an internal optimization and would have no syntactic impact.
I'd like to know why else you think that?
Ye old poverty of the imagination argument, coupled with the assumption that Guido had time to finish what he'd started I guess =).
I'm fairly confident that I can submit a patch that accomplishes this (and even fix the following issue).
Great! I can't channel Guido very well, but everytime that he's talked about slots in my presence, he talked about their main purpose as a memory-savings one. If he didn't have other intents, and if you can limit their impact to pure memory savings, then more power to all of us thanks to you.
Ouch! You've discovered another problem with the current implementation. You have effectively removed the slot descriptor from class A and replaced it with a class attribute. In fact, I don't think you can ever re-create the slot descriptor!
Ooh, cool. You're right: after deleting the class attribute:
a.a = 1100 Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'A' object has no attribute 'a'
which is a really wacky error message if you look at the case of the letters... --david
On Wed, 20 Feb 2002, David Ascher wrote:
My take on this thread is that I think it's simply not going to happen that slots are going to act 100% like attributes except for performance/memory considerations. It would be nice, but if that had been possible, then they'd simply be an internal optimization and would have no syntactic impact.
I'd like to know why else you think that?
Ye old poverty of the imagination argument, coupled with the assumption that Guido had time to finish what he'd started I guess =).
This is why I haven't unleashed a patch, even though I pretty much know exactly how to fix all of the problems we've noticed and make slots work as I imagine they should. Some of the things I heard Guido say at IPC10 lead me to believe that he has something up his sleeve wrt slots (specifically some plan about storing dicts in __slots__ that do something nifty). So I'm going to sit on my hands until Guido gets back into town and sets us all straight. Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
participants (4)
-
David Ascher -
Gordon McMillan -
Kevin Jacobs -
Moore, Paul