[Python-Dev] Meta-reflections

[Moore, Paul]

From: Kevin Jacobs [mailto:jacobs@penguin.theopalgroup.com]
> On Thu, 21 Feb 2002, Moore, Paul wrote:
> > > I agree, but that "official" support has clear limitations.
> >
> > I'm not sure what you mean.
> 
> When you request dir(object), there is a fairly significant 
> amount of work done.
[...]
> I know that we are talking about Python, and performance is 
> not of paramount importance.

Hmm. I tend to favour "do it right, then do it fast". If there's a
performance hit on dir(), why can't it be made faster? If nothing else, as a
part of the core, dir() has the right to access __dict__ and __slots__. So
there's no a priori reason why dir() should be slower than *any* user-coded
way of doing the same.

Of course, we *really* want vars() here, as we're otherwise doing work in
dir() to get entries that we then throw away. But that's the only issue. Get
vars() to work, and if it's too slow, you can argue that it's a bug because
"I can get the same results by using the following code, which is faster".

> I'm just not looking forward to 25% slowdowns in pickling
> (number pulled out of hat) and I'm sure the Zope guys
> aren't either...

There's bound to be some slowdown, from the (new) need to find slots as well
as dict-based attributes. I'm happy if you want it minimised. But that's a
new point you've raised, which I don't have the expertise to comment on.

> That is not the bug -- if for no other reason, the standard 
> library is free to use implementation specific knowledge.  Getting 
> obj.__dict__ is a really slick and efficient way to reflect
> on all normal instance variables.

I'm not sure I agree here - it's better if the standard library uses
interfaces which are available to the user. And if pickling can be made
fast, why shouldn't the machinery that makes this possible be made available
to the end user?

You could say that this argues in favour of making __dict__ and __slots__
part of the "official" reflection API. My view is that it argues for making
the "official" API (which I'm assuming will be vars() for now) efficient
enough that people don't need to use __disct__ and __slots__. Encapsulation
is good.

> I don't see how filling slots with default values is 
> compatible with the premise that we want slots to act
> as close to normal instance attributes as possible.

Fair enough. I offered that as one option. Clearly you prefer the other
(that what's in dir() and/or __slots__ cannot be guaranteed not to raise
AttributeError). I'm happy either way, not having a vested interest in the
issue.

> > Why not just change the line stuff = object.__dict__ to
> >
> >     stuff = [a for a in dir(object) if hasattr(object,a) and not
> > callable(getattr(object,a))]
> 
> Um, because its wrong?

Sorry - it was an off-the-top-of-the-head suggestion. But it made my real
point, which was that you can do it with dir().

> stuff = dict([ (a,getattr(object,a)) for a in vars(object) 
> if hasattr(object,a)])
> 
> Note that it does an unnecessary getattr, hasattr, memory 
> allocation and incurs loop overhead on every dict attribute,
> but otherwise it should work once vars is fixed.

Efficiency again. I'd have to bow to your greater experience here. Although
with pickling, doesn't I/O usually outweigh any performance cost?

> > 3. Document __dict__ as legacy usage, not slots-aware
> 
> Agree, though __dict__ should still be a valid way of 
> accessing all non-slot instance attributes.  Too much
> legacy code would break if this were not so.

That's what I meant. Document it as the historical way of getting at
instance attributes. Still available, but code which uses it will not
support slots. After all, if you pass classes using slots into code which
uses __dict__, things will go wrong. That's just another sort of breakage.
Nobody's arguing that __dict__ should go away. Except possibly from the
documentation :-)

> Calling base class setattr?  I'm not sure what you mean?

It's in the part of the descrintro document I pointed you at. Traditional
implementations of setattr used assignment to self.__dict__['attr'] to avoid
infinite recursion. The "new way" discussed in the descrintro document is to
call the base class setattr.

> By your logic, people don't have any business reading 
> __dict__, but they do.

They don't have any business *any more*. An important distinction. (And it's
not anywhere near as black and white as that comment implies - I know that).

> Imagine what would happen if we didn't expose __dict__ in Python 2.3?

Nothing at all, if we provide an alternative. Except for backward
compatibility issues, which there's a well-documented deprecation process to
address. Of course, nobody is proposing the removal of __dict__. All I'm
suggesting is that we document its limitations, point out better ways, and
leave it at that.

Paul.