[Python-bugs-list] [ python-Bugs-493462 ] Attribute Search in MRO w/o tp_getattr
Mon, 17 Dec 2001 05:37:38 -0800
Bugs item #493462, was opened at 2001-12-14 13:17
You can respond by visiting:
Category: Type/class unification
Group: Python 2.2
Resolution: Wont Fix
Submitted By: Robert Stewart (rstewart)
Assigned to: Guido van Rossum (gvanrossum)
Summary: Attribute Search in MRO w/o tp_getattr
In writing a metatype, I wanted to resolve attribute
references via a tp_getattr-style function -- with
either a char * or a Python string. Unfortunately,
PyObject_GenericGetAttr() first tries _PyType_Lookup()
to find a descriptor via base class dictionaries.
Searching according to the MRO is fine. The problem
is that _PyType_Lookup() only searches the dictionary
of each base class. It never checks those bases to
see it tp_getattr is set.
My workaround, which is not ideal, is to implement my
own tp_getattro function which first calls
PyObject_GenericGetAttr() to locate the attribute. If
that succeeds, that attribute is returned. If it
fails, then the tp_getattr-style function is called to
locate the attribute. This approach will return a
base class attribute when the derived class overrides
it, which is why I said it was not ideal.
The only way I can improve this process is to write my
own replacement for PyObject_GenericGetAttr() and
_PyType_Lookup(). That would make my code very
brittle, so that's not a good solution.
What I'd like to see is for _PyType_Lookup() to try
the dictionary, as it does now. If that fails, I'd
like it to check to see if tp_getattr is set. If so,
I'd like it to call the tp_getattr function to resolve
the attribute. Perhaps this change violates some
ideal of Python that I can't foresee, but it would
sure work great for me.
>Comment By: Robert Stewart (rstewart)
Date: 2001-12-17 05:37
Logged In: YES
Let me explain in more detail and, hopefully, answer your
questions. I'm sure there's something better I can do, but
I do have performance constraints which may override
anything else. (This is for a real time financial program,
so I need to keep overhead to a minimum.)
> When you talk about the bases and their tp_getattr,
> are you talking about the tp_getattr of the base
> type, or of its metatype?
I'm referring to the base type so that the lookup occurs on
instances of the Python class derived from that base.
Perhaps my terminology is too loose. I've created some
metatypes (right?) that are created in C++ for derivation
by Python programmers.
Several of the metatypes include the special behavior of
hooking the class parsing by including a meta-metatype (or
is it a metatype type?) whose tp_new creates the class
object and then searches for and saves certain methods that
will be used to override virtual functions in C++. (The
C++ virtual functions will check to see whether the Python
programmer defined the corresponding method and, if so,
calls that method.)
This has the behavior of only crossing into Python when the
Python programmer wants it. That is, my metatype does not
define those methods, so unless the derived class does, no
Python calls will be made.
> (Note: forget about tp_getattr -- you should leave
> that NULL, and only implement tp_getatto.)
I already have functionality in place -- from implementing
many extension types -- that relies on the tp_getattr
mechanism to map more easily into C++. It's that
functionality that I'm trying to use during attribute
lookup on the Python instance. Implementing the
tp_getattro mechanism means that I must put objects into
the metatype's dictionary and rely on Python to locate them
and call them, which in turn relies on my defining
extern "C" functions which can call the C++ code. I'm
trying to avoid that overhead. Is there something I'm
> PyObject_GenericGetAttr() should only be used in
> situations where the base type's tp_getattro is also
There's no documentation to this effect that I've seen.
> Why do you think that writing your own replacement
> for PyObject_GenericGetAttr() and _PyType_Lookup()
> would be brittle? That's what you are *supposed* to
> do if you're not happy with them.
The mechanism for locating attributes changed drastically
in Python 2.2. The logic in PyObject_GenericGetAttr() and
_PyType_Lookup() changed accordingly. Any version of those
functions that I might write will be completely dependent
upon the implementation of many of the underlying details
such as tp_mro, tp_getattro, etc. Furthermore, the correct
behavior of the Python derived class depends upon a correct
implementation of tp_getattro. Thus, since the
documentation for PyObject_GenericGetAttr() does not detail
its contract with the caller, how can I write an equivalent
function that won't break when the next version of Python
is released? You could change any underlying aspect and
break my implementation of tp_getattro. That's why I think
it is brittle.
IOW, if you supplied more fundamental building blocks by
which I could implement my own tp_getattro -- indeed, upon
which PyObject_GenericGetAttr() is built -- then I would
rely on you to make the low level implementation details
right. That means I only have to keep the higher level
details of tp_getattro right.
For example, _PyType_Lookup() is documented as being
an "Internal API." By definition, that means that what it
is not something I should need to be concerned with.
Clearly, the implementation of _PyType_Lookup() is not
rocket science, but it seems like a reasonable piece to
expose to developers like me. Why should we have to
reimplement such a fundamental piece if you expect us to
implement our own tp_getattro functions? (Mind you, as
implemented, _PyType_Lookup() doesn't do what I want, so
I'd need to reimplement it for my purposes, but the point
is valid in general.)
As another example, the high level logic of
PyObject_GenericGetAttr() is not at all obvious to me. I
haven't studied descriptors in detail, so that aspect of
the behavior is fuzzy. Nevertheless,
PyObject_GenericGetAttr() does some oddball things: it
uses _PyType_Lookup() to locate something that is
ostensibly a descriptor, since it's saved in "descr", and,
if "descr" refers to the right kind of thing, it is called;
but, after some other gamesmanship (_PyObject_GetDictPtr(),
etc.) looking for something else, "descr" is called.
That behavior is not obvious, not documented, and closely
tied to the underlying implementation of types. That
doesn't bode well for me implementing a replacement.
> What do you want to do with your metatype? Why do
> you need one? Have you considered prototyping it in
The metatype is a means to allow the Python programmer to
implement what our C++ programmers can. In C++, a
programmer derives from certain classes and overrides
various virtual functions. In Python, a programmer derives
from metatypes named the same as the C++ base classes and
implements methods that match the C++ virtual function
signatures for those s/he wishes to override.
As I said in the beginning, this has to handle real time
data, so I need to eliminate as much overhead as I can:
milliseconds are significant. Thus, I'm trying to handle
as much as I can myself, and in C++, without writing code
that is closely coupled to the current implementation of
Python. If there's a published API for my purpose, I'll
gladly use it. If not, I'm wary.
So, what have I missed? Is there a better way to do what I
want? I don't have to use the tp_getattr mechanism, but I
do need a high performance attribute lookup/C++ call
mechanism. Thus, my thought was that _PyType_Lookup()
could check to see whether the type object set tp_getattr.
If so, it could call that function before moving on to the
next type in the MRO. This would allow developers like me
the flexibility to put things in the dictionary or to use a
dispatch function like tp_getattr (or both) as we see fit.
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-14 13:41
Logged In: YES
When you talk about the bases and their tp_getattr, are you
talking about the tp_getattr of the base type, or of its
metatype? (Note: forget about tp_getattr -- you should leave
that NULL, and only implement tp_getatto.)
PyObject_GenericGetAttr() should only be used in situations
where the base type's tp_getattro is also
Why do you think that writing your own replacement for
PyObject_GenericGetAttr() and _PyType_Lookup() would be
brittle? That's what you are *supposed* to do if you're not
happy with them.
What do you want to do with your metatype? Why do you need
one? Have you considered prototyping it in Python?
You can respond by visiting: