[Python-bugs-list] [ python-Bugs-493462 ] Attribute Search in MRO w/o tp_getattr

[noreply@sourceforge.net]

Bugs item #493462, was opened at 2001-12-14 13:17
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=493462&group_id=5470

Category: Type/class unification
Group: Python 2.2
Status: Closed
Resolution: Wont Fix
Priority: 5
Submitted By: Robert Stewart (rstewart)
Assigned to: Guido van Rossum (gvanrossum)
Summary: Attribute Search in MRO w/o tp_getattr

Initial Comment:
In writing a metatype, I wanted to resolve attribute 
references via a tp_getattr-style function -- with 
either a char * or a Python string.  Unfortunately, 
PyObject_GenericGetAttr() first tries _PyType_Lookup() 
to find a descriptor via base class dictionaries.

Searching according to the MRO is fine.  The problem 
is that _PyType_Lookup() only searches the dictionary 
of each base class.  It never checks those bases to 
see it tp_getattr is set.

My workaround, which is not ideal, is to implement my 
own tp_getattro function which first calls 
PyObject_GenericGetAttr() to locate the attribute.  If 
that succeeds, that attribute is returned.  If it 
fails, then the tp_getattr-style function is called to 
locate the attribute.  This approach will return a 
base class attribute when the derived class overrides 
it, which is why I said it was not ideal.

The only way I can improve this process is to write my 
own replacement for PyObject_GenericGetAttr() and 
_PyType_Lookup().  That would make my code very 
brittle, so that's not a good solution.

What I'd like to see is for _PyType_Lookup() to try 
the dictionary, as it does now.  If that fails, I'd 
like it to check to see if tp_getattr is set.  If so, 
I'd like it to call the tp_getattr function to resolve 
the attribute.  Perhaps this change violates some 
ideal of Python that I can't foresee, but it would 
sure work great for me.

----------------------------------------------------------------------

>Comment By: Robert Stewart (rstewart)
Date: 2001-12-17 05:37

Message:
Logged In: YES 
user_id=402463

Guido,

Let me explain in more detail and, hopefully, answer your 
questions.  I'm sure there's something better I can do, but 
I do have performance constraints which may override 
anything else.  (This is for a real time financial program, 
so I need to keep overhead to a minimum.)

> When you talk about the bases and their tp_getattr,
> are you talking about the tp_getattr of the base 
> type, or of its metatype?

I'm referring to the base type so that the lookup occurs on 
instances of the Python class derived from that base.  
Perhaps my terminology is too loose.  I've created some 
metatypes (right?) that are created in C++ for derivation 
by Python programmers.

Several of the metatypes include the special behavior of 
hooking the class parsing by including a meta-metatype (or 
is it a metatype type?) whose tp_new creates the class 
object and then searches for and saves certain methods that 
will be used to override virtual functions in C++.  (The 
C++ virtual functions will check to see whether the Python 
programmer defined the corresponding method and, if so, 
calls that method.)

This has the behavior of only crossing into Python when the 
Python programmer wants it.  That is, my metatype does not 
define those methods, so unless the derived class does, no 
Python calls will be made.

> (Note: forget about tp_getattr -- you should leave 
> that NULL, and only implement tp_getatto.)

I already have functionality in place -- from implementing 
many extension types -- that relies on the tp_getattr 
mechanism to map more easily into C++.  It's that 
functionality that I'm trying to use during attribute 
lookup on the Python instance.  Implementing the 
tp_getattro mechanism means that I must put objects into 
the metatype's dictionary and rely on Python to locate them 
and call them, which in turn relies on my defining 
extern "C" functions which can call the C++ code.  I'm 
trying to avoid that overhead.  Is there something I'm 
missing?

> PyObject_GenericGetAttr() should only be used in 
> situations where the base type's tp_getattro is also 
> PyObject_GenericGetAttr().

There's no documentation to this effect that I've seen.

> Why do you think that writing your own replacement 
> for PyObject_GenericGetAttr() and _PyType_Lookup() 
> would be brittle?  That's what you are *supposed* to 
> do if you're not happy with them.

The mechanism for locating attributes changed drastically 
in Python 2.2.  The logic in PyObject_GenericGetAttr() and 
_PyType_Lookup() changed accordingly.  Any version of those 
functions that I might write will be completely dependent 
upon the implementation of many of the underlying details 
such as tp_mro, tp_getattro, etc.  Furthermore, the correct 
behavior of the Python derived class depends upon a correct 
implementation of tp_getattro.  Thus, since the 
documentation for PyObject_GenericGetAttr() does not detail 
its contract with the caller, how can I write an equivalent 
function that won't break when the next version of Python 
is released?  You could change any underlying aspect and 
break my implementation of tp_getattro.  That's why I think 
it is brittle.

IOW, if you supplied more fundamental building blocks by 
which I could implement my own tp_getattro -- indeed, upon 
which PyObject_GenericGetAttr() is built -- then I would 
rely on you to make the low level implementation details 
right.  That means I only have to keep the higher level 
details of tp_getattro right.

For example, _PyType_Lookup() is documented as being 
an "Internal API."  By definition, that means that what it 
is not something I should need to be concerned with.  
Clearly, the implementation of _PyType_Lookup() is not 
rocket science, but it seems like a reasonable piece to 
expose to developers like me.  Why should we have to 
reimplement such a fundamental piece if you expect us to 
implement our own tp_getattro functions?  (Mind you, as 
implemented, _PyType_Lookup() doesn't do what I want, so 
I'd need to reimplement it for my purposes, but the point 
is valid in general.)

As another example, the high level logic of 
PyObject_GenericGetAttr() is not at all obvious to me.  I 
haven't studied descriptors in detail, so that aspect of 
the behavior is fuzzy.  Nevertheless, 
PyObject_GenericGetAttr() does some oddball things:  it 
uses _PyType_Lookup() to locate something that is 
ostensibly a descriptor, since it's saved in "descr", and, 
if "descr" refers to the right kind of thing, it is called; 
but, after some other gamesmanship (_PyObject_GetDictPtr(), 
etc.) looking for something else, "descr" is called.

That behavior is not obvious, not documented, and closely 
tied to the underlying implementation of types.  That 
doesn't bode well for me implementing a replacement.

> What do you want to do with your metatype?  Why do 
> you need one?  Have you considered prototyping it in 
> Python?

The metatype is a means to allow the Python programmer to 
implement what our C++ programmers can.  In C++, a 
programmer derives from certain classes and overrides 
various virtual functions.  In Python, a programmer derives 
from metatypes named the same as the C++ base classes and 
implements methods that match the C++ virtual function 
signatures for those s/he wishes to override.

As I said in the beginning, this has to handle real time 
data, so I need to eliminate as much overhead as I can: 
milliseconds are significant.  Thus, I'm trying to handle 
as much as I can myself, and in C++, without writing code 
that is closely coupled to the current implementation of 
Python.  If there's a published API for my purpose, I'll 
gladly use it.  If not, I'm wary.

So, what have I missed?  Is there a better way to do what I 
want?  I don't have to use the tp_getattr mechanism, but I 
do need a high performance attribute lookup/C++ call 
mechanism.  Thus, my thought was that _PyType_Lookup() 
could check to see whether the type object set tp_getattr.  
If so, it could call that function before moving on to the 
next type in the MRO.  This would allow developers like me 
the flexibility to put things in the dictionary or to use a 
dispatch function like tp_getattr (or both) as we see fit.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-14 13:41

Message:
Logged In: YES 
user_id=6380

When you talk about the bases and their tp_getattr, are you
talking about the tp_getattr of the base type, or of its
metatype? (Note: forget about tp_getattr -- you should leave
that NULL, and only implement tp_getatto.)

PyObject_GenericGetAttr() should only be used in situations
where the base type's tp_getattro is also
PyObject_GenericGetAttr().

Why do you think that writing your own replacement for 
PyObject_GenericGetAttr() and _PyType_Lookup() would be
brittle? That's what you are *supposed* to do if you're not
happy with them.

What do you want to do with your metatype? Why do you need
one? Have you considered prototyping it in Python?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=493462&group_id=5470