super_getattro() Behaviour
In PyQt, wrapped types implement lazy access to the type dictionary through tp_getattro. If the normal attribute lookup fails, then private tables are searched and the attribute (if found) is created on the fly and returned. It is also put into the type dictionary so that it is found next time through the normal lookup. This is done to speed up the import of, and the memory consumed by, the qt module which contains thousands of class methods. This all works fine - except when super is used. The implementation of super_getattro() doesn't use the normal attribute lookup (ie. doesn't go via tp_getattro). Instead it walks the MRO hierarchy itself and searches instance dictionaries explicitly. This means that attributes that have not yet been referenced (ie. not yet been cached in the type dictionary) will not be found. Questions... 1. What is the reason why it doesn't go via tp_getattro? Bug or feature? 2. A possible workaround is to subvert the ma_lookup function of the type dictionary after creating the type to do something similar to what my tp_getattro function is doing. Are there any inherent problems with that? 3. Why, when creating a new type and eventually calling type_new() is a copy of the dictionary passed in made? Why not take a reference to it? This would allow a dict sub-class to be used as the type dictionary. I could then implement a lazy-dict sub-class with the behaviour I need. 4. Am I missing a more correct/obvious technique? (There is no need to support classic classes.) Many thanks, Phil
"Phil Thompson"
In PyQt, wrapped types implement lazy access to the type dictionary through tp_getattro. If the normal attribute lookup fails, then private tables are searched and the attribute (if found) is created on the fly and returned. It is also put into the type dictionary so that it is found next time through the normal lookup. This is done to speed up the import of, and the memory consumed by, the qt module which contains thousands of class methods.
This all works fine - except when super is used.
The implementation of super_getattro() doesn't use the normal attribute lookup (ie. doesn't go via tp_getattro). Instead it walks the MRO hierarchy itself and searches instance dictionaries explicitly. This means that attributes that have not yet been referenced (ie. not yet been cached in the type dictionary) will not be found.
Questions...
1. What is the reason why it doesn't go via tp_getattro?
Because it wouldn't work if it did? I'm not sure what you're suggesting here.
2. A possible workaround is to subvert the ma_lookup function of the type dictionary after creating the type to do something similar to what my tp_getattro function is doing.
Eek!
Are there any inherent problems with that?
Well, I think the layout of dictionaries is fiercely private. IIRC, the only reason it's in a public header is to allow some optimzations in ceval.c (though this isn't at all obvious from the headers, so maybe I'm mistaken).
3. Why, when creating a new type and eventually calling type_new() is a copy of the dictionary passed in made?
I think this is to prevent changes to tp_dict behind the type's back. It's important to keep the dict and the slots in sync.
Why not take a reference to it? This would allow a dict sub-class to be used as the type dictionary. I could then implement a lazy-dict sub-class with the behaviour I need.
Well, not really, because super_getattro uses PyDict_GetItem, which doesn't respect subclasses...
4. Am I missing a more correct/obvious technique? (There is no need to support classic classes.)
Hum, I can't think of one, I'm afraid. There has been some vague talk of having a tp_lookup slot in typeobjects, so PyDict_GetItem(t->tp_dict, x); would become t->tp_lookup(x); (well, ish, it might make more sense to only do that if the dict lookup fails). For now, not being lazy seems your only option :-/ (it's what PyObjC does). Cheers, mwh -- Many of the posts you see on Usenet are actually from moths. You can tell which posters they are by their attraction to the flames. -- Internet Oracularity #1279-06
"Phil Thompson"
writes: In PyQt, wrapped types implement lazy access to the type dictionary through tp_getattro. If the normal attribute lookup fails, then private tables are searched and the attribute (if found) is created on the fly and returned. It is also put into the type dictionary so that it is found next time through the normal lookup. This is done to speed up the import of, and the memory consumed by, the qt module which contains thousands of class methods.
This all works fine - except when super is used.
The implementation of super_getattro() doesn't use the normal attribute lookup (ie. doesn't go via tp_getattro). Instead it walks the MRO hierarchy itself and searches instance dictionaries explicitly. This means that attributes that have not yet been referenced (ie. not yet been cached in the type dictionary) will not be found.
Questions...
1. What is the reason why it doesn't go via tp_getattro?
Because it wouldn't work if it did? I'm not sure what you're suggesting here.
I'm asking for an explanation for the current implementation. Why wouldn't it work if it got the attribute via tp_getattro?
2. A possible workaround is to subvert the ma_lookup function of the type dictionary after creating the type to do something similar to what my tp_getattro function is doing.
Eek!
Agreed.
Are there any inherent problems with that?
Well, I think the layout of dictionaries is fiercely private. IIRC, the only reason it's in a public header is to allow some optimzations in ceval.c (though this isn't at all obvious from the headers, so maybe I'm mistaken).
Yes, having looked in more detail at the dict implementation I really don't want to go there.
3. Why, when creating a new type and eventually calling type_new() is a copy of the dictionary passed in made?
I think this is to prevent changes to tp_dict behind the type's back. It's important to keep the dict and the slots in sync.
Why not take a reference to it? This would allow a dict sub-class to be used as the type dictionary. I could then implement a lazy-dict sub-class with the behaviour I need.
Well, not really, because super_getattro uses PyDict_GetItem, which doesn't respect subclasses...
I suppose I was hoping for more C++ like behaviour.
4. Am I missing a more correct/obvious technique? (There is no need to support classic classes.)
Hum, I can't think of one, I'm afraid.
There has been some vague talk of having a tp_lookup slot in typeobjects, so
PyDict_GetItem(t->tp_dict, x);
would become
t->tp_lookup(x);
(well, ish, it might make more sense to only do that if the dict lookup fails).
That would be perfect. I can't Google any reference to a discussion - can you point me at something?
For now, not being lazy seems your only option :-/ (it's what PyObjC does).
Not practical I'm afraid. I think I can only document that super doesn't work in this context. Thanks, Phil
"Phil Thompson"
Questions...
1. What is the reason why it doesn't go via tp_getattro?
Because it wouldn't work if it did? I'm not sure what you're suggesting here.
I'm asking for an explanation for the current implementation. Why wouldn't it work if it got the attribute via tp_getattro?
Well, using type->tp_getattro is just different to looking in tp_dict -- it finds metamethods, for example. Hmm. Well, I'm fairly sure there is a difference, I'm not sure I can explain it right now :(
2. A possible workaround is to subvert the ma_lookup function of the type dictionary after creating the type to do something similar to what my tp_getattro function is doing.
[...]
Yes, having looked in more detail at the dict implementation I really don't want to go there.
Good :)
4. Am I missing a more correct/obvious technique? (There is no need to support classic classes.)
Hum, I can't think of one, I'm afraid.
There has been some vague talk of having a tp_lookup slot in typeobjects, so
PyDict_GetItem(t->tp_dict, x);
would become
t->tp_lookup(x);
(well, ish, it might make more sense to only do that if the dict lookup fails).
That would be perfect. I can't Google any reference to a discussion - can you point me at something?
Well, most of the discussion so far has been in my head :) There was a little talk of it in the thread "can we stop pretending _PyType_Lookup is internal" here and possibly on pyobjc-dev around the same time. I'm not that likely to work on it soon -- I have enough moderately complex patches to core Python I'm persuading people to think about :-/.
For now, not being lazy seems your only option :-/ (it's what PyObjC does).
Not practical I'm afraid. I think I can only document that super doesn't work in this context.
Oh well. I can't even think of a way to make it fail reliably... Cheers, mwh -- Java sucks. [...] Java on TV set top boxes will suck so hard it might well inhale people from off their sofa until their heads get wedged in the card slots. --- Jon Rabone, ucam.chat
4. Am I missing a more correct/obvious technique? (There is no need to support classic classes.)
Hum, I can't think of one, I'm afraid.
There has been some vague talk of having a tp_lookup slot in typeobjects, so
PyDict_GetItem(t->tp_dict, x);
would become
t->tp_lookup(x);
(well, ish, it might make more sense to only do that if the dict lookup fails).
That would be perfect. I can't Google any reference to a discussion - can you point me at something?
Well, most of the discussion so far has been in my head :)
There was a little talk of it in the thread "can we stop pretending _PyType_Lookup is internal" here and possibly on pyobjc-dev around the same time.
I'm not that likely to work on it soon -- I have enough moderately complex patches to core Python I'm persuading people to think about :-/.
Anything I can do to help push it along? Phil
participants (2)
-
Michael Hudson
-
Phil Thompson