PEP 231, __findattr__()
I've just uploaded PEP 231, which describes a new hook in the instance access mechanism, called __findattr__() after a similar mechanism that exists in Jython (but is not exposed at the Python layer). You can do all kinds of interesting things with __findattr__(), including implement the __of__() protocol of ExtensionClass, and thus implicit and explicit acquisitions, in pure Python. You can also do Java Bean-like interfaces and C++-like access control. The PEP contains sample implementations of all of these, although the latter isn't as clean as I'd like, due to other restrictions in Python. My hope is that __findattr__() would eliminate most, if not all, the need for ExtensionClass, at least within the Zope and ZODB contexts. I haven't tried to implement Persistent using it though. Since it's a long PEP, I won't include it here. You can read about it at this URL http://python.sourceforge.net/peps/pep-0231.html It includes a link to the patch implementing this feature on SourceForge. Enjoy, -Barry
On Fri, 1 Dec 2000, Barry A. Warsaw wrote:
There's one thing that bothers me about this: what exactly is "the call stack"? Let me clarify: what happens when you have threads. Either machine-level threads and stackless threads confuse the issues here, not to talk about stackless continuations. Can you add a few words to the PEP about dealing with those?
Moshe Zadka wrote:
As far as I understood the patch (just skimmed), thee is no stack involved directly, but the instance increments and decrments a variable infindattr. + if (v != NULL && !inst->infindaddr && + (func = inst->in_class->cl_findattr)) + { + PyObject *args, *res; + args = Py_BuildValue("(OOO)", inst, name, v); + if (args == NULL) + return -1; + ++inst->infindaddr; + res = PyEval_CallObject(func, args); + --inst->infindaddr; This is: The call modifies the instance's state, while calling the findattr method. You are right: I see a serious problem with this. It doesn't even need continuations to get things messed up. Guido's proposed coroutines, together with uThread-Switching, might be able to enter the same instance twice with ease. Barry, after second thought, I feel this can become a problem in the future. This infindattr attribute only works correctly if we are guaranteed to use strict stack order of execution. What you're *intending* to to is to tell the PyEval_CallObject that it should not find the __findattr__ attribute. But this should be done only for this call and all of its descendants, but no *fresh* access from elsewhere. The hard way to get out of this would be to stop scheduling in that case. Maybe this is very cheap, but quite unelegant. We have a quite peculiar system state here: A function call acts like an escape, to make all subsequent calls behave differently, until this call is finished. Without blocking microthreads, a clean way to do this would be a search up in the frame chain, if there is a running __findattr__ method of this object. Fairly expensive. Well, the problem also exists with real threads, if they are allowed to switch in such a context. I fear it is necessary to either block this stuff until it is ready, or to maintain some thread-wise structure for the state of this object. Ok, after thinking some more, I'll start an extra message to Barry on this topic. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
"Barry A. Warsaw" wrote:
The PEP does define when and how __findattr__() is called, but makes no statement about what it should do or return... Here's a slightly different idea: Given the name, I would expect it to go look for an attribute and then return the attribute and its container (this doesn't seem to be what you have in mind here, though). An alternative approach given the semantics above would then be to first try a __getattr__() lookup and revert to __findattr__() in case this fails. I don't think there is any need to overload __setattr__() in such a way, because you cannot be sure which object actually gets the new attribute. By exposing the functionality using a new builtin, findattr(), this could be used for all the examples you give too. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M" == M <mal@lemburg.com> writes:
M> The PEP does define when and how __findattr__() is called, M> but makes no statement about what it should do or return... Good point. I've clarified that in the PEP. M> Here's a slightly different idea: M> Given the name, I would expect it to go look for an attribute M> and then return the attribute and its container (this doesn't M> seem to be what you have in mind here, though). No, because some applications won't need a wrapped object. E.g. in the Java bean example, it just returns the attribute (which is stored with a slightly different name). M> An alternative approach given the semantics above would then be M> to first try a __getattr__() lookup and revert to M> __findattr__() in case this fails. I don't think this is as useful. What would that buy you that you can't already do today? The key concept here is that you want to give the class first crack to interpose on every attribute access. You want this hook to get called before anybody else can get at, or set, your attributes. That gives you (the class) total control to implement whatever policy is useful. M> I don't think there is any need to overload __setattr__() in M> such a way, because you cannot be sure which object actually M> gets the new attribute. M> By exposing the functionality using a new builtin, findattr(), M> this could be used for all the examples you give too. No, because then people couldn't use the object in the normal dot-notational way. -Barry
"Barry A. Warsaw" wrote:
I was thinking of a standardised helper which could then be used for all kinds of attribute retrieval techniques. Acquisition would be easy to do, access control too. In most cases __findattr__ would simply return (self, self.attrname).
Forget that idea... *always* calling __findattr__ is the more useful way, just like you intended.
Right.
Uhm, why not ? -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Hi Barry, "Barry A. Warsaw" wrote:
I have been using ExtensionClass for quite a long time, and I have to say that you indeed eliminate most of its need through this super-elegant idea. Congratulations! Besides acquisition and persitency interception, wrapping plain C objects and giving them Class-like behavior while retaining fast access to internal properties but being able to override methods by Python methods was my other use of ExtensionClass. I assume this is the other "20%" part you mention, which is much harder to achieve? But that part also looks easier to implement now, by the support of the __findattr__ method.
Great. I had to read it twice, but it was fun. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
"Barry A. Warsaw" wrote:
Ok, as I announced already, here some thoughts on __findattr__, system state, and how it could work. Looking at your patch, I realize that you are blocking __findattr__ for your whole instance, until this call ends. This is not what you want to do, I guess. This has an effect of affecting the whole system state when threads are involved. Also you cannot use __findattr__ on any other attribute during this call. You want most probably do this: __findattr__ should not be invoked again for this instance, with this attribute name, for this "thread", until you are done. The correct way to find out whether __findattr__ is active or not would be to look upwards the frame chain and inspect it. Moshe also asked about continuations: I think this would resolve quite fine. However we jump around, the current chain of frames dictates the semantics of __findattr__. It even applies to Guido's tamed coroutines, given that an explicit switch were allowed in the context of __findattr__. In a sense, we get some kind of dynamic context here, since we need to do a lookup for something in the dynamic call chain. I guess this would be quite messy to implement, and inefficient. Isn't there a way to accomplish the desired effect without modifying the instance? In the context of __findattr__, *we* know that we don't want to get a recursive call. Let's assume __getattr__ and __setattr__ had yet another optional parameter: infindattr, defaulting to 0. We would than have to pass a positive value in this context, which would object.c tell to not try to invoke __findattr__ again. With explicit passing of state, no problems with threads can occour. Readability might improve as well. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
On Fri, 1 Dec 2000, Barry A. Warsaw wrote:
There's one thing that bothers me about this: what exactly is "the call stack"? Let me clarify: what happens when you have threads. Either machine-level threads and stackless threads confuse the issues here, not to talk about stackless continuations. Can you add a few words to the PEP about dealing with those?
Moshe Zadka wrote:
As far as I understood the patch (just skimmed), thee is no stack involved directly, but the instance increments and decrments a variable infindattr. + if (v != NULL && !inst->infindaddr && + (func = inst->in_class->cl_findattr)) + { + PyObject *args, *res; + args = Py_BuildValue("(OOO)", inst, name, v); + if (args == NULL) + return -1; + ++inst->infindaddr; + res = PyEval_CallObject(func, args); + --inst->infindaddr; This is: The call modifies the instance's state, while calling the findattr method. You are right: I see a serious problem with this. It doesn't even need continuations to get things messed up. Guido's proposed coroutines, together with uThread-Switching, might be able to enter the same instance twice with ease. Barry, after second thought, I feel this can become a problem in the future. This infindattr attribute only works correctly if we are guaranteed to use strict stack order of execution. What you're *intending* to to is to tell the PyEval_CallObject that it should not find the __findattr__ attribute. But this should be done only for this call and all of its descendants, but no *fresh* access from elsewhere. The hard way to get out of this would be to stop scheduling in that case. Maybe this is very cheap, but quite unelegant. We have a quite peculiar system state here: A function call acts like an escape, to make all subsequent calls behave differently, until this call is finished. Without blocking microthreads, a clean way to do this would be a search up in the frame chain, if there is a running __findattr__ method of this object. Fairly expensive. Well, the problem also exists with real threads, if they are allowed to switch in such a context. I fear it is necessary to either block this stuff until it is ready, or to maintain some thread-wise structure for the state of this object. Ok, after thinking some more, I'll start an extra message to Barry on this topic. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
"Barry A. Warsaw" wrote:
The PEP does define when and how __findattr__() is called, but makes no statement about what it should do or return... Here's a slightly different idea: Given the name, I would expect it to go look for an attribute and then return the attribute and its container (this doesn't seem to be what you have in mind here, though). An alternative approach given the semantics above would then be to first try a __getattr__() lookup and revert to __findattr__() in case this fails. I don't think there is any need to overload __setattr__() in such a way, because you cannot be sure which object actually gets the new attribute. By exposing the functionality using a new builtin, findattr(), this could be used for all the examples you give too. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M" == M <mal@lemburg.com> writes:
M> The PEP does define when and how __findattr__() is called, M> but makes no statement about what it should do or return... Good point. I've clarified that in the PEP. M> Here's a slightly different idea: M> Given the name, I would expect it to go look for an attribute M> and then return the attribute and its container (this doesn't M> seem to be what you have in mind here, though). No, because some applications won't need a wrapped object. E.g. in the Java bean example, it just returns the attribute (which is stored with a slightly different name). M> An alternative approach given the semantics above would then be M> to first try a __getattr__() lookup and revert to M> __findattr__() in case this fails. I don't think this is as useful. What would that buy you that you can't already do today? The key concept here is that you want to give the class first crack to interpose on every attribute access. You want this hook to get called before anybody else can get at, or set, your attributes. That gives you (the class) total control to implement whatever policy is useful. M> I don't think there is any need to overload __setattr__() in M> such a way, because you cannot be sure which object actually M> gets the new attribute. M> By exposing the functionality using a new builtin, findattr(), M> this could be used for all the examples you give too. No, because then people couldn't use the object in the normal dot-notational way. -Barry
"Barry A. Warsaw" wrote:
I was thinking of a standardised helper which could then be used for all kinds of attribute retrieval techniques. Acquisition would be easy to do, access control too. In most cases __findattr__ would simply return (self, self.attrname).
Forget that idea... *always* calling __findattr__ is the more useful way, just like you intended.
Right.
Uhm, why not ? -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Hi Barry, "Barry A. Warsaw" wrote:
I have been using ExtensionClass for quite a long time, and I have to say that you indeed eliminate most of its need through this super-elegant idea. Congratulations! Besides acquisition and persitency interception, wrapping plain C objects and giving them Class-like behavior while retaining fast access to internal properties but being able to override methods by Python methods was my other use of ExtensionClass. I assume this is the other "20%" part you mention, which is much harder to achieve? But that part also looks easier to implement now, by the support of the __findattr__ method.
Great. I had to read it twice, but it was fun. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
"Barry A. Warsaw" wrote:
Ok, as I announced already, here some thoughts on __findattr__, system state, and how it could work. Looking at your patch, I realize that you are blocking __findattr__ for your whole instance, until this call ends. This is not what you want to do, I guess. This has an effect of affecting the whole system state when threads are involved. Also you cannot use __findattr__ on any other attribute during this call. You want most probably do this: __findattr__ should not be invoked again for this instance, with this attribute name, for this "thread", until you are done. The correct way to find out whether __findattr__ is active or not would be to look upwards the frame chain and inspect it. Moshe also asked about continuations: I think this would resolve quite fine. However we jump around, the current chain of frames dictates the semantics of __findattr__. It even applies to Guido's tamed coroutines, given that an explicit switch were allowed in the context of __findattr__. In a sense, we get some kind of dynamic context here, since we need to do a lookup for something in the dynamic call chain. I guess this would be quite messy to implement, and inefficient. Isn't there a way to accomplish the desired effect without modifying the instance? In the context of __findattr__, *we* know that we don't want to get a recursive call. Let's assume __getattr__ and __setattr__ had yet another optional parameter: infindattr, defaulting to 0. We would than have to pass a positive value in this context, which would object.c tell to not try to invoke __findattr__ again. With explicit passing of state, no problems with threads can occour. Readability might improve as well. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
participants (4)
-
barry＠digicool.com -
Christian Tismer -
M.-A. Lemburg -
Moshe Zadka