Summary: Chistian is right after all. instancemethod_getattro should always prefer bound method attributes over function attributes.
No no no, I'm not fiddling around with any internals, here. I just want to use the machinary as it is, and to be able to pickle almost everything.
Sorry, this was a lie.
Sigh. OK, you're forgiven.
Sure I'm fiddling internaly, but simply by installing some __reduce__ methids, hoping that they work.
OK, so you *could* just make the change you want, but you are asking why it isn't like that in the first place. Good idea...
This worked most of the time, but I'm having problems with bound methods.
We've established that without a doubt, yes. :-)
But how, I wonder, are you providing it? You can't subclass instancemethod -- how do you manage to add a __reduce__ method to it without fiddling with any internals?
I added __reduce__ to the PyMethod type and tried to figure out why it didn't take it.
OK. Stating that upfront would have helped...
My expectation is that C().x.__reduce__ gives me the bound __reduce__ method of the bound x method of a C instance.
Yes, unfortunately you get the __reduce__ method of the unbound function instead.
I think Martin is right: copy_reg may be your last hope. (Or subclassing pickle to special-case instancemethod.)
Well, I see your point, but please let me explain mine, again: If there is a class C which has a method x, then C().x is a perfectly fine expression, yielding a bound method.
Of course.
If I now like to pickle this expression, I would use the __reduce__ protocol and ask C().x for its __reduce__ property.
Which unfortunately gets the __reduce__ property of the underlying *function* object (also named x) used to implement the method. This function can be accessed as C.__dict__['x']. (Not as C.x, that returns an unbound method object, which is the same kind of object as a bound method object but without an instance. :-)
Now, please see that __reduce__ has no parameters, i.e. it has no other chance to do the right thing(TM) but by relying on to be bound to the right thing. So, doesn't it make sense to have __reduce__ to be always returned as a method of some bound anything?
In other words, shouldn't things that are only useful as bound things, always be bound?
This question doesn't address the real issue, which is the attribute delegation to the underlying function object. What *should* happen when the same attribute name exists on the function and on the bound method? In 2.1, when function attributes were first introduced, this was easy: a few attributes were special for the bound method (im_func, im_self, im_class) and for these the bound method attribute wins (if you set an attribute with one of those names on the function, you can't access it through the bound method). The *intention* was for the 2.2 version to have the same behavior: only im_func, im_self and im_class would be handled by the bound method, other attributes would be handled by the function object. This is what the IsData test is attempting to do -- the im_* attributes are represented by data descriptors now. The __class__ attribute is also a data descriptor, so that C().x.__class__ gives us <type 'instancemethod'> rather than <type 'function'>. But for anything else, including the various methods that all objects inherit from 'object' unless they override them, the choice was made to let the function attribute win. But when we look at the attributes where both function and bound method provide a value, it seems that the bound method's offering is always more useful! You've already established this for __reduce__; the same is true for __call__ and __str__, and there I stopped. (Actually, I also looked at __setattr__, where delegation to the function also seems a mistake: C().x.foo = 42 is refused, but C().x.__setattr__('foo', 42) sets the attribute on the function, because this returns the (bound) method __setattr__ on functions.)
The pickling machinery wasn't intended to pickle bound methods or functions etc., and doesn't particularly go out of its way to allow you to add that functionality.
The pickling machinery gives me an __reduce__ interface, and I'm expecting that this is able to pickle everything.
I don't think you'd have a chance of pickle classes if you only relied on __reduce__. Fortunately there are other mechanisms. :-) (I wonder if the pickling code shouldn't try to call x.__class__.__reduce__(x) rather than x.__reduce__() -- then none of these problems would have occurred... :-)
...
And it would be except for the delegation of method attributes to function attributes. It is a similar aliasing problem as you see when you try to access the __getattr__ implementation for classes as C.__getattr__ -- you get the __getattr__ for C instances instead. So you have to use type(C).__getattr__ instead. That would work for __reduce__ too I think: new.instancemethod.__reduce__(C().f).
I agree! But I can't do this in this context, using __reduce__ only. In other words, I'd have to add stuff to copyreg.py, which I tried to circumvent.
Or you could change the pickling system. Your choice of what to change and what not to change seems a bit arbitrary. :-)
...
OK, so you *are* messing with internals after all (== changing C code), right? Or else how do you accomplish this?
Yessir, I'm augmenting all things-to-be-pickled with __reduce__ methods. And this time is the first time that it doesn't work.
But not necessarily the last time. :-)
...
I have a real job too, that's why I have little time to help you. :-(
I agree (and I didn't ask *you* in the first place), but still I'd like to ask the general question: Is this really the right way to handle bound objects? Is the is_data criterion correct? If I am asking for an attribute that makes *only* sense if it is bound, like in the parameter-less __reduce__ case, wouldn't it be the correct behavior to give me that bound object?
I have the strong impression that there is some difference in methods which isn't dealt with, correctly, at the moment. If a method wants to be bound to something, it should be get bound to something. Especially, if this method is useless without being bound.
It's not that it isn't being bound. It's that the *wrong* attribute is being bound (the function's __reduce__ method, bound to the function object, is returned!).
Please, swallow this idea a little bit, before rejecting it. I think that "is_data" is too rough and doesn't fit the requirements, all the time.
I agree. The bound method's attributes should always win, since bound methods only have a small, fixed number of attributes, and they are all special for bound methods. This *is* a change in functionality, even though there appear to be no unit tests for it, so I'm reluctant to fix it in 2.3. But I think in 2.4 it should definitely change. --Guido van Rossum (home page: http://www.python.org/~guido/)