how to test behavior wrt an extension type?
copy.py, as recently discussed starting from a post by /F, has two kinds of misbehaviors since 2.3 (possibly 2.2, haven't checked), both connected to instance/type/metatype confusion (where do special methods come from? in classic classes and types, from the instance, which may delegate to the type/class; in newstype one, from the class/type which _must not_ delegate to the metaclass): type/metatype confusion, and misbehavior with instances of extension types. So, as per discussion here, I have prepared a patch (to the maintenance branch of 2.3, to start with) which adds unit tests to highlight these issues, and fixes them in copy.py. This patch should go in the maintenance of 2.3 and 2.4, but in 2.5 a different approach based on new special descriptors for special methods is envisaged (though keeping compatibility with classic extension types may also require some patching to copy.py along the lines of my patch). Problem: to write unit tests showing that the current copy.py misbehaves with a classic extension type, I need a classic extension type which defines __copy__ and __deepcopy__ just like /F's cElementTree does. So, I made one: a small trycopy.c and accompanying setup.py whose only purpose in life is checking that instances of a classic type get copied correctly, both shallowly and deeply. But now -- where do I commit this extension type, so that the unit tests in test_copy.py can do their job...? Right now I've finessed the issue by having a try/except ImportError in the two relevant unit tests (for copy and deepcopy) -- if the trycopy module is not available I just don't test how its instances behave under deep or shallow copying. However, if I just commit or send the patch as-is, without putting trycopy.c and its setup.py somewhere, then I'm basically doing a fix without unit tests to back it up, from the point of view of anybody but myself. I do not know what the recommended practice is for this kind of issues, so, I'm asking for guidance (and specifically asking Anthony since my case deals with 2.3 and 2.4 maintenance and he's release manager for both, but, of course, everybody's welcome to help!). Surely this can't be the first case in which a bug got triggered only by a certain behavior in an extension type, but I couldn't find precedents. Ideas, suggestions, ...? Alex
So, as per discussion here, I have prepared a patch (to the
[Alex] maintenance
branch of 2.3, to start with) which adds unit tests to highlight these issues, and fixes them in copy.py. This patch should go in the maintenance of 2.3 and 2.4, but in 2.5 a different approach based on new special descriptors for special methods is envisaged (though keeping compatibility with classic extension types may also require some patching to copy.py along the lines of my patch).
For Py2.5, do you have in mind changing something other than copy.py? If so, please outline your plan. I hope your not planning on wrapping all special method access as descriptor look-ups -- that would be a somewhat radical change. Raymond
On 2005 Jan 16, at 11:17, Raymond Hettinger wrote:
So, as per discussion here, I have prepared a patch (to the
[Alex] maintenance
branch of 2.3, to start with) which adds unit tests to highlight these issues, and fixes them in copy.py. This patch should go in the maintenance of 2.3 and 2.4, but in 2.5 a different approach based on new special descriptors for special methods is envisaged (though keeping compatibility with classic extension types may also require some patching to copy.py along the lines of my patch).
For Py2.5, do you have in mind changing something other than copy.py? If so, please outline your plan. I hope your not planning on wrapping all special method access as descriptor look-ups -- that would be a somewhat radical change.
The overall plan does appear to be exactly the "somewhat radical change" which you hope is not being proposed, except it's not my plan -- it's Guido's. Quoting his first relevant post on the subject: ''' From: gvanrossum@gmail.com Subject: Re: getting special from type, not instance (was Re: [Python-Dev] copy confusion) Date: 2005 January 12 18:59:13 CET ... I wonder if the following solution wouldn't be more useful (since less code will have to be changed). The descriptor for __getattr__ and other special attributes could claim to be a "data descriptor" which means that it gets first pick *even if there's also a matching entry in the instance __dict__*. Quick illustrative example:
class C(object): foo = property(lambda self: 42) # a property is always a "data descriptor"
a = C() a.foo 42 a.__dict__["foo"] = "hello" a.foo 42
Normal methods are not data descriptors, so they can be overridden by something in __dict__; but it makes some sense that for methods implementing special operations like __getitem__ or __copy__, where the instance __dict__ is already skipped when the operation is invoked using its special syntax, it should also be skipped by explicit attribute access (whether getattr(x, "__getitem__") or x.__getitem__ -- these are entirely equivalent). We would need to introduce a new decorator so that classes overriding these methods can also make those methods "data descriptors", and so that users can define their own methods with this special behavior (this would be needed for __copy__, probably). I don't think this will cause any backwards compatibility problems -- since putting a __getitem__ in an instance __dict__ doesn't override the x[y] syntax, it's unlikely that anybody would be using this. "Ordinary" methods will still be overridable. PS. The term "data descriptor" now feels odd, perhaps we can say "hard descriptors" instead. Hard descriptors have a __set__ method in addition to a __get__ method (though the __set__ method may always raise an exception, to implement a read-only attribute). ''' All following discussion was, I believe, in the same thread, mostly among Guido, Phillip and Armin. I'm focusing on getting copy.py fixed in 2.3 and 2.4, w/o any plan yet to implement Guido's idea. If you dislike Guido's idea (which Phillip, Armin and I all liked, in different degrees), it might be best for you to read that other thread and explain the issues there, I think. Alex
Alex Martelli wrote:
Problem: to write unit tests showing that the current copy.py misbehaves with a classic extension type, I need a classic extension type which defines __copy__ and __deepcopy__ just like /F's cElementTree does. So, I made one: a small trycopy.c and accompanying setup.py whose only purpose in life is checking that instances of a classic type get copied correctly, both shallowly and deeply. But now -- where do I commit this extension type, so that the unit tests in test_copy.py can do their job...?
Modules/_testcapimodule.c ? (I'm using the C api to define an extension type, after all...) </F>
On 2005 Jan 16, at 12:03, Fredrik Lundh wrote:
Alex Martelli wrote:
Problem: to write unit tests showing that the current copy.py misbehaves with a classic extension type, I need a classic extension type which defines __copy__ and __deepcopy__ just like /F's cElementTree does. So, I made one: a small trycopy.c and accompanying setup.py whose only purpose in life is checking that instances of a classic type get copied correctly, both shallowly and deeply. But now -- where do I commit this extension type, so that the unit tests in test_copy.py can do their job...?
Modules/_testcapimodule.c ?
(I'm using the C api to define an extension type, after all...)
Fine with me, if there are no objections I'll alter the patch accordingly and submit it that way. Thanks, Alex
On Sunday 16 January 2005 20:38, Alex Martelli wrote:
Problem: to write unit tests showing that the current copy.py misbehaves with a classic extension type, I need a classic extension type which defines __copy__ and __deepcopy__ just like /F's cElementTree does. So, I made one: a small trycopy.c and accompanying setup.py whose only purpose in life is checking that instances of a classic type get copied correctly, both shallowly and deeply. But now -- where do I commit this extension type, so that the unit tests in test_copy.py can do their job...?
I do not know what the recommended practice is for this kind of issues, so, I'm asking for guidance (and specifically asking Anthony since my case deals with 2.3 and 2.4 maintenance and he's release manager for both, but, of course, everybody's welcome to help!). Surely this can't be the first case in which a bug got triggered only by a certain behavior in an extension type, but I couldn't find precedents. Ideas, suggestions, ...?
Beats me - worst comes to worst, I guess we ship the unittest code
there with a try/except around the ImportError on the new 'copytest'
module, and the test skips if it's not built. Then we don't build it by
default, but if someone wants to build it and check it, they can. I don't
like this much, but I can't think of a better alternative. Shipping a new
extension module just for this unittest seems like a bad idea.
Anthony
--
Anthony Baxter
On 2005 Jan 17, at 14:45, Anthony Baxter wrote: ...
both, but, of course, everybody's welcome to help!). Surely this can't be the first case in which a bug got triggered only by a certain behavior in an extension type, but I couldn't find precedents. Ideas, suggestions, ...?
Beats me - worst comes to worst, I guess we ship the unittest code there with a try/except around the ImportError on the new 'copytest' module, and the test skips if it's not built. Then we don't build it by default, but if someone wants to build it and check it, they can. I don't like this much, but I can't think of a better alternative. Shipping a new extension module just for this unittest seems like a bad idea.
Agreed about this issue not warranting the shipping of a new extension module -- however, in the patch (to the 2.3 maintenance branch) which I uploaded (and assigned to you), I followed the effbot's suggestion, and added the type needed for testing to the already existing "extension module for testing purposes", namely Modules/_testcapi.c -- I don't think it can do any harm there, and lets test/test_copy.py do all of its testing blissfully well. I haven't even made the compilation of the part of Modules/_testcapi.c which hold the new type conditional upon anything, because I don't think that having it there unconditionally can possibly break anything anyway... _testcapi IS only used for testing, after all...! Alex
participants (4)
-
Alex Martelli
-
Anthony Baxter
-
Fredrik Lundh
-
Raymond Hettinger