Mailman 3 May 2004 - Python-Dev

Re: [Python-Dev] cmp(x,x)
by Gustavo Niemeyer 25 May '04

25 May '04

> [Greg Ewing] > >> What if the list contains a NaN? > > [Armin Rigo] > > Uh, right. The whole issue with NaNs is muddy. Let's focus on Numeric as > > a much saner example of why you need 'a==a' to return something else than > > True. > > Yes. Please <wink>. I belive that either my question was misinterpreted, or I'm missing something. I'm not suggesting to limit 'a==a' to return True/False only. I'm suggesting to introduce a shortcut in the *list* comparison (more specifically, in list_richcompare), since this is what is currently being done, but with a more expensive logic. Going through the function, in a case where v is w and op is either Py_EQ or Py_NE, we have 1) Both objects are lists, go on. 2) Both objects have the same size, go on. 3) Searching for the first index where items are different, using PyObject_RichCompareBool(), won't find anything. Go on. 4) No more items to compare. Check sizes. 5) Sizes match. Check comparison operator. 6) Return either Py_True or Py_False. This whole logic might be replaced by simply: if (v == w && (op == Py_EQ || op == Py_NE)) { res = (op == Py_EQ ? Py_True : Py_False); Py_INCREF(res); return res; } -- Gustavo Niemeyer http://niemeyer.net

2 4

RE: [Python-Dev] cmp(x,x)
by Michael Chermside 25 May '04

25 May '04

Raymond writes: > The code in question is in Py_RichCompareBool() which *always* just > returns True or False. That routine is called by list.__contains__() > and many other functions that expect a yes or no answer. > > The regular rich comparison function, Py_RichCompare() is the same as it > always was and can still return arrays of bools, complex numbers, or > anything at all. All right, I took a look at exactly where PyObject_RichCompareBool is called. Here is a COMPLETE list of all uses in 2.3.3 (that's what I had lying around): * In listobject.c, to test containment in a list (ie: "x in [x]"). * In tupleobject.c to perform containment tests on tuples (ie: "x in (a,b,c)"). * In arraymodule.c, in array_contains to test containment in an array. * In listobject.c, in listindex(), listremove(), and listcount() to find the first occurance, delete the first occurance, or count the number of occurances of an object in the list. * In arraymodule.c, in array_index(), array_remove(), and array_count() to find the first occurance, delete the first occurance, or count the number of occurances of an object in the array. * In abstract.c, in _PySequence_IterSearch() to test containment of, find the first occurance of, or count the occurances of some object. * In listobject.c, in list_richcompare(), to skip past identical list elements when comparing lists. * In tupleobject.c, in tuplerichcompare() to skip past identical tuple elements when comparing tuples. * In arraymodule.c, in array_richcompare, to skip past identical array elements when comparing arrays. * In listobject.c, used in sorting if the user does not provide a user-defined comparison function. * In bltinmodule.c, in min_max(), to compare two objects within the min and max functions. * In iterobject.c, to test whether an object is the sentinal object. * In dictobject.c, in lookdict(), to test whether an key in the dict matches the key being looked up. * In dictobject.c, in characterize() ... I'm not quite sure what this is doing. * In dictobject.c when comparing two dicts for equality. * In typeobject.c, in method_is_overloaded, to test whether the methods defined in a class and some subclass are actually the same object. * In ceval.c, for matching keyword arguments to code object varnames in PyEval_EvalCodeEx. * In ceval.c in _PyEval_SliceIndex() to compare a bound to the number 0L. * And finally, it is part of the published API so it could appear anywhere in extension modules... but I would guess that people use PyObject_RichCompare normally and only call PyObject_RichCompareBool if they really want a boolean. What I learned by writing this list is that Raymond is right... being able to assume that "x is x" implies "x == x" is very useful for implementors. And if that assumption is made in PyObject_RichCompareBool() and NOT in PyObject_RichCompare(), then it hardly ever needs to conflict with the desire for user-defined comparison functions to return peculiar things (like Numeric arrays, values that are artifically not equal (like NANs or a poorly-designed user-implemented Max object). Okay, I know this is too much analysis, but I've started now, so I'm going to go through with it and send off this email. Here's my thought: Raymond has convinced me that we want to make this assumption in the interpreter. I want the rules for when a user-defined comparison function (__cmp__, __eq__, or __ne__) is invoked and when it isn't. Maybe we can achieve both. Here's that list above, grouped by function: 1. Test Containment in a sequence, also index(), count(), and remove() on sequences. 2. Comparing sequences or dicts to each other. 3. Sorting lists when no user-defined comparison function is given. 4. in min() and max() 5. checking for sentinal in iteration 6. looking up dict keys 7. characterize() in dictobject (what's this for?) 8. Checking if a method is the same to tell if it's overloaded. 9. Matching keyword arguments to code object varnames in EvalCode 10. Compare a slice bound to 0L. 11. Extension modules that call PyObject_RichCompareBool instead of PyObject_RichCompare. This is the list of all places where RichCompareBool is called instead of RichCompare, and thus of all places where a user-defined comparison function might (surprisingly) not be called. Some are not relevent (eg: #9 compares only strings, a built-in type). For some, I can concoct artifical examples where someone would care (eg: #5: creating a sentinal for iter() that tries to be so VERY clever that it allows the sentinal object itself to occur in the sequence sometimes without stopping the iteration; or #3: trying to understand the behavior of timsort by creating objects that log comparisons rather than by reading the code). But these feel artificial to me. The big ones seem to be #1 and #2. I'd even say that it makes "intuitive sense" to me somehow that containment, index(), count() and remove() all act as if identity implied equality (although I can think of only *slightly* absurd examples where a user might try to alter this behavior. But when comparing sequences or dicts to each other (something people do a LOT), I would intuitively expect that any contained objects with customized comparison methods would have those methods invoked. Okay... I've talked myself into a corner now. Raymond has convinced me that my original idea was misguided, and I've looked closely at the problem, but I don't see an "obvious" solution. I'm tending to think it's best to put the test for identity in PyObject_RichCompareBool, but then how do we explain (in simple terms) when user-defined comparison methods are invoked and when they're not necessarily? Well, even without an answer, I'd better send this email off and get back to my own work. I'm not sure if I've gotten anywhere with this or not. -- Michael Chermside

1 0

Re: [Python-Dev] cmp(x,x)
by Jewett, Jim J 25 May '04

25 May '04

Gustavo Niemeyer: <niemeyer(a)conectiva.com> > Looking again, this is true for any operation, not only > Py_EQ or Py_NE. CVS has previously restricted the shortcut to those two, but I agree it should be all or nothing. Even unordered objects normally compare equal to themselves; if "is" does not imply "__eq__", then we are by definition dealing with an exceptional case. (NaN, or SQL NULL, or DUMMY, or ...) Since different exceptional cases will have different rules, it might be reasonable to ask the creator to use a custom comparison function. They will often have to do this anyway. Instead of if (a == b): they can write if (a is not NULL) and (a == b): or if (a == b) and (not isNaN(a)): -jJ

2 1

Re: [Python-checkins] python/dist/src/Doc/tut tut.tex, 1.228, 1.229
by Fred L. Drake, Jr. 25 May '04

25 May '04

On Tuesday 25 May 2004 12:08 pm, rhettinger(a)users.sourceforge.net wrote: > Modified Files: > tut.tex > Log Message: > Follow official recommendation for domain names in examples. Yay! Thanks, Raymond! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

1 0

RE: [Python-Dev] cmp(x,x)
by Michael Chermside 25 May '04

25 May '04

Armin Rigo writes: > Unless there are serious objections I suggest to (i.e. I plan to) remove the > short-cut in PyObject_RichCompareBool() -- performance is probably not an > issue here -- and then review all built-in comparison methods and make sure > that they return "equal" for identical objects. Robert H. Ledwith writes: > I have a serious objection to this because it is a major performance > problem. Performance is certainly a legitimate concern. But wouldn't your performance problems be fixed just as well if the tuple class implemented "identical-objects-are-immediately-equal" instead of putting it in the general comparison logic for all objects? Because if I understand correctly, that's what Armin is suggesting. Tuples would still be compared just as rapidly. -- Michael Chermside

8 10

RE: cmp(x,x)
by Michael Chermside 24 May '04

24 May '04

[...discussion of Armin's idea of moving the short-circuiting of equality tests comparing an object to itself from richcompare (where it affects all objects) to individual object implementations (like tuple)...] Raymond writes: > Actually, it's a ton of indirection (many steps) not just a single C > lookup. Equality testing is one of the most common and most expensive > ops in Python. Okay, I may be mistaken about the performance impact. Can anyone do actual measurements? Given that equality testing is already quite slow, I'd be surprised if there were a noticable performance difference -- but I'm often surprised by performance behavior... show me. > My main concern is that Armin is about to do backflips (inserting new > code into the details of every object's comparison code) just to > facilitate the otherwise unholy goal of having an non-reflexsive > equality comparison. I think that the REAL goal is consistancy in when user-defined equality comparison overrides (__eq__, __ne__, __cmp__) are invoked and when they're not. I find this program surprising: Python 2.3.4c1 (#52, May 12 2004, 19:37:24) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> class NoisyCompare(int): ... def __cmp__(self, other): ... print 'comparing' ... return int.__cmp__(self, other) ... >>> n = NoisyCompare(3) >>> n == n # CASE 1 comparing True >>> n < n # CASE 2 comparing False >>> cmp(n,n) # CASE 3 0 >>> [n] == [n] # CASE 4 comparing True (Actually, I find it surprising for two reasons... because case 3 does NOT print 'comparing', and because case 4 DOES (and Armin said it wouldn't). Allowing NANs to have wierd behavior is a side effect, probably less important than allowing USERS to define their own classes that misbehave. > My main concern is that Armin is about to do backflips (inserting new > code into the details of every object's comparison code) just to > facilitate the otherwise unholy goal of having an non-reflexsive > equality comparison. Now hang on... why is this in EVERY object's code? I would think that list, tuple, and dict need it because they often contain recursive structures. But for most types, why do it at all? > IMO, it is not unreasonable to insist that equality be reflexsive. That > is somewhat basic to the whole notion of equality and certainly reflects > the assumptions made throughtout the code base. No, I don't think it's unreasonable either. But is it necessary? We should try to have a single, simple rule to explain when user-defined operator-overrides of comparison functions are invoked. If the rule is that they are ALWAYS invoked, and that the short-circuiting of identity comparisons (which is certainly necessary for things like tuple and list) is simply part of the implementation of that type, then the rule is VERY simple. If the rule is that the user-defined function is always invoked on differing objects and MAY be invoked if an object is compared to itself, then that's a lot less clean. Anything else I can think of is probably too complex to consider. By the way, Armin introduced this by saying: > A minor semantic change that creeped in some time ago was an implicit > assumption that any object x should "reasonably" be expected to compare > equal to itself. How long ago was that? -- Michael Chermside

1 0

RE: [Python-Dev] factory functions in the datetime module (& others)
by Michael Chermside 24 May '04

24 May '04

Brian Quinlan writes: > It seems to me that the utility of subclassing certain > library classes would > be increased if the user could set a factory that the base > class would use > when creating new instances. [...] > Any thoughts? Hmm... I guess I'm -0 on the idea... it wouldn't really bother me, but I think it's a feature we might as well leave out. I guess my philosophy on this comes down to a fundamental approach to subclassing basic built-in types. Types like int, float, string, datetime, list and tuple are simple, fundamental types built into Python (or its standard libraries). The usual way to use these is by composition -- create a new type that contains fields which are ints, floats, etc. And that is enough to solve nearly all problems -- in fact, up until Python 2.1, that was pretty much the only choice. But Python wants to go the extra mile to give the programmer flexibility, so we've allowed even these basic types to be subclassed by the developer. The question is just how far we should go in supporting (and encouraging) this use. I feel that 99% of all uses of these basic types should be by composition, and only 1% (or less) by inheritance. So while I am in favor of *permitting* the subclassing, I'd be wary of tweaking other parts of the language to support it. I agree that providing customizable factory functions for these would be very helpful to users of fundamental type subclasses, but I think they are (and ought to be) a fairly small minority. As for HOW to do it... a global factory seems like a recipe for encouraging different packages to conflict with each other -- it only works so long as only ONE module tries to install a subtype (and perhaps not even then). A per-object approach is questionable in terms of performance... and it is not acceptable (to me) to slow the performance of code which uses fundamental types by composition (essentially all Python code) for a feature like this. Just my thoughts on the matter... I certainly agree that subclassing fundamental types is less powerful because there is no way to register some sort of factory function, but I'm not sure it's an important-enough feature to be worth the cost of supporting registrable factory functions. -- Michael Chermside

1 0

RE: [Python-Dev] cmp(x,x)
by Michael Chermside 24 May '04

24 May '04

I wrote: > Performance is certainly a legitimate concern. Armin replied: > Ok, then the current situation looks like the best one. I > suggest we just > drop a note in the documentation saying that if there are > objects for which > x.__eq__(x) isn't necessarily true, you shouldn't expect them > to be handled in > a fully consistent way. [...] > There is an extra costly indirection until the execution > reaches that point. Armin: Performance is a *legitimate* concern, but not an *overwhelming* one. If we are in danger of having to walk over the entirety of a (possibly large) recusive structure (eg: tuple of tuples) to do a comparison, then that's dangerously slow, and it's worth mucking with the definition of equality to avoid it. But a single indirection is "costly" only to someone capable of reaching your level of close-to-the-machine performance understanding. I say, go with your original plan -- the current behavior is a mess. -- Michael Chermside

1 0

SystemError: com_backpatch: offset too large
by Sebastian 21 May '04

21 May '04

I've got a problem with python script. My script return exceptinon: SystemError: com_backpatch: offset too large I'm beginer in python programming, and don't known what do with this message.

2 1

Re: cmp(x,x)
by Casey Duncan 21 May '04

21 May '04

On Fri, 21 May 2004 15:21:09 -0400 "Tim Peters" <tim.one(a)comcast.net> wrote: > [Armin Rigo] [..] > NaN comparison is a red herring not only because Python doesn't have a > sane story in any area involving NaNs, but also because the 32 > possible 754 comparison predicates can't be mapped onto 6 operator > symbols. Note that in C99, the infix C relational operators raise an > exception (754's "invalid operation") when applied to a NaN, and C99 > adds a pile of new macros for non-exceptional NaN comparison (my > favorite is the fetchingly named"islessgreater(x, y)" -- which "is > similar to (x) < (y) || (x) > (y); however, islessgreater(x,y) does > not raise the invalid exception when x and y are unordered (nor does > it evaluate x and y twice)"). Finally, a use for <>. Barry will be so happy ;^) -Casey

3 3

Python-Dev May 2004