various unix platform build/test issues
It ain't real pretty, but it could be worse: http://metaslash.com/py/2.3-problems.html I'm trying to commit changes to the logging package, but SF has a stale CVS lock for anoncvs_python. I filed a support request. Hopefully they will have it fixed within a month or two, at least by the time 2.3 is ready to be released. I don't know if the updates are supposed to fix the test hangs/failures for test_logging. Here's the problem highlights (there are more, follow the link): * There are 3 64-bit issues: http://python.org/sf/688424 * test_pty hangs on 3 platforms, possible fix: http://python.org/sf/671384 * test_resource can fail when filesize is limited http://python.org/sf/678264 * iconv module calls PyFatalError() if the C module initialization fails, is there a better way to handle this? I had to disable building iconv on AIX. * AIX sorts numbers and strings in a different order than Linux: >>> 'ick' >= 0 # Linux True >>> 'ick' >= 0 # AIX False This causes failures for test_descrtut and test_random. I don't expect to get these taken care of for 2.3a2 assuming it's released tomorrow. It would be nice to have most of the platforms clean for the beta. Neal
[Neal Norwitz]
* AIX sorts numbers and strings in a different order than Linux:
>>> 'ick' >= 0 # Linux True
>>> 'ick' >= 0 # AIX False
This causes failures for test_descrtut and test_random.
It shouldn't. Regardless of platform, this should end up at the tail end of default_3way_compare(), resolved by the c = strcmp(vname, wname); line with vname "str" and wname "" (an empty string). But-- ick! --it doesn't, and not on any platform anymore. vname and wname are both empty strings now, so it goes on to compare the objects by memory address and the outcome is pure accident ... this is because rev 2.197 of stringobject.c gave string objects a non-NULL tp_as_number slot for the first time. The intent of the emtpy-string gimmick was to make all numeric objects "less than" all non-numeric objects (excepting None, which is less than every non-None value), in particular so that sorting a list would bring all the number-like thingies next to each other. Strings weren't intended to take part in this number gimmick too. Then again, comparison is so snaky it's hard to predict what it will do. In any case, this is incompatible with previous Python releases. Neil(S), was giving strings a tp_as_number slot necessary? Or just the least painful way to fix whatever bug it was fixing? I suppose the tail end of default_3way_compare could grow more special cases to exempt strings from being treated like numbers for the purpose of comparison.
[Neal Norwitz]
* AIX sorts numbers and strings in a different order than Linux:
>>> 'ick' >= 0 # Linux True
>>> 'ick' >= 0 # AIX False
This causes failures for test_descrtut and test_random.
It shouldn't. Regardless of platform, this should end up at the tail end of default_3way_compare(), resolved by the
c = strcmp(vname, wname);
line with vname "str" and wname "" (an empty string).
But-- ick! --it doesn't, and not on any platform anymore. vname and wname are both empty strings now, so it goes on to compare the objects by memory address and the outcome is pure accident ... this is because rev 2.197 of stringobject.c gave string objects a non-NULL tp_as_number slot for the first time. The intent of the emtpy-string gimmick was to make all numeric objects "less than" all non-numeric objects (excepting None, which is less than every non-None value), in particular so that sorting a list would bring all the number-like thingies next to each other. Strings weren't intended to take part in this number gimmick too. Then again, comparison is so snaky it's hard to predict what it will do. In any case, this is incompatible with previous Python releases. Neil(S), was giving strings a tp_as_number slot necessary? Or just the least painful way to fix whatever bug it was fixing? I suppose the tail end of default_3way_compare could grow more special cases to exempt strings from being treated like numbers for the purpose of comparison.
I forget what Neil's motivation was; SF bug #615506 doesn't really mention much for motivation. I suspect it's simply cleanliness. Maybe PyNumber_Check() could be changed? This has been proposed before. It could check whether the type implements conversion to int, for example. Or it could check whether the type derives from a new-to-introduce abstract base class ("basenumber"). --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido, on string-vs-number comparisons falling back to comparison by memory address now, instead of previous number < string behavior]
I forget what Neil's motivation was; SF bug #615506 doesn't really mention much for motivation. I suspect it's simply cleanliness.
Maybe PyNumber_Check() could be changed? This has been proposed before. It could check whether the type implements conversion to int, for example. Or it could check whether the type derives from a new-to-introduce abstract base class ("basenumber").
The code in question doesn't invoke PyNumber_Check(), so changing that wouldn't change behavior here. The code in question could be changed to start invoking PyNumber_Check, though. Looks like, in the core, PyNumber_Check is used only by select_select and poll_pool. Try passing a string as the timeout arg to a select() call! Looks to me like it will end up segfaulting here in PyFloat_AsDouble: fo = (PyFloatObject*) (*nb->nb_float) (op); So the only PyNumber_Check() uses in the core appear to believe that nb->nb_float isn't NULL if the test passes (but PyNumber_Check isn't actually testing for that, and nb_float isn't always NULL when the test passes). NEWS sez: """ - PyNumber_Check() now returns true for string and unicode objects. This is a result of these types having a partially defined tp_as_number slot. (This is not a feature, but an indication that PyNumber_Check() is not very useful to determine numeric behavior. It may be deprecated.) """
[Guido, on string-vs-number comparisons falling back to comparison by memory address now, instead of previous number < string behavior]
I forget what Neil's motivation was; SF bug #615506 doesn't really mention much for motivation. I suspect it's simply cleanliness.
Maybe PyNumber_Check() could be changed? This has been proposed before. It could check whether the type implements conversion to int, for example. Or it could check whether the type derives from a new-to-introduce abstract base class ("basenumber").
[Tim]
The code in question doesn't invoke PyNumber_Check(), so changing that wouldn't change behavior here. The code in question could be changed to start invoking PyNumber_Check, though.
Good idea.
Looks like, in the core, PyNumber_Check is used only by select_select and poll_pool. Try passing a string as the timeout arg to a select() call! Looks to me like it will end up segfaulting here in PyFloat_AsDouble:
fo = (PyFloatObject*) (*nb->nb_float) (op);
No, that code is guarded by a test whether tp_as_number == NULL or nb_float is NULL; it issues the expected message "a float is required". [Neal, later]
The tp_as_number check in default_3way_compare pretty bogus though, IMHO. With the new type changes a lot of types have a tp_as_number slot. I think the compare code needs to change.
Yes, but how? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Tim]
fo = (PyFloatObject*) (*nb->nb_float) (op);
[Guido]
No, that code is guarded by a test whether tp_as_number == NULL or nb_float is NULL; it issues the expected message "a float is required".
Oops! You're right, of course. [Neal, later]
The tp_as_number check in default_3way_compare pretty bogus though, IMHO. With the new type changes a lot of types have a tp_as_number slot. I think the compare code needs to change.
[Guido]
Yes, but how?
It's a good question. All builtin numeric types have a non-NULL nb_float slot, but some non-numeric types do too (proxies, and even _winreg.c's PyHKEY_Type).
It's a good question. All builtin numeric types have a non-NULL nb_float slot, but some non-numeric types do too (proxies, and even _winreg.c's PyHKEY_Type).
Well, proxies deserve to be considered numeric. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
[Neal, later]
The tp_as_number check in default_3way_compare pretty bogus though, IMHO. With the new type changes a lot of types have a tp_as_number slot. I think the compare code needs to change.
Yes, but how?
Maybe PyNumber_Check should check for (tp_as_number and (nb_int or nb_float)) and then default_3way_compare should use it instead of checking for tp_as_number. Neil
Maybe PyNumber_Check should check for (tp_as_number and (nb_int or nb_float)) and then default_3way_compare should use it instead of checking for tp_as_number.
Good idea. I'll check that in, and use PyNumber_Check() in default_3way_compare() as in my previous post. --Guido van Rossum (home page: http://www.python.org/~guido/)
Tim Peters wrote:
The code in question doesn't invoke PyNumber_Check(), so changing that wouldn't change behavior here. The code in question could be changed to start invoking PyNumber_Check, though.
PyNumber_Check is pretty useless. Don't blame me, it was useless before I got to it. :-)
Looks like, in the core, PyNumber_Check is used only by select_select and poll_pool. Try passing a string as the timeout arg to a select() call! Looks to me like it will end up segfaulting here in PyFloat_AsDouble:
fo = (PyFloatObject*) (*nb->nb_float) (op);
You are thinking of PyFloat_AS_DOUBLE, I think. Python 2.3a1 (#6, Feb 4 2003, 15:46:59) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import select >>> select.select([], [], [], "ha ha") Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: a float is required It's unclear what PyNumber_Check should be replaced with. Perhaps a test that determined if an object could be turned into a float or an int. Something like: PyNumber_AsIntegerCheck(o) { if PyFloat_Check(o) return false; /* truncating a float is not acceptable */ else return true of nb_int exists } PyNumber_AsFloatCheck(o) { return true if nb_float exists } PyNumber_Check could perhaps be changed to look for nb_float or nb_int. I'm not sure if that breaks more or less extension modules. :-( Neil
Tim Peters wrote:
Neil(S), was giving strings a tp_as_number slot necessary? Or just the least painful way to fix whatever bug it was fixing? I suppose the tail end of default_3way_compare could grow more special cases to exempt strings from being treated like numbers for the purpose of comparison.
It was not necessary but I thought it was cleaner. Here is the important part of the patch: diff -u -r2.106 abstract.c --- Objects/abstract.c 5 Nov 2002 18:05:49 -0000 2.106 +++ Objects/abstract.c 17 Nov 2002 18:28:22 -0000 @@ -639,12 +639,6 @@ PyObject * PyNumber_Remainder(PyObject *v, PyObject *w) { - if (PyString_Check(v)) - return PyString_Format(v, w); -#ifdef Py_USING_UNICODE - else if (PyUnicode_Check(v)) - return PyUnicode_Format(v, w); -#endif return binary_op(v, w, NB_SLOT(nb_remainder), "%"); } The problem with special casing "unicode" and "str" is that subclasses could not override __mod__. I think it would also work to remove the tp_as_number slot and then do the PyUnicode_Check and PyString_Check after trying binary_op(). The tp_as_number check in default_3way_compare pretty bogus though, IMHO. With the new type changes a lot of types have a tp_as_number slot. I think the compare code needs to change. Neil
It ain't real pretty, but it could be worse:
Thanks!
I'm trying to commit changes to the logging package, but SF has a stale CVS lock for anoncvs_python. I filed a support request. Hopefully they will have it fixed within a month or two, at least by the time 2.3 is ready to be released. I don't know if the updates are supposed to fix the test hangs/failures for test_logging.
I this done now? (I saw a bunch of checkins to logging stuff.)
Here's the problem highlights (there are more, follow the link):
* There are 3 64-bit issues: http://python.org/sf/688424
You fixed these now. (I'm not real excited about the following to subjects, do what you see fit.)
* test_pty hangs on 3 platforms, possible fix: http://python.org/sf/671384
(Sounds find to me.)
* test_resource can fail when filesize is limited http://python.org/sf/678264
(Ditto.)
* iconv module calls PyFatalError() if the C module initialization fails, is there a better way to handle this? I had to disable building iconv on AIX.
Yes, you can raise an exception and return. This is a bit weird (there's no return value) but AFAIK it is documented that a module init function can raise an exception and return to indicate failure.
* AIX sorts numbers and strings in a different order than Linux:
>>> 'ick' >= 0 # Linux True
>>> 'ick' >= 0 # AIX False
This causes failures for test_descrtut and test_random.
I have a tentative fix for this (use PyNumber_Check() in default_3way_compare() and augment PyNumber_Check() to test whether the type defines nb_int). Can you check this on AIX? [*]
I don't expect to get these taken care of for 2.3a2 assuming it's released tomorrow. It would be nice to have most of the platforms clean for the beta.
Yes. [*] Patch: Index: Objects/object.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/object.c,v retrieving revision 2.197 diff -c -r2.197 object.c *** Objects/object.c 5 Feb 2003 19:35:19 -0000 2.197 --- Objects/object.c 18 Feb 2003 16:15:53 -0000 *************** *** 633,640 **** if (w == Py_None) return 1; ! /* different type: compare type names */ ! if (v->ob_type->tp_as_number) vname = ""; else vname = v->ob_type->tp_name; --- 633,640 ---- if (w == Py_None) return 1; ! /* different type: compare type names; numbers are smaller */ ! if (PyNumber_Check(v)) vname = ""; else vname = v->ob_type->tp_name; Index: Objects/abstract.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/abstract.c,v retrieving revision 2.117 diff -c -r2.117 abstract.c *** Objects/abstract.c 12 Feb 2003 03:36:05 -0000 2.117 --- Objects/abstract.c 18 Feb 2003 16:15:54 -0000 *************** *** 308,314 **** int PyNumber_Check(PyObject *o) { ! return o && o->ob_type->tp_as_number; } /* Binary operators */ --- 308,315 ---- int PyNumber_Check(PyObject *o) { ! return o && o->ob_type->tp_as_number && ! o->ob_type->tp_as_number->nb_int; } /* Binary operators */ --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
I have a tentative fix for this (use PyNumber_Check() in default_3way_compare() and augment PyNumber_Check() to test whether the type defines nb_int).
Sounds good to me. It's interesting then that complex numbers will pass the PyNumber_Check test because they set nb_int to a routine that unconditionally raises an exception. Checking for nb_negative may be crisper?
Tim Peters wrote:
[Guido]
I have a tentative fix for this (use PyNumber_Check() in default_3way_compare() and augment PyNumber_Check() to test whether the type defines nb_int).
Sounds good to me.
+1
It's interesting then that complex numbers will pass the PyNumber_Check test because they set nb_int to a routine that unconditionally raises an exception. Checking for nb_negative may be crisper?
Isn't that caveat in the complex implementation ? Converting a complex with 0 img part would not cause any loss of information (apart from the usual integer truncations ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 19 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 41 days left EuroPython 2003, Charleroi, Belgium: 125 days left
[M.-A. Lemburg]
Isn't that caveat in the complex implementation ? Converting a complex with 0 img part would not cause any loss of information (apart from the usual integer truncations ;-)
Hmm. Have you ever met a coercion you didn't like <0.9 wink>? float(complex) also raises an exception unconditionally, and I think for good reasons -- what someone *intends* by trying to convert a complex number to a float or an int is a mystery. The exceptions raised suggest one plausible intent and how to get at it clearly:
float(1+0j) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: can't convert complex to float; use e.g. abs(z)
Tim Peters wrote:
[M.-A. Lemburg]
Isn't that caveat in the complex implementation ? Converting a complex with 0 img part would not cause any loss of information (apart from the usual integer truncations ;-)
Hmm. Have you ever met a coercion you didn't like <0.9 wink>?
I'd even coerce strings to floats if possible ;-) BTW, I implemented that in mxNumber for the fun of it:
from mx.Number import * Float(1.23) + '3.45' 4.67999999999999998223e+0 Rational(2,3) + '3/4' 17/12
Seriously, the above was only meant as hint to not rely on nb_negative for PyNumber_Check(), but to use nb_int for this. After all, code using PyNumber_Check() will expect that at least PyNumber_Int() to work on the object. If you check for nb_negative, this will not always work, since then complex objects would get accepted.
float(complex) also raises an exception unconditionally, and I think for good reasons -- what someone *intends* by trying to convert a complex number to a float or an int is a mystery. The exceptions raised suggest one plausible intent and how to get at it clearly:
float(1+0j)
Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: can't convert complex to float; use e.g. abs(z)
I know :-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 20 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 40 days left EuroPython 2003, Charleroi, Belgium: 124 days left
[M.-A. Lemburg]
I'd even coerce strings to floats if possible ;-)
BTW, I implemented that in mxNumber for the fun of it:
from mx.Number import * Float(1.23) + '3.45' 4.67999999999999998223e+0 Rational(2,3) + '3/4' 17/12
FixedPoint does a similar thing, coercing strings to FixedPoints by magic. The lack of a native literal notation for FixedPoints (and rationals) makes that more attractive than it should be <wink>.
Seriously, the above was only meant as hint to not rely on nb_negative for PyNumber_Check(), but to use nb_int for this. After all, code using PyNumber_Check() will expect that at least PyNumber_Int() to work on the object. If you check for nb_negative, this will not always work, since then complex objects would get accepted.
Reality check: PyNumber_Check() checks for nb_int or nb_float now, and complex numbers have both slots, but both raise TypeError unconditionally. So if you believe that PyNumber_Check()'s caller expects PyNumber_Int() to work, then PyNumber_Check() is already broken in your view. My view is that a complex is a number whether or not it can be coerced to some other type, but that complex is passing PyNumber_Check() more by accident than by design now. nb_negative seems a sharper check to me.
From: "Tim Peters" <tim.one@comcast.net>
Hmm. Have you ever met a coercion you didn't like <0.9 wink>?
After Neal's changes to PyArg_ParseTuple, will Tim still be able to use a floating argument to the wink() function which has only been documented to accept a boolean state? Alternatively, will the wink() implementation accept a FixedPoint object so that ten <0.1 winks> will reliably sum exactly to a full wink? On comp.lang.py, the currently leading proposal is to tighten the semantics to boolean but to support backward compatibility for partial winks through an explictly named function: blink(). Raymond Hettinger _ ~ @ @ \_/
On Tue, Feb 18, 2003 at 11:17:04AM -0500, Guido van Rossum wrote:
I'm trying to commit changes to the logging package, but SF has a stale CVS lock for anoncvs_python. I filed a support request. Hopefully they will have it fixed within a month or two, at least by the time 2.3 is ready to be released. I don't know if the updates are supposed to fix the test hangs/failures for test_logging.
I this done now? (I saw a bunch of checkins to logging stuff.)
Yes, SF cleared the lock. The tests still fail on some machines.
(I'm not real excited about the following to subjects, do what you see fit.)
These are low priority. I'll wait for someone to review.
* AIX sorts numbers and strings in a different order than Linux:
>>> 'ick' >= 0 # Linux True
>>> 'ick' >= 0 # AIX False
This causes failures for test_descrtut and test_random.
I have a tentative fix for this (use PyNumber_Check() in default_3way_compare() and augment PyNumber_Check() to test whether the type defines nb_int). Can you check this on AIX? [*]
The patch fixed test_random, but not test_descrtut. I didn't run the full test suite. Neal
[Neal Norwitz]
The patch fixed test_random, but not test_descrtut. I didn't run the full test suite.
How about showing us the failing output from test_descrtut? Offhand I don't see anything that would be affected apart from the last line of this doctest: >>> def sorted(seq): ... seq.sort() ... return seq >>> print sorted(a.keys()) [1, 2] >>> exec "x = 3; print x" in a 3 >>> print sorted(a.keys()) [1, 2, '__builtins__', 'x']
I noticed that one of the doctests in test_descrtut displayed a dictionary without sorting it first. I just checked in a repair for that.
On Tue, Feb 18, 2003 at 11:55:38AM -0500, Tim Peters wrote:
I noticed that one of the doctests in test_descrtut displayed a dictionary without sorting it first. I just checked in a repair for that.
Looks good: test_descr test_descrtut test_random All 3 tests OK. I'll run all the tests now on various platform. But things look pretty good on this side. Neal
participants (7)
-
Guido van Rossum -
M.-A. Lemburg -
Neal Norwitz -
Neil Schemenauer -
Raymond Hettinger -
Tim Peters -
Tim Peters