Re: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
On 27 July 2012 23:30, Bradley Froehle <brad.froehle@gmail.com> wrote:
Thanks to the work of Yaroslav Halchenko, there is an experimental Debian package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
However, the builds are showing a lot of test failures on non-amd64 sytems. See https://buildd.debian.org/status/package.php?p=cython&suite=experimental Many of the errors seem related to assumptions made about the sizes of various integers (int vs. long vs long long). Ideally we'll be able to clean up these failures before the final release.
-Brad
Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.
mark florisson, 28.07.2012 14:04:
On 27 July 2012 23:30, Bradley Froehle wrote:
Thanks to the work of Yaroslav Halchenko, there is an experimental Debian package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
However, the builds are showing a lot of test failures on non-amd64 sytems. See https://buildd.debian.org/status/package.php?p=cython&suite=experimental Many of the errors seem related to assumptions made about the sizes of various integers (int vs. long vs long long). Ideally we'll be able to clean up these failures before the final release.
Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.
I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors hint at platform specific differences, which might trigger bugs (aka. incorrect assumptions) in the generated memory view code. Specifically, plain "char" is unsigned at least on ARM and PowerPC, both of which fail with this error. Stefan
On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
mark florisson, 28.07.2012 14:04:
On 27 July 2012 23:30, Bradley Froehle wrote:
Thanks to the work of Yaroslav Halchenko, there is an experimental Debian package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
However, the builds are showing a lot of test failures on non-amd64 sytems. See https://buildd.debian.org/status/package.php?p=cython&suite=experimental Many of the errors seem related to assumptions made about the sizes of various integers (int vs. long vs long long). Ideally we'll be able to clean up these failures before the final release.
Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.
I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors hint at platform specific differences, which might trigger bugs (aka. incorrect assumptions) in the generated memory view code. Specifically, plain "char" is unsigned at least on ARM and PowerPC, both of which fail with this error.
Yes, I think somewhere we're assuming char is signed, but wasn't able to see where in my quick investigations. On another note, I updated jenkins to the latest release of Sage (5.2) and all seems well. - Robert
Robert Bradshaw, 31.07.2012 19:24:
On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel wrote:
mark florisson, 28.07.2012 14:04:
On 27 July 2012 23:30, Bradley Froehle wrote:
Thanks to the work of Yaroslav Halchenko, there is an experimental Debian package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
However, the builds are showing a lot of test failures on non-amd64 sytems. See https://buildd.debian.org/status/package.php?p=cython&suite=experimental Many of the errors seem related to assumptions made about the sizes of various integers (int vs. long vs long long). Ideally we'll be able to clean up these failures before the final release.
Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.
I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors hint at platform specific differences, which might trigger bugs (aka. incorrect assumptions) in the generated memory view code. Specifically, plain "char" is unsigned at least on ARM and PowerPC, both of which fail with this error.
Yes, I think somewhere we're assuming char is signed, but wasn't able to see where in my quick investigations.
Yes, it wasn't immediately obvious to me either. Here is a patch that *might* fix the issue - obviously untested for the platforms in question. Yaroslav, could you give it a try on the Debian build servers? Stefan
sorry about the delay -- was not monitoring the ML tight enough ;) So it is the commit 8443607d7dffc7c8443d70036e0cce6aaa9c26e2 Author: Stefan Behnel <stefan_ml@behnel.de> Date: Tue Jul 31 21:49:20 2012 +0200 determine buffer typegroup of integer dtypes based on signedness at C compile time ... I pulled current master and that specific test which failed before passes now: (s390x-sid)yoh@zelenka:~/cython/cython$ git describe --tags 0.16rc1-547-g68811fa (s390x-sid)yoh@zelenka:~/cython/cython$ OPT="-g -O0" /usr/bin/python runtests.py -vv memoryview_compare_type_pointers --no-cleanup Python 2.7.3 (default, Jul 14 2012, 05:19:55) [GCC 4.6.3] Running tests against Cython 0.17.beta1 68811fa9946e4253ad405ba3011512a32807bc7b Backends: c,cpp runTest (__main__.EndToEndTest) End-to-end memoryview_compare_type_pointers ... ok ---------------------------------------------------------------------- Ran 1 test in 8.641s OK ALL DONE Let me run the entire suite and report back if any oddity. Thanks! On Tue, 31 Jul 2012, Stefan Behnel wrote:
Yaroslav, could you give it a try on the Debian build servers?
Stefan
diff -r 0d14a856f2cd Cython/Compiler/Buffer.py --- a/Cython/Compiler/Buffer.py Tue Jul 31 20:05:37 2012 +0200 +++ b/Cython/Compiler/Buffer.py Tue Jul 31 21:10:13 2012 +0200 @@ -680,32 +680,25 @@ rep = str(dtype)
flags = "0" - + is_unsigned = "0" if dtype.is_int: - if dtype.signed == 0:
-- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
ok, the output of running OPT="-g -O0" /usr/bin/python runtests.py -vv --no-cleanup 2>&1 | tee ../0.16rc1-547-g68811fa-tests-output.txt on a cleaned git repository on an s390x boiling down to Ran 6891 tests in 1098.734s FAILED (failures=42, errors=2) is at http://www.onerussian.com/tmp/0.16rc1-547-g68811fa-tests-output.txt.gz especially interesting are failures like: ValueError: Buffer dtype mismatch, expected 'char' but got 'char' in 'UnpackedStruct.a' ;-) or may be I should have invoked tests anyhow differently (build manually first etc)? On Wed, 01 Aug 2012, Yaroslav Halchenko wrote:
sorry about the delay -- was not monitoring the ML tight enough ;)
So it is the commit 8443607d7dffc7c8443d70036e0cce6aaa9c26e2 Author: Stefan Behnel <stefan_ml@behnel.de> Date: Tue Jul 31 21:49:20 2012 +0200
determine buffer typegroup of integer dtypes based on signedness at C compile time ...
I pulled current master and that specific test which failed before passes now:
(s390x-sid)yoh@zelenka:~/cython/cython$ git describe --tags 0.16rc1-547-g68811fa (s390x-sid)yoh@zelenka:~/cython/cython$ OPT="-g -O0" /usr/bin/python runtests.py -vv memoryview_compare_type_pointers --no-cleanup Python 2.7.3 (default, Jul 14 2012, 05:19:55) [GCC 4.6.3]
Running tests against Cython 0.17.beta1 68811fa9946e4253ad405ba3011512a32807bc7b Backends: c,cpp
runTest (__main__.EndToEndTest) End-to-end memoryview_compare_type_pointers ... ok
---------------------------------------------------------------------- Ran 1 test in 8.641s
OK ALL DONE
Let me run the entire suite and report back if any oddity. Thanks!
-- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
minor note -- could someone push the tag (annotated or signed preferably) for 0.17rc1? NB I am not sure what is the status on PEP 386 [1] (not yet adopted afaik) but it might be worthwhile following it and/or existing disutils.version.StrictVersion since having In [2]: Cython.__version__ Out[2]: '0.17.beta1' is not digested well by some depending projects -- there were 2 failures to build against this beta release of Cython in Debian due to difficulty with parsing the version -- bzr and xpra I believe [2]. So may be it is worth making it '0.17b1' (and use corresponding tags accordingly)? [1] http://www.python.org/dev/peps/pep-0386/ [2] http://bugs.debian.org/683133 On Wed, 01 Aug 2012, Yaroslav Halchenko wrote:
ok, the output of running
OPT="-g -O0" /usr/bin/python runtests.py -vv --no-cleanup 2>&1 | tee ../0.16rc1-547-g68811fa-tests-output.txt on a cleaned git repository on an s390x boiling down to
Ran 6891 tests in 1098.734s FAILED (failures=42, errors=2)
is at http://www.onerussian.com/tmp/0.16rc1-547-g68811fa-tests-output.txt.gz
especially interesting are failures like:
ValueError: Buffer dtype mismatch, expected 'char' but got 'char' in 'UnpackedStruct.a'
;-)
or may be I should have invoked tests anyhow differently (build manually first etc)?
-- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it: https://code.google.com/p/mpi4py/source/detail?r=841e9df -Brad
NB I am not sure what is the status on PEP 386 [1] (not yet adopted afaik) but it might be worthwhile following it and/or existing disutils.version.StrictVersion since having
In [2]: Cython.__version__ Out[2]: '0.17.beta1'
is not digested well by some depending projects -- there were 2 failures to build against this beta release of Cython in Debian due to difficulty with parsing the version -- bzr and xpra I believe [2]. So may be it is worth making it '0.17b1' (and use corresponding tags accordingly)?
[1] http://www.python.org/dev/peps/pep-0386/ [2] http://bugs.debian.org/683133
Bradley M. Froehle, 01.08.2012 18:35:
Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it:
https://code.google.com/p/mpi4py/source/detail?r=841e9df
I am not sure what is the status on PEP 386 [1] (not yet adopted afaik)
It's in state "Accepted".
but it might be worthwhile following it and/or existing
disutils.version.StrictVersion since having
In [2]: Cython.__version__ Out[2]: '0.17.beta1'
is not digested well by some depending projects -- there were 2 failures to build against this beta release of Cython in Debian due to difficulty with parsing the version -- bzr and xpra I believe [2]. So may be it is worth making it '0.17b1' (and use corresponding tags accordingly)?
I guess so. That would fit the StrictVersion scheme. Stefan
On Thu, 02 Aug 2012, Stefan Behnel wrote:
Bradley M. Froehle, 01.08.2012 18:35:
Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it:
I am not sure what is the status on PEP 386 [1] (not yet adopted afaik)
It's in state "Accepted".
ah right -- now it is in python3 (but not in python2): (git)novo:~/proj/misc/cpython[2.7]git $> git grep NormalizedVersion 2.7 $> git grep NormalizedVersion master master:Doc/library/packaging.version.rst:.. class:: NormalizedVersion(self, s, error_on_huge_major_num=True) master:Doc/library/packaging.version.rst: >>> NormalizedVersion('1.2b1') < NormalizedVersion('1.2') master:Doc/library/packaging.version.rst: :class:`NormalizedVersion` is used internally by :class:`VersionPredicate` to master:Doc/library/packaging.version.rst: :class:`NormalizedVersion`. master:Doc/library/packaging.version.rst: >>> NormalizedVersion("irrational_version_number") master:Lib/packaging/command/bdist_msi.py:from packaging.version import NormalizedVersion master:Lib/packaging/command/bdist_msi.py:class MSIVersion(NormalizedVersion): master:Lib/packaging/errors.py: See `error_on_huge_major_num` option in `NormalizedVersion` for details. master:Lib/packaging/pypi/dist.py:from packaging.version import (suggest_normalized_version, NormalizedVersion, master:Lib/packaging/pypi/dist.py: self._version = NormalizedVersion(version) master:Lib/packaging/tests/test_version.py:from packaging.version import NormalizedVersion as V master:Lib/packaging/tests/test_version.py: self.assertEqual(repr(V('1.0')), "NormalizedVersion('1.0')") master:Lib/packaging/tests/test_version.py: TypeError: cannot compare NormalizedVersion and str master:Lib/packaging/tests/test_version.py: TypeError: cannot compare NormalizedVersion and str master:Lib/packaging/version.py:__all__ = ['NormalizedVersion', 'suggest_normalized_version', master:Lib/packaging/version.py:class NormalizedVersion: master:Lib/packaging/version.py: """Create a NormalizedVersion instance from a version string. master:Lib/packaging/version.py: the introduction of `NormalizedVersion` was version numbers like master:Lib/packaging/version.py: if not isinstance(other, NormalizedVersion): master:Lib/packaging/version.py: if not isinstance(other, NormalizedVersion): master:Lib/packaging/version.py: If you have a version string that isn't rational (i.e. NormalizedVersion master:Lib/packaging/version.py: - 2312 (53.93%) match NormalizedVersion without change master:Lib/packaging/version.py: NormalizedVersion(s) master:Lib/packaging/version.py: NormalizedVersion(rs) master:Lib/packaging/version.py: return comp, NormalizedVersion(version) master:Lib/packaging/version.py: version = NormalizedVersion(version)
parsing the version -- bzr and xpra I believe [2]. So may be it is worth making it '0.17b1' (and use corresponding tags accordingly)?
I guess so. That would fit the StrictVersion scheme.
just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree: $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' True at least in python3 it pukes: $> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ c = self._cmp(other) File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp if self.version < other.version: TypeError: unorderable types: tuple() < list() -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Yaroslav Halchenko, 02.08.2012 19:02:
just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree:
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' True
at least in python3 it pukes:
$> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ c = self._cmp(other) File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp if self.version < other.version: TypeError: unorderable types: tuple() < list()
That's surprising at first sight, but I don't think it hurts all that much. People would normally use either of them, not both. Anyway, the next release candidate of Cython will be called 0.17c1. Both the LooseVersion and the NormalizedVersion should be able to handle that. Stefan
On Thu, 02 Aug 2012, Stefan Behnel wrote:
just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree:
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' True
at least in python3 it pukes:
$> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ c = self._cmp(other) File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp if self.version < other.version: TypeError: unorderable types: tuple() < list()
That's surprising at first sight, but I don't think it hurts all that much. People would normally use either of them, not both.
I understand that but absence of error in python2.x case (thus suggesting supporting such a comparison) is misleading at least
Anyway, the next release candidate of Cython will be called 0.17c1. Both the LooseVersion and the NormalizedVersion should be able to handle that.
not quite: 1. $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17c1"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/distutils/version.py", line 40, in __init__ self.parse(vstring) File "/usr/lib/python2.7/distutils/version.py", line 107, in parse raise ValueError, "invalid version number '%s'" % vstring ValueError: invalid version number '0.17c1' so it needs to be 0.17b1 I guess $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17b1"))' True 2. LooseVersion is too loose to be used with standardized suffixes, so it would not sort those release candidates appropriately $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(LV("0.17") > LV("0.17b1"))' False -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Yaroslav Halchenko, 02.08.2012 21:54:
On Thu, 02 Aug 2012, Stefan Behnel wrote:
just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree:
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' True
at least in python3 it pukes:
$> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ c = self._cmp(other) File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp if self.version < other.version: TypeError: unorderable types: tuple() < list()
That's surprising at first sight, but I don't think it hurts all that much. People would normally use either of them, not both.
I understand that but absence of error in python2.x case (thus suggesting supporting such a comparison) is misleading at least
Anyway, the next release candidate of Cython will be called 0.17c1. Both the LooseVersion and the NormalizedVersion should be able to handle that.
not quite:
1.
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17c1"))' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/distutils/version.py", line 40, in __init__ self.parse(vstring) File "/usr/lib/python2.7/distutils/version.py", line 107, in parse raise ValueError, "invalid version number '%s'" % vstring ValueError: invalid version number '0.17c1'
so it needs to be 0.17b1 I guess
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17b1"))' True
2. LooseVersion is too loose to be used with standardized suffixes, so it would not sort those release candidates appropriately
$> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(LV("0.17") > LV("0.17b1"))' False
Great. Then we'll have a "release candidate" called "beta" for the sake of backwards compatibility. I'm not entirely happy about that. Stefan
On 31 July 2012 20:14, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 31.07.2012 19:24:
On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel wrote:
mark florisson, 28.07.2012 14:04:
On 27 July 2012 23:30, Bradley Froehle wrote:
Thanks to the work of Yaroslav Halchenko, there is an experimental Debian package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
However, the builds are showing a lot of test failures on non-amd64 sytems. See https://buildd.debian.org/status/package.php?p=cython&suite=experimental Many of the errors seem related to assumptions made about the sizes of various integers (int vs. long vs long long). Ideally we'll be able to clean up these failures before the final release.
Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.
I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors hint at platform specific differences, which might trigger bugs (aka. incorrect assumptions) in the generated memory view code. Specifically, plain "char" is unsigned at least on ARM and PowerPC, both of which fail with this error.
Yes, I think somewhere we're assuming char is signed, but wasn't able to see where in my quick investigations.
Yes, it wasn't immediately obvious to me either. Here is a patch that *might* fix the issue - obviously untested for the platforms in question.
Yaroslav, could you give it a try on the Debian build servers?
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Thanks for the fix. I also pushed a fix for one more test numpy_test related to fused types dispatching. That passes all tests for me on 32 bit linux.
On Wed, 01 Aug 2012, mark florisson wrote:
Thanks for the fix. I also pushed a fix for one more test numpy_test related to fused types dispatching. That passes all tests for me on 32 bit linux.
Yaroslav, could you give it a try on the Debian build servers?
FWIW -- 0.16rc1-550-g8880c78 hasn't added nor fixed any of the failures on s390x in comparison to 0.16rc1-547-g68811fa ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
participants (5)
-
Bradley M. Froehle -
mark florisson -
Robert Bradshaw -
Stefan Behnel -
Yaroslav Halchenko