Numpy 1.4.1 fails to build on (Debian) alpha and powepc
Hello, I finally found the time to update numpy in Debian. But, there is a problem... As you probably know, we support several architectures and we need to have any package available on each of them. After the upload I noticed numpy has problem building on alpha [1] (ieee754.c compilation error, probably related to the conversion from .src) and powerpc [2] (long_double identification) [1] https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=alpha&ver=1%3A1.4.1-2&stamp=1279394977&file=log&as=raw [2] https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-2&stamp=1279394008&file=log&as=raw They are quite specific pieces of code, so I'm here asking your help in order to be able to build numpy on all Debian supported architectures and so be released in the upcoming stable release :) Of course, feel free to ask me any kind of support, also if you need to test/execute something on those archs, just tell me and I'll do. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Sun, Jul 18, 2010 at 12:53 PM, Sandro Tosi <morph@debian.org> wrote:
Hello, I finally found the time to update numpy in Debian. But, there is a problem...
As you probably know, we support several architectures and we need to have any package available on each of them. After the upload I noticed numpy has problem building on alpha [1] (ieee754.c compilation error, probably related to the conversion from .src) and powerpc [2] (long_double identification)
The alpha issue should be easy to fix: it is just a code path which has never been tested so far, but the code is there. The ppc one is more annoying, and known. The problem is that linux on ppc uses the IBM format for long double (sum of two double), which requires to implement non trivial, heavily platform dependent code. I have just added code to correctly detect the long double format at the configure stage. Do you have shell access to the machines ? It would makes the work much easier for me to fix those issues, David
Hi David, thanks for your reply! On Mon, Jul 19, 2010 at 20:10, David Cournapeau <cournape@gmail.com> wrote:
On Sun, Jul 18, 2010 at 12:53 PM, Sandro Tosi <morph@debian.org> wrote:
Hello, I finally found the time to update numpy in Debian. But, there is a problem...
As you probably know, we support several architectures and we need to have any package available on each of them. After the upload I noticed numpy has problem building on alpha [1] (ieee754.c compilation error, probably related to the conversion from .src) and powerpc [2] (long_double identification)
The alpha issue should be easy to fix: it is just a code path which has never been tested so far, but the code is there.
ah if you say so, I trust you :)
The ppc one is more annoying, and known. The problem is that linux on ppc uses the IBM format for long double (sum of two double), which requires to implement non trivial, heavily platform dependent code. I have just added code to correctly detect the long double format at the configure stage.
yes, I see it at r8510
Do you have shell access to the machines ? It would makes the work much easier for me to fix those issues,
Yes, as a Debian developer I have access to a porterbox of any architecture we support, so I can do any test/recompilation and so you want. I can understand that using me as a proxy might be a little bit tedious and slow, so if you prefer to have direct access I have to ask for it, but it might be refused (I'm not sure), and/or only be temporary. Just let me know: I'm fully available for you. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Mon, Jul 19, 2010 at 8:44 PM, Sandro Tosi <morph@debian.org> wrote:
yes, I see it at r8510
I quickly adapted the code from the Sun math library for the linux ppc long double format. Let me know if it works (if possible, you should run the test suite). I will look at the alpha thing later, David
Hi David, On Tue, Jul 20, 2010 at 10:34, David Cournapeau <cournape@gmail.com> wrote:
yes, I see it at r8510
I quickly adapted the code from the Sun math library for the linux ppc long double format. Let me know if it works (if possible, you should run the test suite).
thanks for working on it :) I checked out numpy at r8511 and built it on powerpc, attached the buildlog (it crashed at doc generation, but the setup.py build were done before, and went fine). I've python2.x setup.py install --prefix install/ and execute the tests from there with: ~/numpy/install$ PYTHONPATH=lib/python2.6/site-packages/ python2.6 -c "import numpy; print numpy.test()" &> ../testlog_2.6 ~/numpy/install$ PYTHONPATH=lib/python2.5/site-packages/ python2.5 -c "import numpy; print numpy.test()" &> ../testlog_2.5 attached the testlogs too: there are a couple of failures.
I will look at the alpha thing later,
Thanks a lot for your support! Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Tue, Jul 20, 2010 at 4:21 PM, Sandro Tosi <morph@debian.org> wrote:
Hi David,
On Tue, Jul 20, 2010 at 10:34, David Cournapeau <cournape@gmail.com> wrote:
yes, I see it at r8510
I quickly adapted the code from the Sun math library for the linux ppc long double format. Let me know if it works (if possible, you should run the test suite).
thanks for working on it :)
I checked out numpy at r8511 and built it on powerpc, attached the buildlog (it crashed at doc generation, but the setup.py build were done before, and went fine).
I've python2.x setup.py install --prefix install/ and execute the tests from there with:
~/numpy/install$ PYTHONPATH=lib/python2.6/site-packages/ python2.6 -c "import numpy; print numpy.test()" &> ../testlog_2.6 ~/numpy/install$ PYTHONPATH=lib/python2.5/site-packages/ python2.5 -c "import numpy; print numpy.test()" &> ../testlog_2.5
attached the testlogs too: there are a couple of failures.
The failures seem to be related to the long double not conforming to IEEE754 standard on linux ppc. I am not sure how to deal with them - maybe raising a warning if the user uses long double, as its usage will always be flaky on that platform anyway (numpy assumes IEEE 754-like support) David
Hi David & others, On Tue, Jul 20, 2010 at 19:09, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 20, 2010 at 4:21 PM, Sandro Tosi <morph@debian.org> wrote:
Hi David,
On Tue, Jul 20, 2010 at 10:34, David Cournapeau <cournape@gmail.com> wrote:
yes, I see it at r8510
I quickly adapted the code from the Sun math library for the linux ppc long double format. Let me know if it works (if possible, you should run the test suite).
thanks for working on it :)
I checked out numpy at r8511 and built it on powerpc, attached the buildlog (it crashed at doc generation, but the setup.py build were done before, and went fine).
I've python2.x setup.py install --prefix install/ and execute the tests from there with:
~/numpy/install$ PYTHONPATH=lib/python2.6/site-packages/ python2.6 -c "import numpy; print numpy.test()" &> ../testlog_2.6 ~/numpy/install$ PYTHONPATH=lib/python2.5/site-packages/ python2.5 -c "import numpy; print numpy.test()" &> ../testlog_2.5
attached the testlogs too: there are a couple of failures.
The failures seem to be related to the long double not conforming to IEEE754 standard on linux ppc. I am not sure how to deal with them - maybe raising a warning if the user uses long double, as its usage will always be flaky on that platform anyway (numpy assumes IEEE 754-like support)
yeah a warning might be nice. Just to keep you informed, I tested 1.4.1+r8510+r8510 and it builds fine on the porterbox only showing this failure on test(): FAIL: test_umath.TestComplexFunctions.test_loss_of_precision(<type 'numpy.complex64'>,) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest self.test(*self.arg) File "/usr/lib/python2.6/dist-packages/numpy/core/tests/test_umath.py", line 524, in check_loss_of_precision assert np.all(d < 1e-15) AssertionError ---------------------------------------------------------------------- Ran 2016 tests in 52.670s FAILED (KNOWNFAIL=2, failures=1) <nose.result.TextTestResult run=2016 errors=0 failures=1> Hence I decided to upload and the build went fine on the buildd machine [1]. [1] https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-3&stamp=1279914539&file=log&as=raw the other big show-stopper for numpy is the fail to build on alpha you're already aware of: if you need any kind of support, just ask me. Thanks a lot for your support for powerpc issue! Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Tue, Jul 20, 2010 at 3:44 AM, Sandro Tosi <morph@debian.org> wrote:
ah if you say so, I trust you :)
Could you try the last version of the trunk, I added the missing macros for alpha ? cheers, David
Hi, On Tue, Jul 27, 2010 at 12:28, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:44 AM, Sandro Tosi <morph@debian.org> wrote:
ah if you say so, I trust you :)
Could you try the last version of the trunk, I added the missing macros for alpha ?
I just built trunk (both for 2.5 and 2.6) on alpha, successfully :) Now I'll extract only the needed patch (I think it's enough to apply r8526, right?) and try to build the package using 1.4.1+patch, if it builds successfully I'll upload numpy in Debian again. David, let me thank you very very much for your support!! Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Tue, Jul 27, 2010 at 14:52, Sandro Tosi <morph@debian.org> wrote:
Hi,
On Tue, Jul 27, 2010 at 12:28, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:44 AM, Sandro Tosi <morph@debian.org> wrote:
ah if you say so, I trust you :)
Could you try the last version of the trunk, I added the missing macros for alpha ?
I just built trunk (both for 2.5 and 2.6) on alpha, successfully :) Now I'll extract only the needed patch (I think it's enough to apply r8526, right?) and try to build the package using 1.4.1+patch, if it builds successfully I'll upload numpy in Debian again.
and indeed it build! yay \o/
David, let me thank you very very much for your support!!
Renew the great THANK YOU! Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Wed, Jul 28, 2010 at 12:36 AM, Sandro Tosi <morph@debian.org> wrote:
On Tue, Jul 27, 2010 at 14:52, Sandro Tosi <morph@debian.org> wrote:
Hi,
On Tue, Jul 27, 2010 at 12:28, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:44 AM, Sandro Tosi <morph@debian.org> wrote:
ah if you say so, I trust you :)
Could you try the last version of the trunk, I added the missing macros for alpha ?
I just built trunk (both for 2.5 and 2.6) on alpha, successfully :) Now I'll extract only the needed patch (I think it's enough to apply r8526, right?) and try to build the package using 1.4.1+patch, if it builds successfully I'll upload numpy in Debian again.
and indeed it build! yay \o/
David, let me thank you very very much for your support!!
Renew the great THANK YOU!
If possible, I would really like to get build and test output logs for both alpha/ppc. Also, if there is another issue preventing numpy 1.4.x integration on debian and ubuntu, please speak up. Ideally, I would like to remove the need for downstream patches (none of them was necessary IIRC), cheers, David
Hi! sorry it took so long to reply. On Thu, Jul 29, 2010 at 03:27, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Jul 28, 2010 at 12:36 AM, Sandro Tosi <morph@debian.org> wrote:
On Tue, Jul 27, 2010 at 14:52, Sandro Tosi <morph@debian.org> wrote:
Hi,
On Tue, Jul 27, 2010 at 12:28, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 20, 2010 at 3:44 AM, Sandro Tosi <morph@debian.org> wrote:
ah if you say so, I trust you :)
Could you try the last version of the trunk, I added the missing macros for alpha ?
I just built trunk (both for 2.5 and 2.6) on alpha, successfully :) Now I'll extract only the needed patch (I think it's enough to apply r8526, right?) and try to build the package using 1.4.1+patch, if it builds successfully I'll upload numpy in Debian again.
and indeed it build! yay \o/
David, let me thank you very very much for your support!!
Renew the great THANK YOU!
If possible, I would really like to get build and test output logs for both alpha/ppc.
For the build logs it's easy: alpha: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=alpha&ver=1%3A1.4.1-4&stamp=1280296333&file=log&as=raw powerpc: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-4&stamp=1280297029&file=log&as=raw for powerpc "import numpy; numpy.test()" I've already sent you the output, want me to re-run them? for alpha, attached you can find the log for both 2.5 and 2.6; there are some errors/warnings but nothing too dramatic?
Also, if there is another issue preventing numpy 1.4.x integration on debian and ubuntu, please speak up. Ideally, I would like to remove
I don't think there is anything else (for now :) ) from the numpy side: Thanks a lot for the support!! Now on Debian we have to fix some packages to avoid breakages when we upgrade numpy in the future (the biggest issue was that dtype was extended with new fields at the end, but some packages were checking the size of dtype with the one the packge was compiled with (1.3.*) and failed). As usual, Ubuntu will just sit and wait for us to do the work and then just sync it (sigh).
the need for downstream patches (none of them was necessary IIRC),
Here is the list of the patches we currently have in the Debian package (you can look at them at [2]): 02_build_dotblas.patch - Patch to build _dotblas.c when ATLAS is not installed. -- dunno exactly what it does, it seems to infer _dotblas is compiled is ATLAS is missing 03_force_f2py_version.patch - force generation f2py postfixed with interpreter version -- Debian specific: we ship f2py2.5 and f2py2.6 and we make f2py a symlink towards f2py2.6 05_fix_endianness_detection.patch - Fix endianness detection: endian.h should be present on all Debian machines. This patch forces the use of endian.h, this reventing several reverse dependencies os Numpy from failing to build. -- Debian specific: we want to enforce the usage of endian.h file available on all of our architectures 07_bts585309_string_exceptions.diff - Remove string exceptions -- patch from trunk, we can remove it once a new release is out 10_use_local_python.org_object.inv_sphinx.diff - Use a local copy of object.inv from doc.python.org, instead of downloading it each time from the internet -- Debian specific, we are not allowed to download stuff from internet when building package, so I just download object.inv from d.p.o and use it in interpshinx_mapping changeset_r8510.diff changeset_r8511.diff - support for powerpc -- patch from trunk, we can remove it once a new release is out changeset_r8526.diff - support for alpha - patch from trunk, we can remove it once a new release is out [2] http://svn.debian.org/viewsvn/python-modules/packages/numpy/trunk/debian/pat... In case of any doubts or so, don't hesitate to contact me: I'd be more than happy to give all the help I can. Cheers. -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On 07/30/2010 06:47 AM, Sandro Tosi wrote:
For the build logs it's easy:
alpha: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=alpha&ver=1%3A1.4.1-4&stamp=1280296333&file=log&as=raw powerpc: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-4&stamp=1280297029&file=log&as=raw
for powerpc "import numpy; numpy.test()" I've already sent you the output, want me to re-run them? for alpha, attached you can find the log for both 2.5 and 2.6; there are some errors/warnings but nothing too dramatic?
Wow, I am genuily surprised that the alpha test suite has no error (the 2.5 error has nothing to do with the fixes).
Also, if there is another issue preventing numpy 1.4.x integration on debian and ubuntu, please speak up. Ideally, I would like to remove
I don't think there is anything else (for now :) ) from the numpy side: Thanks a lot for the support!! Now on Debian we have to fix some packages to avoid breakages when we upgrade numpy in the future (the biggest issue was that dtype was extended with new fields at the end, but some packages were checking the size of dtype with the one the packge was compiled with (1.3.*) and failed).
Yes, we have improved quite a bit our support here in 1.4.x - we hope that those issues won't arise in the 1.x series anymore. Note also that if those dtype errors appear with pyrex/cython-generated code, using a more recent cython will prevent the error from happening (warnings raised instead).
As usual, Ubuntu will just sit and wait for us to do the work and then just sync it (sigh).
the need for downstream patches (none of them was necessary IIRC),
Here is the list of the patches we currently have in the Debian package (you can look at them at [2]):
02_build_dotblas.patch - Patch to build _dotblas.c when ATLAS is not installed. -- dunno exactly what it does, it seems to infer _dotblas is compiled is ATLAS is missing
This is is caused by not having atlas as a build dependency I guess. Strictly speaking, dotblas only requires cblas, but we don't have the check in place to do so. Since numscons already does this, and it has worked pretty well, maybe I will take time to add this as well in numpy.distutils. But this has relatively little consequence I think.
03_force_f2py_version.patch - force generation f2py postfixed with interpreter version -- Debian specific: we ship f2py2.5 and f2py2.6 and we make f2py a symlink towards f2py2.6
ok.
05_fix_endianness_detection.patch - Fix endianness detection: endian.h should be present on all Debian machines. This patch forces the use of endian.h, this reventing several reverse dependencies os Numpy from failing to build. -- Debian specific: we want to enforce the usage of endian.h file available on all of our architectures
This one has been fixed in the 1.4.x branch (and trunk of course)
07_bts585309_string_exceptions.diff - Remove string exceptions -- patch from trunk, we can remove it once a new release is out
This one as well
In case of any doubts or so, don't hesitate to contact me: I'd be more than happy to give all the help I can.
great, thanks, David
Hi David, sorry for the late reply. On Fri, Jul 30, 2010 at 04:58, David <david@silveregg.co.jp> wrote:
On 07/30/2010 06:47 AM, Sandro Tosi wrote:
For the build logs it's easy:
alpha: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=alpha&ver=1%3A1.4.1-4&stamp=1280296333&file=log&as=raw powerpc: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-4&stamp=1280297029&file=log&as=raw
for powerpc "import numpy; numpy.test()" I've already sent you the output, want me to re-run them? for alpha, attached you can find the log for both 2.5 and 2.6; there are some errors/warnings but nothing too dramatic?
Wow, I am genuily surprised that the alpha test suite has no error (the 2.5 error has nothing to do with the fixes).
:) Anyhow, I plan to enable tests execution at package build time, so that we can spot "strange" behaviors/faults on all the archs supported by Debian.
Also, if there is another issue preventing numpy 1.4.x integration on debian and ubuntu, please speak up. Ideally, I would like to remove
I don't think there is anything else (for now :) ) from the numpy side: Thanks a lot for the support!! Now on Debian we have to fix some packages to avoid breakages when we upgrade numpy in the future (the biggest issue was that dtype was extended with new fields at the end, but some packages were checking the size of dtype with the one the packge was compiled with (1.3.*) and failed).
Yes, we have improved quite a bit our support here in 1.4.x - we hope that those issues won't arise in the 1.x series anymore. Note also that if those dtype errors appear with pyrex/cython-generated code, using a more recent cython will prevent the error from happening (warnings raised instead).
Ah, thanks for letting me know! I'll keep in mind.
As usual, Ubuntu will just sit and wait for us to do the work and then just sync it (sigh).
the need for downstream patches (none of them was necessary IIRC),
Here is the list of the patches we currently have in the Debian package (you can look at them at [2]):
02_build_dotblas.patch - Patch to build _dotblas.c when ATLAS is not installed. -- dunno exactly what it does, it seems to infer _dotblas is compiled is ATLAS is missing
This is is caused by not having atlas as a build dependency I guess. Strictly speaking, dotblas only requires cblas, but we don't have the check in place to do so. Since numscons already does this, and it has worked pretty well, maybe I will take time to add this as well in numpy.distutils. But this has relatively little consequence I think.
yep, I agree, it would be nice tho to have it directly upstream.
03_force_f2py_version.patch - force generation f2py postfixed with interpreter version -- Debian specific: we ship f2py2.5 and f2py2.6 and we make f2py a symlink towards f2py2.6
ok.
05_fix_endianness_detection.patch - Fix endianness detection: endian.h should be present on all Debian machines. This patch forces the use of endian.h, this reventing several reverse dependencies os Numpy from failing to build. -- Debian specific: we want to enforce the usage of endian.h file available on all of our architectures
This one has been fixed in the 1.4.x branch (and trunk of course)
07_bts585309_string_exceptions.diff - Remove string exceptions -- patch from trunk, we can remove it once a new release is out
This one as well
I'll live those patches there, until a new release will be out, or do you suggest to sync the package with the 1.4.x branch? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi
On Tue, Jul 27, 2010 at 9:52 PM, Sandro Tosi <morph@debian.org> wrote:
I just built trunk (both for 2.5 and 2.6) on alpha, successfully :) Now I'll extract only the needed patch (I think it's enough to apply r8526, right?) and try to build the package using 1.4.1+patch, if it builds successfully I'll upload numpy in Debian again.
I have updated the 1.4.x and 1.5.x branches as well so that you could use this instead, cheers, David
participants (3)
-
David -
David Cournapeau -
Sandro Tosi