Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Cheers, Matthew
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection? Chuck
Hi, On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
I'm on it - will get back to you tomorrow. See you, Matthew
Hi, On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log: * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe] So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad) There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a: gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 See you, Matthew
On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests
~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log:
* df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe]
So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad)
There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a:
If you enable separate compilation by setting an environment variable, these commits should build as well. $ export ENABLE_SEPARATE_COMPILATION=1 <proceed with builds> Cheers, Mark
gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o
build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1
See you,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests
~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log:
* df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe]
So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad)
There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a:
If you enable separate compilation by setting an environment variable, these commits should build as well.
$ export ENABLE_SEPARATE_COMPILATION=1 <proceed with builds>
I might be doing something wrong but: In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 (bare-env)[matthew@vagus ~/dev_trees/numpy ((9194b3a...)|BISECTING)]$ echo $ENABLE_SEPARATE_COMPILATION 1 Best, Matthew
On Sun, Mar 4, 2012 at 10:08 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com
wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <
matthew.brett@gmail.com>
wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests
~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log:
* df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe]
So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad)
There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a:
If you enable separate compilation by setting an environment variable,
On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe <mwwiebe@gmail.com> wrote: these
commits should build as well.
$ export ENABLE_SEPARATE_COMPILATION=1 <proceed with builds>
I might be doing something wrong but:
I made a mistake, sorry! I even copy/pasted the variable name to make sure I didn't misspell it, but I didn't notice that setup.py uses a different but nearly identical name internally as the environment variable. That should have been $ export NPY_SEPARATE_COMPILATION=1 The place this is used inside of setup.py is here: https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L14 I really dislike this build feature, it repeatedly trips me up. In my opinion, the build should be changed to always do separate compilation, and the single file mode should be eradicated. Thanks , Mark
In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o
build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1
(bare-env)[matthew@vagus ~/dev_trees/numpy ((9194b3a...)|BISECTING)]$ echo $ENABLE_SEPARATE_COMPILATION 1
Best,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Sun, Mar 4, 2012 at 8:32 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 10:08 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote: > > Hi, > > Sorry that this report is not complete, I don't have full access to > this box but, on a Debian squeeze machine running linux > 2.6.32-5-sparc64-smp: > > nosetests > > > ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values > > test_user_missing_values (test_io.TestFromTxt) ... Bus error > > This on current master : 1.7.0.dev-b9872b4 >
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log:
* df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe]
So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad)
There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a:
If you enable separate compilation by setting an environment variable, these commits should build as well.
$ export ENABLE_SEPARATE_COMPILATION=1 <proceed with builds>
I might be doing something wrong but:
I made a mistake, sorry! I even copy/pasted the variable name to make sure I didn't misspell it, but I didn't notice that setup.py uses a different but nearly identical name internally as the environment variable. That should have been
$ export NPY_SEPARATE_COMPILATION=1
Thanks, that did it: 9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit Let me know if I can debug further, See you, Matthew
On Sun, Mar 4, 2012 at 10:34 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
<snip>
$ export NPY_SEPARATE_COMPILATION=1
Thanks, that did it:
9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit
Let me know if I can debug further,
That commit was a rewrite of np.concatenate, and I've traced the test function you got the crash in. The only call to concatenate is as follows:
a = np.array([True], dtype=object) np.concatenate((a,)*3) array([True, True, True], dtype=object)
Can you try this and see if it crashes? Another thing you can do is compile with debug information enabled, then run the crashing case in gdb. This will look something like this: $ *export CFLAGS=-g* $ *rm -rf build* # make sure it's a fresh build from scratch $ *python setup.py install --prefix=<dir>* # or however you do it [... build printout] $ *gdb python <script/options>* [... gdb info] (gdb) *run* [... script runs] Program received signal SIGSEGV, Segmentation fault, [crash address] (gdb) *backtrace* [full backtrace printout] Such a backtrace would be immensely helpful in tracking down why it's crashing. Thanks, Mark
See you,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Sun, Mar 4, 2012 at 11:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 10:34 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
<snip>
$ export NPY_SEPARATE_COMPILATION=1
Thanks, that did it:
9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit
Let me know if I can debug further,
That commit was a rewrite of np.concatenate, and I've traced the test function you got the crash in. The only call to concatenate is as follows:
a = np.array([True], dtype=object) np.concatenate((a,)*3) array([True, True, True], dtype=object)
Can you try this and see if it crashes?
No, that doesn't crash. Further investigation revealed the crash to be: (bare-env)[matthew@vagus ~]$ nosetests ~/dev_trees/numpy/numpy/lib/tests/test_io.py:TestFromTxt.test_with_masked_column_various nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] Test masked column ... Bus error Accordingly: In [1]: import numpy as np In [2]: from StringIO import StringIO In [3]: data = StringIO('True 2 3\nFalse 5 6\n') In [4]: test = np.genfromtxt(data, dtype=None, missing_values='2,5', usemask=True) In [6]: from numpy import ma In [7]: control = ma.array([(1, 2, 3), (0, 5, 6)], mask=[(0, 1, 0), (0, 1, 0)], dtype=[('f0', bool), ('f1', bool), ('f2', int)]) In [8]: test == control Bus error
Another thing you can do is compile with debug information enabled, then run the crashing case in gdb. This will look something like this:
$ export CFLAGS=-g $ rm -rf build # make sure it's a fresh build from scratch $ python setup.py install --prefix=<dir> # or however you do it [... build printout] $ gdb python <script/options> [... gdb info] (gdb) run [... script runs] Program received signal SIGSEGV, Segmentation fault, [crash address] (gdb) backtrace [full backtrace printout]
Such a backtrace would be immensely helpful in tracking down why it's crashing.
I'll get back to you with this. See you, Matthew
Hi, On Mon, Mar 5, 2012 at 11:11 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Mar 4, 2012 at 11:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 10:34 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
<snip>
$ export NPY_SEPARATE_COMPILATION=1
Thanks, that did it:
9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit
Let me know if I can debug further,
That commit was a rewrite of np.concatenate, and I've traced the test function you got the crash in. The only call to concatenate is as follows:
a = np.array([True], dtype=object) np.concatenate((a,)*3) array([True, True, True], dtype=object)
Can you try this and see if it crashes?
No, that doesn't crash.
Further investigation revealed the crash to be:
(bare-env)[matthew@vagus ~]$ nosetests ~/dev_trees/numpy/numpy/lib/tests/test_io.py:TestFromTxt.test_with_masked_column_various nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] Test masked column ... Bus error
Accordingly:
In [1]: import numpy as np
In [2]: from StringIO import StringIO
In [3]: data = StringIO('True 2 3\nFalse 5 6\n')
In [4]: test = np.genfromtxt(data, dtype=None, missing_values='2,5', usemask=True)
In [6]: from numpy import ma
In [7]: control = ma.array([(1, 2, 3), (0, 5, 6)], mask=[(0, 1, 0), (0, 1, 0)], dtype=[('f0', bool), ('f1', bool), ('f2', int)])
In [8]: test == control Bus error
Another thing you can do is compile with debug information enabled, then run the crashing case in gdb. This will look something like this:
$ export CFLAGS=-g $ rm -rf build # make sure it's a fresh build from scratch $ python setup.py install --prefix=<dir> # or however you do it [... build printout] $ gdb python <script/options> [... gdb info] (gdb) run [... script runs] Program received signal SIGSEGV, Segmentation fault, [crash address] (gdb) backtrace [full backtrace printout]
Such a backtrace would be immensely helpful in tracking down why it's crashing.
I'll get back to you with this.
<atest.py> from StringIO import StringIO import numpy as np from numpy import ma data = StringIO('True 2 3\nFalse 5 6\n') test = np.genfromtxt(data, dtype=None, missing_values='2,5', usemask=True) control = ma.array([(1, 2, 3), (0, 5, 6)], mask=[(0, 1, 0), (0, 1, 0)], dtype=[('f0', bool), ('f1', bool), ('f2', int)]) test == control </atest.py gdb python run atest.py backtrace full attached. See you, Matthew
And simplifying: In [1]: import numpy as np In [2]: control = np.array([(1, 2, 3), (0, 5, 6)], dtype=[('f0', bool), ('f1', bool), ('f2', int)]) In [3]: control == control Out[3]: array([ True, True], dtype=bool) In [4]: from numpy import ma In [5]: control = ma.array([(1, 2, 3), (0, 5, 6)], dtype=[('f0', bool), ('f1', bool), ('f2', int)]) In [6]: control == control Bus error Cheers, Matthew
I've pushed a bugfix to github, can you confirm that the crash goes away on your test box? Thanks for tracking that down, the stack trace was very helpful. Since x86 machines don't have as strict alignment requirements, bugs like this one will generally remain undetected until someone tests on an architecture like sparc. Cheers, Mark On Mon, Mar 5, 2012 at 6:39 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
And simplifying:
In [1]: import numpy as np
In [2]: control = np.array([(1, 2, 3), (0, 5, 6)], dtype=[('f0', bool), ('f1', bool), ('f2', int)])
In [3]: control == control Out[3]: array([ True, True], dtype=bool)
In [4]: from numpy import ma
In [5]: control = ma.array([(1, 2, 3), (0, 5, 6)], dtype=[('f0', bool), ('f1', bool), ('f2', int)])
In [6]: control == control Bus error
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Mon, Mar 5, 2012 at 8:04 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
I've pushed a bugfix to github, can you confirm that the crash goes away on your test box? Thanks for tracking that down, the stack trace was very helpful. Since x86 machines don't have as strict alignment requirements, bugs like this one will generally remain undetected until someone tests on an architecture like sparc.
Thanks - no Bus error. For your enjoyment: there were some failures and errors from numpy.test("full") ====================================================================== ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1, 1000020.0000000099], [0, nan, 1000000.0]) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/tests/test_numeric.py", line 1288, in tst_isclose_allclose assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y)) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/numeric.py", line 2020, in allclose return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) RuntimeWarning: invalid value encountered in absolute ====================================================================== ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan, nan]) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/tests/test_numeric.py", line 1288, in tst_isclose_allclose assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y)) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/numeric.py", line 2020, in allclose return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) RuntimeWarning: invalid value encountered in absolute ====================================================================== ERROR: Test a special case for var ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/ma/tests/test_core.py", line 2725, in test_varstd_specialcases _ = method(out=nout) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/ma/core.py", line 4778, in std dvar = sqrt(dvar) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/ma/core.py", line 849, in __call__ m |= self.domain(d) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/ma/core.py", line 801, in __call__ return umath.less(x, self.critical_value) RuntimeWarning: invalid value encountered in less ====================================================================== FAIL: test_complex_dtype_repr (test_dtype.TestString) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/tests/test_dtype.py", line 401, in test_complex_dtype_repr "dtype([('a', '<M8[D]'), ('b', '<m8[us]')])") File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: "dtype([('a', '>M8[D]'), ('b', '>m8[us]')])" DESIRED: "dtype([('a', '<M8[D]'), ('b', '<m8[us]')])" ====================================================================== FAIL: test_complex_dtype_str (test_dtype.TestString) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/core/tests/test_dtype.py", line 353, in test_complex_dtype_str "[('a', '<m8[D]'), ('b', '<M8[us]')]") File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: "[('a', '>m8[D]'), ('b', '>M8[us]')]" DESIRED: "[('a', '<m8[D]'), ('b', '<M8[us]')]" ====================================================================== FAIL: test_kind.TestKind.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/f2py/tests/test_kind.py", line 30, in test_all 'selectedrealkind(%s): expected %r but got %r' % (i, selected_real_kind(i), selectedrealkind(i))) File "/home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: selectedrealkind(16): expected 10 but got 16 ---------------------------------------------------------------------- Ran 3693 tests in 362.416s FAILED (KNOWNFAIL=3, SKIP=6, errors=3, failures=3) Running unit tests for numpy NumPy version 1.7.0.dev-7c07089 NumPy is installed in /home/matthew/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy Python version 2.6.6 (r266:84292, Dec 26 2010, 23:29:26) [GCC 4.4.5 20100913 (prerelease)] nose version 1.1.2 Thanks, Matthew
On Tue, Mar 6, 2012 at 6:53 AM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
On Mon, Mar 5, 2012 at 8:04 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
I've pushed a bugfix to github, can you confirm that the crash goes away on your test box? Thanks for tracking that down, the stack trace was very helpful. Since x86 machines don't have as strict alignment requirements, bugs like this one will generally remain undetected until someone tests on an architecture like sparc.
Thanks - no Bus error. For your enjoyment: there were some failures and errors from numpy.test("full")
The two SPARC build slaves also stopped crashing. They show a few more failures than on Matthew's box: http://buildbot.scipy.org/builders/Linux_SPARC_64_Debian/builds/1199/steps/s... A big thanks for Stefan by the way for fixing up the buildbot so it triggers on commits again! Ralf
I'm not sure if it's been re-hashed again or not, but this is the sort of error that showed up all the time while debugging NumPy on Sparc hard-ware. The problem is that memory has to be aligned before calculations take place. With this kind of dtype, field 'f2' will be aligned on a 2-byte boundary but likely must be aligned on a 4-byte boundary before any kind of comparison operation is done. Multiple times, this bug was fixed by just looking hard at the code again and making sure that memory copies were performed prior to calculation if the data is not aligned. Thanks for the error reports and for all the hard work in tracking this down and squashing it. -Travis On Mar 5, 2012, at 1:11 PM, Matthew Brett wrote:
Hi,
On Sun, Mar 4, 2012 at 11:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Sun, Mar 4, 2012 at 10:34 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
<snip>
$ export NPY_SEPARATE_COMPILATION=1
Thanks, that did it:
9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit
Let me know if I can debug further,
That commit was a rewrite of np.concatenate, and I've traced the test function you got the crash in. The only call to concatenate is as follows:
a = np.array([True], dtype=object) np.concatenate((a,)*3) array([True, True, True], dtype=object)
Can you try this and see if it crashes?
No, that doesn't crash.
Further investigation revealed the crash to be:
(bare-env)[matthew@vagus ~]$ nosetests ~/dev_trees/numpy/numpy/lib/tests/test_io.py:TestFromTxt.test_with_masked_column_various nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] Test masked column ... Bus error
Accordingly:
In [1]: import numpy as np
In [2]: from StringIO import StringIO
In [3]: data = StringIO('True 2 3\nFalse 5 6\n')
In [4]: test = np.genfromtxt(data, dtype=None, missing_values='2,5', usemask=True)
In [6]: from numpy import ma
In [7]: control = ma.array([(1, 2, 3), (0, 5, 6)], mask=[(0, 1, 0), (0, 1, 0)], dtype=[('f0', bool), ('f1', bool), ('f2', int)])
In [8]: test == control Bus error
Another thing you can do is compile with debug information enabled, then run the crashing case in gdb. This will look something like this:
$ export CFLAGS=-g $ rm -rf build # make sure it's a fresh build from scratch $ python setup.py install --prefix=<dir> # or however you do it [... build printout] $ gdb python <script/options> [... gdb info] (gdb) run [... script runs] Program received signal SIGSEGV, Segmentation fault, [crash address] (gdb) backtrace [full backtrace printout]
Such a backtrace would be immensely helpful in tracking down why it's crashing.
I'll get back to you with this.
See you,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
The place this is used inside of setup.py is here:
https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L14
I really dislike this build feature, it repeatedly trips me up. In my opinion, the build should be changed to always do separate compilation, and the single file mode should be eradicated.
Nobody *likes* this build "feature". If it can be eradicated let's do it, but when I wanted to early on I couldn't for various reasons. I'm not convinced these reasons have been addressed. The big problem I had (if I remember correctly) was getting the NumPy C-API to be exportable on all platforms (yet having the variables be static and hidden from the shared symbol-space) if it was assembled from multiple files. The NumPy C-API uses a clever trick that was invented for Python back in the late 90s which was important for getting extensions that used the NumPy C-API to work on several platforms. If we can be assured that platforms that might have needed that are no longer needing NumPy (or there are work-arounds), then we can certainly get rid of a single-file build mode. -Travis
On Mon, Mar 5, 2012 at 1:20 AM, Travis Oliphant <travis@continuum.io> wrote:
The place this is used inside of setup.py is here:
https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L14
I really dislike this build feature, it repeatedly trips me up. In my opinion, the build should be changed to always do separate compilation, and the single file mode should be eradicated.
Nobody *likes* this build "feature". If it can be eradicated let's do it, but when I wanted to early on I couldn't for various reasons. I'm not convinced these reasons have been addressed.
This was David's contribution and wasn't put in, IIRC, until the files were broken up in 1.3. The reason it isn't standard is that it was experimental at the time.
The big problem I had (if I remember correctly) was getting the NumPy C-API to be exportable on all platforms (yet having the variables be static and hidden from the shared symbol-space) if it was assembled from multiple files. The NumPy C-API uses a clever trick that was invented for Python back in the late 90s which was important for getting extensions that used the NumPy C-API to work on several platforms.
If we can be assured that platforms that might have needed that are no longer needing NumPy (or there are work-arounds), then we can certainly get rid of a single-file build mode.
We should probably make is standard for a while to see if it causes problems. Chuck
On Sun, Mar 4, 2012 at 12:27 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp:
nosetests
~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values
test_user_missing_values (test_io.TestFromTxt) ... Bus error
This on current master : 1.7.0.dev-b9872b4
Hmm, some tests might have been recently enabled. Any chance of doing a bisection?
Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log:
The effort is much appreciated. At least we are down to a 3 day period.
* df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe]
So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad)
There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a:
gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o
build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1
Chuck
participants (5)
-
Charles R Harris -
Mark Wiebe -
Matthew Brett -
Ralf Gommers -
Travis Oliphant