Mailman 3 October 2013 - NumPy-Discussion

Re: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1
by Robert Kern 02 Oct '13

02 Oct '13

On Tue, Oct 1, 2013 at 5:02 PM, Jack Howarth <howarth(a)bromo.med.uc.edu> wrote: > > On Tue, Oct 01, 2013 at 04:52:06PM +0100, Robert Kern wrote: > > On Tue, Oct 1, 2013 at 4:41 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > > > > > > Hi, > > > > > > 01.10.2013 16:28, Jack Howarth kirjoitti: > > > [clip] > > > > /sw/bin/python2.7 setup.py build > > > > > > > > which fails at... > > > > > > > > /sw/bin/gfortran -Wall -L/sw/lib > > build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_litemodule.o > > build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o > > -L/sw/lib -L/sw/lib/gcc4.8/lib/gcc/x86_64-apple-darwin10.8.0/4.8.1 > > -Lbuild/temp.macosx-10.6-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas > > -lgfortran -o build/lib.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite.so > > > > Undefined symbols for architecture x86_64: > > > > "_main", referenced from: > > > > start in crt1.10.6.o > > > [clip] > > > > > > Something is screwed up in your build environment: the `-shared` flag is > > > missing from the link command. > > > > > > Perhaps you have set one of the the environment variables FFLAGS, > > > CFLAGS, LDFLAGS? > > > > Also the `-undefined dynamic_lookup` flag. > > The consensus of the fink developers is that you are introducing a bug in both > scipy and numpy. The build should be able to pass additional flags on these > variables and the scipy/numpy build should be able to append any additional > flags required. In particular, both MacPorts and fink will want to be able to > pass -L/opt/local/lib or -L/sw/lib via LDFLAGS. The changes added to scipy and > numpy have broken this and now require that these additional flags be manually > patched into the Makefiles of numpy and scipy rather than just passing them > on LDFLAGS as has always worked in the past. Oh no it hasn't. It has been a consistent thorn in our side for a very long time. In the case of Fortran modules built by numpy.distutils, $LDFLAGS has replaced rather than appended flags since time immemorial. It is a compromise solution to work around the fact that the wide variety of Fortran compilers are very finicky about their flags, and distutils is not very accommodating about letting users change the flags to suit their local environments. If you think you have a better solution to this problem that does not degrade the existing flexibility, your PR will be cheerfully accepted. No one thinks this is desirable behavior, but it is most certainly not *new* behavior. -- Robert Kern

1 0

Re: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1
by Robert Kern 01 Oct '13

01 Oct '13

On Tue, Oct 1, 2013 at 4:41 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > > Hi, > > 01.10.2013 16:28, Jack Howarth kirjoitti: > [clip] > > /sw/bin/python2.7 setup.py build > > > > which fails at... > > > > /sw/bin/gfortran -Wall -L/sw/lib build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/sw/lib -L/sw/lib/gcc4.8/lib/gcc/x86_64-apple-darwin10.8.0/4.8.1 -Lbuild/temp.macosx-10.6-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas -lgfortran -o build/lib.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite.so > > Undefined symbols for architecture x86_64: > > "_main", referenced from: > > start in crt1.10.6.o > [clip] > > Something is screwed up in your build environment: the `-shared` flag is > missing from the link command. > > Perhaps you have set one of the the environment variables FFLAGS, > CFLAGS, LDFLAGS? Also the `-undefined dynamic_lookup` flag. -- Robert Kern

1 0

Re: [Numpy-discussion] [pydata] Re: [SciPy-Dev] 1.8.0rc1
by Nathaniel Smith 01 Oct '13

01 Oct '13

I bet the difference is: In master, nansum ultimately calls arr.sum(...), which will be intercepted by Series.sum. In 1.8.x, nansum ultimately calls np.add.reduce(...), which can't be intercepted and will return the wrong thing. AFAICT the np.add.reduce(a, ...) call could just be replaced with a.sum(...), but I might be missing something... surely there must have been some reason it was written that way in the first place? On Tue, Oct 1, 2013 at 2:48 PM, Jeff <jeffreback(a)gmail.com> wrote: > so...looks like a backport issue? > > > On Tuesday, October 1, 2013 6:41:26 AM UTC-4, Nathaniel Smith wrote: >> >> On Tue, Oct 1, 2013 at 3:27 AM, Charles R Harris >> <charles...(a)gmail.com> wrote: >> >>> >> On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke <cgo...(a)uci.edu >> >>> >> <mailto:cgo...@uci.edu>> wrote: >> >>> >> > NumPy 1.8.0rc1 looks good. All tests pass on Windows and >> >>> >> most >> >>> >> 3rd party >> >>> >> > packages test OK now. Thank you. >> >>> >> > >> >>> >> > A few tests still fail in the following packages when >> >>> >> run >> >>> >> with >> >>> >> > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to >> >>> >> > numpy-MKL-1.7.1-win-amd64-py3.3: >> >>> >> > >> >>> >> > 1) Pandas 0.12.0 >> >>> >> > >> >>> >> > ``` >> >>> >> > >> >>> >> >> >>> >> >> >>> >> ====================================================================== >> >>> >> > FAIL: test_nansum_buglet >> >>> >> (pandas.tests.test_series.TestNanops) >> >>> >> > >> >>> >> >> >>> >> >> >>> >> ---------------------------------------------------------------------- >> >>> >> > Traceback (most recent call last): >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\tests\test_series.py", >> >>> >> > line 254, in test_nansum_buglet >> >>> >> > assert_almost_equal(result, 1) >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\util\testing.py", line >> >>> >> > 134, in assert_almost_equal >> >>> >> > np.testing.assert_(isiterable(b)) >> >>> >> > File >> >>> >> "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line >> >>> >> > 44, in assert_ >> >>> >> > raise AssertionError(msg) >> >>> >> > AssertionError >> >>> >> > ``` >> >>> >> > >> >>> >> > Possibly related: >> >>> >> > >> >>> >> > ``` >> >>> >> > >>> import numpy as np >> >>> >> > >>> from pandas import Series >> >>> >> > >>> s = Series([0.0]) >> >>> >> > >>> result = np.nansum(s) >> >>> >> > >>> print(result) >> >>> >> > Traceback (most recent call last): >> >>> >> > File "<stdin>", line 1, in <module> >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\core\base.py", line >> >>> >> > 19, in >> >>> >> > __str__ >> >>> >> > return self.__unicode__() >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\core\series.py", line >> >>> >> > 1115, in __unicode__ >> >>> >> > length=len(self) > 50, >> >>> >> > TypeError: len() of unsized object >> >>> >> > ``` >> [...] >> > >> > The pandas test passes for current pandas dev, so it looks like a bug on >> > their end that has been taken care of. >> > >> > test_nansum_buglet (__main__.TestNanops) ... ok >> >> I'm concerned about this. 0.12.0 is currently the latest pandas >> release, so even if it is a bug on their side, we're going to be >> converting it from a latent bug to a real bug when we release... >> CC'ing pydata, do you guys have any insight into what changed here? >> >> The code is: >> s = pandas.Series([1.0, np.nan]) >> result = np.nansum(s) >> With numpy 1.7.1, 'result' comes out as a np.float64. With numpy >> maintenance/1.8.x, 'result' comes out as a 0-d Series object. Series >> is a subclass of ndarray, but it's supposed to always be 1-d, so all >> kinds of stuff blows up as soon as you have a 0-d Series object. >> >> I'm not sure what changed in numpy's nansum; if I try this same test >> with a simple no-op ndarray subclass: >> class MyArray(np.ndarray): >> pass >> np.nansum(np.array([1.0, np.nan]).view(MyArray)) >> then 1.7.1 and maintenance/1.8.x both act the same, and both return a >> 0-d MyArray, so it's not just a question of whether we remembered to >> handle subclasses at all. >> >> -n > > -- > You received this message because you are subscribed to the Google Groups > "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pydata+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

2 2

Re: [Numpy-discussion] [pydata] Re: [SciPy-Dev] 1.8.0rc1
by Nathaniel Smith 01 Oct '13

01 Oct '13

On Tue, Oct 1, 2013 at 2:39 PM, Jeff <jeffreback(a)gmail.com> wrote: > Just built from numpy master looks ok > > In [7]: pd.__version__ > Out[7]: '0.12.0' > > In [8]: np.__version__ > Out[8]: '1.9.0.dev-74abfa2' > > In [9]: s = pandas.Series([1.0, np.nan]) > > In [10]: np.nansum(s) > Out[10]: 1.0 > > In [11]: type(np.nansum(s)) > Out[11]: numpy.float64 Yeah, I can confirm this... With pandas 0.12.0, this code: np.nansum(pandas.Series([1.0, np.nan])) works correctly using current master (74abfa27), but is broken with current maintenance/1.8.x (f5fe91b9). -n > On Tuesday, October 1, 2013 6:41:26 AM UTC-4, Nathaniel Smith wrote: >> >> On Tue, Oct 1, 2013 at 3:27 AM, Charles R Harris >> <charles...(a)gmail.com> wrote: >> >>> >> On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke <cgo...(a)uci.edu >> >>> >> <mailto:cgo...@uci.edu>> wrote: >> >>> >> > NumPy 1.8.0rc1 looks good. All tests pass on Windows and >> >>> >> most >> >>> >> 3rd party >> >>> >> > packages test OK now. Thank you. >> >>> >> > >> >>> >> > A few tests still fail in the following packages when >> >>> >> run >> >>> >> with >> >>> >> > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to >> >>> >> > numpy-MKL-1.7.1-win-amd64-py3.3: >> >>> >> > >> >>> >> > 1) Pandas 0.12.0 >> >>> >> > >> >>> >> > ``` >> >>> >> > >> >>> >> >> >>> >> >> >>> >> ====================================================================== >> >>> >> > FAIL: test_nansum_buglet >> >>> >> (pandas.tests.test_series.TestNanops) >> >>> >> > >> >>> >> >> >>> >> >> >>> >> ---------------------------------------------------------------------- >> >>> >> > Traceback (most recent call last): >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\tests\test_series.py", >> >>> >> > line 254, in test_nansum_buglet >> >>> >> > assert_almost_equal(result, 1) >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\util\testing.py", line >> >>> >> > 134, in assert_almost_equal >> >>> >> > np.testing.assert_(isiterable(b)) >> >>> >> > File >> >>> >> "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line >> >>> >> > 44, in assert_ >> >>> >> > raise AssertionError(msg) >> >>> >> > AssertionError >> >>> >> > ``` >> >>> >> > >> >>> >> > Possibly related: >> >>> >> > >> >>> >> > ``` >> >>> >> > >>> import numpy as np >> >>> >> > >>> from pandas import Series >> >>> >> > >>> s = Series([0.0]) >> >>> >> > >>> result = np.nansum(s) >> >>> >> > >>> print(result) >> >>> >> > Traceback (most recent call last): >> >>> >> > File "<stdin>", line 1, in <module> >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\core\base.py", line >> >>> >> > 19, in >> >>> >> > __str__ >> >>> >> > return self.__unicode__() >> >>> >> > File >> >>> >> "X:\Python33\lib\site-packages\pandas\core\series.py", line >> >>> >> > 1115, in __unicode__ >> >>> >> > length=len(self) > 50, >> >>> >> > TypeError: len() of unsized object >> >>> >> > ``` >> [...] >> > >> > The pandas test passes for current pandas dev, so it looks like a bug on >> > their end that has been taken care of. >> > >> > test_nansum_buglet (__main__.TestNanops) ... ok >> >> I'm concerned about this. 0.12.0 is currently the latest pandas >> release, so even if it is a bug on their side, we're going to be >> converting it from a latent bug to a real bug when we release... >> CC'ing pydata, do you guys have any insight into what changed here? >> >> The code is: >> s = pandas.Series([1.0, np.nan]) >> result = np.nansum(s) >> With numpy 1.7.1, 'result' comes out as a np.float64. With numpy >> maintenance/1.8.x, 'result' comes out as a 0-d Series object. Series >> is a subclass of ndarray, but it's supposed to always be 1-d, so all >> kinds of stuff blows up as soon as you have a 0-d Series object. >> >> I'm not sure what changed in numpy's nansum; if I try this same test >> with a simple no-op ndarray subclass: >> class MyArray(np.ndarray): >> pass >> np.nansum(np.array([1.0, np.nan]).view(MyArray)) >> then 1.7.1 and maintenance/1.8.x both act the same, and both return a >> 0-d MyArray, so it's not just a question of whether we remembered to >> handle subclasses at all. >> >> -n > > -- > You received this message because you are subscribed to the Google Groups > "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pydata+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

1 0

review/approval request
by Graeme B. Bell 01 Oct '13

01 Oct '13

Regarding 1.8: A few months ago I submitted a patch that fixes some bugs, including a major bug in select() that has been around for years, and optimised the code quite a bit. https://github.com/numpy/numpy/pull/3537 It would be nice to see the patch in 1.8, but it has languished for a couple of months waiting for final approval/merging. The patch is pretty small and the new code is not complicated. Are there any core devs that would have time/interest to help me get this approved/merged into numpy master? I have already adjusted the patch as suggested by njsmith to suit inclusion in master. Graeme. >>> 331701 >> >> I can't tell if these are real bugs in numpy, or tests checking that >> bottleneck is bug-for-bug compatible with old numpy and we just fixed >> some bugs, or what. It's clearly something to do with the >> nanarg{max,min} rewrite -- @charris, do you know what's going on here? >> >> > Yes ;) The previous behaviour of nanarg for all-nan axis was to cast nan to > intp when the result was an array, and return nan when a scalar. The > current behaviour is to return the most negative value of intp as an error > marker in both cases and raise a warning. It is a change in behavior, but I > think one that needs to be made. > > Chuck >

1 0

Issue with np.median and array subclasses in 1.8.0rc (worked with 1.7.0)
by Thomas Robitaille 01 Oct '13

01 Oct '13

Hi, The behavior for ``np.median`` and array sub-classes has changed in 1.8.0rc, which breaks unit-handling code (such as the ``quantities`` package, or ``astropy.units``): https://github.com/numpy/numpy/issues/3846 This previously worked from Numpy 1.5 (at least) to Numpy 1.7. Is there a new (and better) way to override the ``np.median`` behavior? Cheers, Tom

1 0