NumPy-Discussion October 2013

numpy-discussion@python.org

78 participants
74 discussions

Behavior of nan{max, min} and nanarg{max, min} for all-nan slices.
by Charles R Harris 04 Oct '13

04 Oct '13

Hi All, The question is what to do when all-nan slices are encountered in the nan{max,min} and nanarg{max, min} functions. Currently in 1.8.0, the first returns nan and raises a warning, the second returns intp.min and raises a warning. It is proposed that the nanarg{max, min} functions, and possibly the nan{max, min} also, raise an error instead. Raising errors would be consistent with the behavior of the arg{max, min} and amax/amin functions when they encounter empty arrays. OTOH, now that we no longer support Python 2.4/2.5 the catch_warnings context manager can serve the same purpose by changing the warnings into exceptions. So, what to do? Thoughts? Chuck

7 25

subsampling arrays without loops
by Moroney, Catherine M (398D) 04 Oct '13

04 Oct '13

I know I have a lot yet to learn about array striding tricks, so please pardon the triviality of this question. Here is the problem both in words and "dumb" python: I have a large NxM array that I want to break down into smaller nxn chunks where n divides evenly into both N and M. Then I want to calculate the fraction of pixels in each nxn chunk that meets a certain criteria: say (x > 1) & (x < 2). Here is the "dumb" python code: npix = 4 num_true = numpy.zeros((data.shape[0]/npix, data.shape[1]/npix)) for iline in xrange(0, data.shape[0]/npix): for ismp in xrange(0, data.shape[1]/npix): excerpt = data[iline*npix:(iline+1)*npix, ismp*npix:(ismp+1)*npix] num_true[iline,simp] = numpy.where( (excerpt >= 1) & (excerpt <= 2), True, False).sum() so I'm looping through the number of 4x4 subsets in both dimensions, cutting out a chunk of the data ("excerpt"), and then counting the number of pixels in that excerpt that meet a certain criteria and storing that result for each excerpt. I want to avoid all the loops over iline and ismp. What's the best way of doing this in pure python? I could always write a Fortran/C routine for this task, but I want to learn how best to do it with numpy. Thank you for any advice, Catherine

2 1

Products of small float32 values.
by Dave Cook 03 Oct '13

03 Oct '13

Can someone explain what is going on here? In [153]: small = ones(1, dtype='float32') In [154]: small Out[154]: array([ 1.], dtype=float32) In [155]: small*1e-45 Out[155]: array([ 1.40129846e-45], dtype=float32) In [156]: small*1e-46 Out[156]: array([ 0.], dtype=float32) I would expect float32 to be able to represent numbers with exponents as small as -127. Thanks, Dave Cook

2 2

1.8.0, nan functions, and pandas
by Charles R Harris 03 Oct '13

03 Oct '13

Hi All, There was a problem with pandas 0.12 and the numpy nan functions when applying the nansum function to a pandas Series object. We thought we had fixed it by the slight of hand use of `a.sum` instead of `add.reduce(a)` so that the Series object could use its own version of `sum` which overrides the ndarray version. However, the pandas sum method has a different signature, so success was illusory and due to not naming the keyword arguments in the call. The new nan functions nanmean, nanvar, and nanstd also failed, as the actually do name the keyword arguments. For pandas development all the functions work as the pandas sum method now include a `**kwargs` which gobbles up the missing keyword `keepdims`. At this point I don't think there is anything we can legitimately do to fix the pandas 0.12 problem. I propose to name all the keyword arguments arguments so that pandas fails, as it should, and simply document that the nan functions will not work with pandas versions < 0.13. Thoughts? Chuck

1 1

1.8.0rc1
by Charles R Harris 02 Oct '13

02 Oct '13

Hi All, NumPy 1.8.0rc1 is up now on sourceforge<http://sourceforge.net/projects/numpy/files/NumPy/1.8.0rc1/>.The binary builds are included except for Python 3.3 on windows, which will arrive later. Many thanks to Ralf for the binaries, and to those who found and fixed the bugs in the last beta. Any remaining bugs are all my fault ;) I hope this will be the last release before final, so please test it thoroughly. Chuck

10 31

MATLAB fsolve now available in Python
by Dmitrey 02 Oct '13

02 Oct '13

Hi all, New solver for systems of nonlinear equations ( SNLE ) has been connected to free Python framework OpenOpt: fsolve from MATLAB Optimization Toolbox; uploaded into PYPI in v. 0.5112. As well as fmincon , currently it's available for Python 2 only. Unlike scipy.optimize fsolve, it can handle sparse derivatives, user-supplied or from FuncDesigner automatic differentiation. See the example with 15000 equations. To keep discussion in a single place please use the OpenOpt forum thread. Regards, D.

1 0

nanargmax failure case (was: Re: [SciPy-Dev] 1.8.0rc1)
by Nathaniel Smith 01 Oct '13

01 Oct '13

[switching subject to break out from the giant 1.8.0rc1 thread] On Tue, Oct 1, 2013 at 2:52 PM, Charles R Harris <charlesr.harris(a)gmail.com> wrote: > > > > On Tue, Oct 1, 2013 at 7:25 AM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> On Tue, Oct 1, 2013 at 1:56 PM, Charles R Harris >> <charlesr.harris(a)gmail.com> wrote: >> > On Tue, Oct 1, 2013 at 4:43 AM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> >> >> On Mon, Sep 30, 2013 at 10:51 PM, Christoph Gohlke <cgohlke(a)uci.edu> >> >> wrote: >> >> > 2) Bottleneck 0.7.0 >> >> > >> >> > >> >> > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 >> >> >> >> I can't tell if these are real bugs in numpy, or tests checking that >> >> bottleneck is bug-for-bug compatible with old numpy and we just fixed >> >> some bugs, or what. It's clearly something to do with the >> >> nanarg{max,min} rewrite -- @charris, do you know what's going on here? >> >> >> > >> > Yes ;) The previous behaviour of nanarg for all-nan axis was to cast nan >> > to >> > intp when the result was an array, and return nan when a scalar. The >> > current >> > behaviour is to return the most negative value of intp as an error >> > marker in >> > both cases and raise a warning. It is a change in behavior, but I think >> > one >> > that needs to be made. >> >> Ah, okay! I kind of lost track of the nanfunc changes by the end there. >> >> So for the bottleneck issue, it sounds like the problem is just that >> bottleneck is still emulating the old numpy behaviour in this corner >> case, which isn't really a problem. So we don't really need to worry >> about that, both behaviours are correct, just maybe out of sync. >> >> I'm a little dubious about this "make up some weird value that will >> *probably* blow up if people try to use it without checking, and also >> raise a warning" thing, wouldn't it make more sense to just raise an >> error? That's what exceptions are for? I guess I should have said >> something earlier though... >> > > I figure the blowup is safe, as we can't allocate arrays big enough that the > minimum intp value would be a valid index. I considered raising an error, > and if there is a consensus the behavior could be changed. Or we could add a > keyword to determine the behavior. Yeah, the intp value can't be a valid index, so that covers 95% of cases, but I'm worried about that other 5%. It could still pass silently as the endpoint of a slice, or participate in some sort of integer arithmetic calculation, etc. I assume you also share this worry to some extent or you wouldn't have put in the warning ;-). I guess the bigger question is, why would we *not* use the standard method for signaling an exceptional condition here, i.e., exceptions? That way we're 100% guaranteed that if people aren't prepared to handle it then they'll at least know something has gone wrong, and if they are prepared to handle it then it's very easy and standard, just use try/except. Right now I guess you have to check for the special value, but also do something to silence warnings, but just for that one line? Sounds kind of complicated... -n

3 9

Masked arrays: Rationale for "False convention"
by Ondřej Čertík 01 Oct '13

01 Oct '13

Hi, What is the rationale for using False in 'mask' for elements that should be included? http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html As opposed to using True for elements that should be included, which is what I was intuitively expecting when I started using the masked arrays. This "True convention" also happens to be the one used in Fortran, see e.g.: http://gcc.gnu.org/onlinedocs/gfortran/SUM.html So it's confusing why NumPy would chose a "False convention". Could it be, that NumPy views 'mask' as opacity? Then it would make sense to use True to make a value 'opaque'. Ondrej

5 7

Re: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1
by Robert Kern 01 Oct '13

01 Oct '13

On Tue, Oct 1, 2013 at 5:02 PM, Jack Howarth <howarth(a)bromo.med.uc.edu> wrote: > > On Tue, Oct 01, 2013 at 04:52:06PM +0100, Robert Kern wrote: > > On Tue, Oct 1, 2013 at 4:41 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > > > > > > Hi, > > > > > > 01.10.2013 16:28, Jack Howarth kirjoitti: > > > [clip] > > > > /sw/bin/python2.7 setup.py build > > > > > > > > which fails at... > > > > > > > > /sw/bin/gfortran -Wall -L/sw/lib > > build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_litemodule.o > > build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o > > -L/sw/lib -L/sw/lib/gcc4.8/lib/gcc/x86_64-apple-darwin10.8.0/4.8.1 > > -Lbuild/temp.macosx-10.6-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas > > -lgfortran -o build/lib.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite.so > > > > Undefined symbols for architecture x86_64: > > > > "_main", referenced from: > > > > start in crt1.10.6.o > > > [clip] > > > > > > Something is screwed up in your build environment: the `-shared` flag is > > > missing from the link command. > > > > > > Perhaps you have set one of the the environment variables FFLAGS, > > > CFLAGS, LDFLAGS? > > > > Also the `-undefined dynamic_lookup` flag. > > The consensus of the fink developers is that you are introducing a bug in both > scipy and numpy. The build should be able to pass additional flags on these > variables and the scipy/numpy build should be able to append any additional > flags required. In particular, both MacPorts and fink will want to be able to > pass -L/opt/local/lib or -L/sw/lib via LDFLAGS. The changes added to scipy and > numpy have broken this and now require that these additional flags be manually > patched into the Makefiles of numpy and scipy rather than just passing them > on LDFLAGS as has always worked in the past. Oh no it hasn't. It has been a consistent thorn in our side for a very long time. In the case of Fortran modules built by numpy.distutils, $LDFLAGS has replaced rather than appended flags since time immemorial. It is a compromise solution to work around the fact that the wide variety of Fortran compilers are very finicky about their flags, and distutils is not very accommodating about letting users change the flags to suit their local environments. If you think you have a better solution to this problem that does not degrade the existing flexibility, your PR will be cheerfully accepted. No one thinks this is desirable behavior, but it is most certainly not *new* behavior. -- Robert Kern

1 0

Re: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1
by Robert Kern 01 Oct '13

01 Oct '13

On Tue, Oct 1, 2013 at 4:41 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > > Hi, > > 01.10.2013 16:28, Jack Howarth kirjoitti: > [clip] > > /sw/bin/python2.7 setup.py build > > > > which fails at... > > > > /sw/bin/gfortran -Wall -L/sw/lib build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/sw/lib -L/sw/lib/gcc4.8/lib/gcc/x86_64-apple-darwin10.8.0/4.8.1 -Lbuild/temp.macosx-10.6-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas -lgfortran -o build/lib.macosx-10.6-x86_64-2.7/numpy/linalg/lapack_lite.so > > Undefined symbols for architecture x86_64: > > "_main", referenced from: > > start in crt1.10.6.o > [clip] > > Something is screwed up in your build environment: the `-shared` flag is > missing from the link command. > > Perhaps you have set one of the the environment variables FFLAGS, > CFLAGS, LDFLAGS? Also the `-undefined dynamic_lookup` flag. -- Robert Kern

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion October 2013