Mailman 3 July 2014 - NumPy-Discussion

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by Eelco Hoogendoorn 25 Jul '14

25 Jul '14

Ray: I'm not working with Hubble data, but yeah these are all issues I've run into with my terrabytes of microscopy data as well. Given that such raw data comes as uint16, its best to do your calculations as much as possible in good old ints. What you compute is what you get, no obscure shenanigans. It just occurred to me that pairwise summation will lead to highly branchy code, and you can forget about any vector extensions. Tradeoffs indeed. Any such hierarchical summation is probably best done by aggregating naively summed blocks. -----Original Message----- From: "RayS" <rays(a)blue-cove.com> Sent: ‎25-‎7-‎2014 23:26 To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays At 11:29 AM 7/25/2014, you wrote: >On Fri, Jul 25, 2014 at 5:56 PM, RayS <rays(a)blue-cove.com> wrote: > > The important point was that it would be best if all of the > methods affected > > by summing 32 bit floats with 32 bit accumulators had the same Notes as > > numpy.mean(). We went through a lot of code yesterday, assuming that any > > numpy or Scipy.stats functions that use accumulators suffer the same issue, > > whether noted or not, and found it true. > >Do you have a list of the functions that are affected? We only tested a few we used, but scipy.stats.nanmean, or any .*mean() numpy.sum, mean, average, std, var,... via something like: import numpy import scipy.stats print numpy.__version__ print scipy.__version__ onez = numpy.ones((2**25, 1), numpy.float32) step = 2**10 func = scipy.stats.nanmean for s in range(2**24-step, 2**25, step): if func(onez[:s+step])!=1.: print '\nbroke', s, func(onez[:s+step]) break else: print '\r',s, > That said, it does seem that np.mean could be implemented better than >it is, even given float32's inherent limitations. If anyone wants to >implement better algorithms for computing the mean, variance, sums, >etc., then we would love to add them to numpy. Others have pointed out the possible tradeoffs in summation algos, perhaps a method arg would be appropriate, "better" depending on your desire for speed vs. accuracy. It just occurred to me that if the STSI folks (who count photons) took the mean() or other such func of an image array from Hubble sensors to find background value, they'd better always be using float64. - Ray _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 25 Jul '14

25 Jul '14

At 11:29 AM 7/25/2014, you wrote: >On Fri, Jul 25, 2014 at 5:56 PM, RayS <rays(a)blue-cove.com> wrote: > > The important point was that it would be best if all of the > methods affected > > by summing 32 bit floats with 32 bit accumulators had the same Notes as > > numpy.mean(). We went through a lot of code yesterday, assuming that any > > numpy or Scipy.stats functions that use accumulators suffer the same issue, > > whether noted or not, and found it true. > >Do you have a list of the functions that are affected? We only tested a few we used, but scipy.stats.nanmean, or any .*mean() numpy.sum, mean, average, std, var,... via something like: import numpy import scipy.stats print numpy.__version__ print scipy.__version__ onez = numpy.ones((2**25, 1), numpy.float32) step = 2**10 func = scipy.stats.nanmean for s in range(2**24-step, 2**25, step): if func(onez[:s+step])!=1.: print '\nbroke', s, func(onez[:s+step]) break else: print '\r',s, > That said, it does seem that np.mean could be implemented better than >it is, even given float32's inherent limitations. If anyone wants to >implement better algorithms for computing the mean, variance, sums, >etc., then we would love to add them to numpy. Others have pointed out the possible tradeoffs in summation algos, perhaps a method arg would be appropriate, "better" depending on your desire for speed vs. accuracy. It just occurred to me that if the STSI folks (who count photons) took the mean() or other such func of an image array from Hubble sensors to find background value, they'd better always be using float64. - Ray

2 1

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 25 Jul '14

25 Jul '14

At 07:22 AM 7/25/2014, you wrote: > > We were talking on this in the office, as we > > realized it does affect a couple of lines dealing > > with large arrays, including complex64. > > As I expect Python modules to work uniformly > > cross platform unless documented otherwise, to me > > that includes 32 vs 64 bit platforms, implying > > that the modules should automatically use large > > enough accumulators for the data type input. > >The 32/64-bitness of your platform has nothing to do with floating >point. As a naive end user, I can, and do, download different binaries for different CPUs/Windows versions and will get different results http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070747.html > Nothing discussed in this thread is platform-specific (modulo >some minor details about the hardware FPU, but that should be taken as >read). And compilers, apparently. The important point was that it would be best if all of the methods affected by summing 32 bit floats with 32 bit accumulators had the same Notes as numpy.mean(). We went through a lot of code yesterday, assuming that any numpy or Scipy.stats functions that use accumulators suffer the same issue, whether noted or not, and found it true. "Depending on the input data, this can cause the results to be inaccurate, especially for float32 (see example below). Specifying a higher-precision accumulator using the <http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html#numpy.…>dtype keyword can alleviate this issue." seems rather un-Pythonic. - Ray

2 1

ANN: Pandas 0.14.0 Release Candidate 1
by Jeff Reback 25 Jul '14

25 Jul '14

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.14.0. Please try this RC and report any issues here: Pandas Issues<https://github.com/pydata/pandas/issues> We will be releasing officially in about 2 weeks or so. This is a major release from 0.13.1 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. Highlights include: - Officially support Python 3.4 - SQL interfaces updated to use sqlalchemy, - Display interface changes - MultiIndexing Using Slicers - Ability to join a singly-indexed DataFrame with a multi-indexed DataFrame - More consistency in groupby results and more flexible groupby specifications - Holiday calendars are now supported in CustomBusinessDay - Several improvements in plotting functions, including: hexbin, area and pie plots. - Performance doc section on I/O operations Since there are some significant changes in the default way DataFrames are displayed. I have put up a comment issue looking for some feedback here<https://github.com/pydata/pandas/issues/7146> Here are the full whatsnew and documentation links: v0.14.0 Whatsnew<http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html> v0.14.0 Documentation Page<http://pandas-docs.github.io/pandas-docs-travis/> Source tarballs, and windows builds are available here: Pandas v0.14rc1 Release <https://github.com/pydata/pandas/releases> A big thank you to everyone who contributed to this release! Jeff

4 5

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 25 Jul '14

25 Jul '14

At 01:22 AM 7/25/2014, you wrote: > Actually the maximum precision I am not so > sure of, as I personally prefer to make an > informed decision about precision used, and get > an error on a platform that does not support > the specified precision, rather than obtain > subtly or horribly broken results without > warningÂ when moving your code to a different platform/compiler whatever. We were talking on this in the office, as we realized it does affect a couple of lines dealing with large arrays, including complex64. As I expect Python modules to work uniformly cross platform unless documented otherwise, to me that includes 32 vs 64 bit platforms, implying that the modules should automatically use large enough accumulators for the data type input. http://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html does mention inaccuracy. http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.msta… http://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html etc do not, exactly - Ray

2 1

Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
by Lars Buitinck 25 Jul '14

25 Jul '14

Wed, 23 Jul 2014 22:13:33 +0100 Nathaniel Smith <njs(a)pobox.com>: > On Wed, Jul 23, 2014 at 9:57 PM, Robert Kern <robert.kern(a)gmail.com> wrote: >> That's perhaps what you want, but numpy has never claimed to do this. ... except in np.where, which promises to return indices but actually returns arrays of longs and thus doesn't work with large arrays on Windows. I know this is a bug that can be fixed without changing the size of np.int, but it goes to show that even core functionality in NumPy gets it wrong. > This is true, but it's not very compelling on its own -- "big as a > pointer" is a much much more useful property than "big as a long". The > only real reason this made sense in the first place is the equivalence > between Python int and C long, but even that is gone now with Python > 3. IMO at this point backcompat is really the only serious reason for > keeping int32 as the default integer type in win64. But of course this > is a pretty serious concern... Hear, hear. The C type long is only useful as an "at least 32-bit" integer, but on the platforms that NumPy targets, int is also at least that large. The only real benefit of long is that it makes porting more interesting </sarcasm>. If you have intp and a bunch of explicitly-sized integer types, you don't need an additional type that behaves like a long *except* for backward compat. The Go people got this right; they only have explicitly-sized integer types and an int type the size of a pointer [1]. [1] http://golang.org/doc/go1.1#int

3 2

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by Eelco Hoogendoorn 25 Jul '14

25 Jul '14

Perhaps it is a slightly semantical discussion; but all fp calculations have errors, and there are always strategies for making them smaller. We just don't happen to like the error for this case; but rest assured it won't be hard to find new cases of 'blatantly wrong' results, no matter what accumulator is implemented. That's no reason to not try and be clever about it, but there isn't going to be an algorithm that is best for all possible inputs, and in the end the most important thing is that the algorithm used is specified in the docs. -----Original Message----- From: "Alan G Isaac" <alan.isaac(a)gmail.com> Sent: ‎25-‎7-‎2014 00:10 To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays On 7/24/2014 4:42 PM, Eelco Hoogendoorn wrote: > This isn't a bug report, but rather a feature request. I'm not sure statement this is correct. The mean of a float32 array can certainly be computed as a float32. Currently this is not necessarily what happens, not even approximately. That feels a lot like a bug, even if we can readily understand how the algorithm currently used produces it. To say whether it is a bug or not, don't we have to ask about the intent of `mean`? If the intent is to sum and divide, then it is not a bug. If the intent is to produce the mean, then it is a bug. Alan Isaac _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

1 1

Re: [Numpy-discussion] numpy.mean still broken for large float32arrays
by Eelco Hoogendoorn 24 Jul '14

24 Jul '14

Inaccurate and utterly wrong are subjective. If You want To Be sufficiently strict, floating point calculations are almost always 'utterly wrong'. Granted, It would Be Nice if the docs specified the algorithm used. But numpy does not produce anything different than what a standard c loop or c++ std lib func would. This isn't a bug report, but rather a feature request. That said, support for fancy reduction algorithms would certainly be nice, if implementing it in numpy in a coherent manner is feasible. -----Original Message----- From: "Joseph Martinot-Lagarde" <joseph.martinot-lagarde(a)m4x.org> Sent: ‎24-‎7-‎2014 20:04 To: "numpy-discussion(a)scipy.org" <numpy-discussion(a)scipy.org> Subject: Re: [Numpy-discussion] numpy.mean still broken for large float32arrays Le 24/07/2014 12:55, Thomas Unterthiner a écrit : > I don't agree. The problem is that I expect `mean` to do something > reasonable. The documentation mentions that the results can be > "inaccurate", which is a huge understatement: the results can be utterly > wrong. That is not reasonable. At the very least, a warning should be > issued in cases where the dtype might not be appropriate. > Maybe the problem is the documentation, then. If this is a common error, it could be explicitly documented in the function documentation. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

Re: [Numpy-discussion] numpy.mean still broken for large float32 arrays
by RayS 24 Jul '14

24 Jul '14

Probably a number of scipy places as well import numpy import scipy.stats print numpy.__version__ print scipy.__version__ for s in range(16777214, 16777944): if scipy.stats.nanmean(numpy.ones((s, 1), numpy.float32))[0]!=1: print '\nbroke', s, scipy.stats.nanmean(numpy.ones((s, 1), numpy.float32)) break else: print '\r',s, c:\temp>python np_sum.py 1.8.0b2 0.11.0 16777216 broke 16777217 [ 0.99999994]

2 1

masked_where broadcasting?
by Benjamin Root 24 Jul '14

24 Jul '14

I ran into this this morning while writing up a new test for matplotlib. Shouldn't these two arrays be broadcasted automatically or maybe np.ma is being overly cautious? u = np.ma.masked_where((-0.4 < x) & (x < 0.1), u, copy=False) File "/home/ben/.local/lib/python2.7/site-packages/numpy/ma/core.py", line 1806, in masked_where " (got %s and %s)" % (cshape, ashape)) IndexError: Inconsistant shape between the condition and the input (got (10, 1, 1) and (10, 10, 3)) x has shape (10, 1, 1) and u has shape (10, 10, 3). This is on a recent-ish numpy master. Cheers! Ben Root

1 0