Mailman 3 July 2014 - NumPy-Discussion

ANN: bcolz 0.7.1 released
by Francesc Alted 30 Jul '14

30 Jul '14

====================== Announcing bcolz 0.7.1 ====================== What's new ========== This is maintenance release, where bcolz got rid of the nose dependency for Python 2.6 (only unittest2 should be required). Also, some small fixes for the test suite, specially in 32-bit has been done. Thanks to Ilan Schnell for pointing out the problems and for suggesting fixes. ``bcolz`` is a renaming of the ``carray`` project. The new goals for the project are to create simple, yet flexible compressed containers, that can live either on-disk or in-memory, and with some high-performance iterators (like `iter()`, `where()`) for querying them. Together, bcolz and the Blosc compressor, are finally fullfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… For more detailed info, see the release notes in: https://github.com/Blosc/bcolz/wiki/Release-Notes What it is ========== bcolz provides columnar and compressed data containers. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Installing ========== bcolz is in the PyPI repository, so installing it is easy: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt ---- **Enjoy data!** -- Francesc Alted

1 0

length - sticks algorithm
by Josè Luis Mietta 29 Jul '14

29 Jul '14

Hi experts! Im working with conductivity of sticks film - systems. In my algorithm (N sticks) I have the intersection graph matrix M (M is a NxN matrix, M_ij=1 if sticks 'i' and 'j' do intersect, and M_ij=0 if sticks 'i' and 'j' do not). Also I have 2 lists with the end-points of each stick. In addition, I can calculate the intersection point (If exist) between sticks. I want to calculate all the distances between the points of intersection (1,2,3,...N) in the next figure: without lose the connectivity information (which intersection is connected to which). In the figure, (A) is the system with sticks. I dont know how to do this. Im a python + numpy user. Waiting for your answers! Thans a lot

3 4

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 29 Jul '14

29 Jul '14

At 02:04 AM 7/27/2014, you wrote: >You won't be able to do it by accident or omission or a lack of >discipline. It's not a tempting public target like, say, np.seterr(). BTW, why not throw an overflow error in the large float32 sum() case? Is it too expensive to check while accumulating? - Ray

12 17

64-bit windows numpy / scipy wheels for testing
by Matthew Brett 28 Jul '14

28 Jul '14

Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_am… https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_a… You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-s… Many thanks for Carl in particular for doing all the hard work, Cheers, Matthew

13 72

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 27 Jul '14

27 Jul '14

Thanks for the clarification, but how is the numpy rounding directed? Round to nearest, ties to even? http://en.wikipedia.org/wiki/IEEE_floating_point#Rounding_rules Just curious, as I couldn't find a reference. - Ray At 07:44 AM 7/27/2014, you wrote: >On Sun, Jul 27, 2014 at 3:16 PM, RayS <rays(a)blue-cove.com> wrote: > > At 02:04 AM 7/27/2014, you wrote: > > > >>You won't be able to do it by accident or omission or a lack of > >>discipline. It's not a tempting public target like, say, np.seterr(). > > > > BTW, why not throw an overflow error in the large float32 sum() case? > > Is it too expensive to check while accumulating? > >In the example that started this thread, there's no overflow (in the >technical sense) occurring. Overflow for ints means wrapping around, >and for floats it means exceeding the maximum possible value and >overflowing to infinity. > >The problem here is that when summing up the values, the sum gets >large enough that after rounding, x + 1 = x and the sum stops >increasing. (For float32's all this requires is x > 16777216.) So >while the final error is massive, the mechanism is just ordinary >floating-point round-off error. > >-n > >-- >Nathaniel J. Smith >Postdoctoral researcher - Informatics - University of Edinburgh >http://vorpus.org >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion(a)scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by Eelco Hoogendoorn 27 Jul '14

27 Jul '14

Arguably, the whole of floating point numbers and their related shenanigans is not very pythonic in the first place. The accuracy of the output WILL depend on the input, to some degree or another. At the risk of repeating myself: explicit is better than implicit -----Original Message----- From: "RayS" <rays(a)blue-cove.com> Sent: ‎25-‎7-‎2014 19:56 To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays At 07:22 AM 7/25/2014, you wrote: > We were talking on this in the office, as we > realized it does affect a couple of lines dealing > with large arrays, including complex64. > As I expect Python modules to work uniformly > cross platform unless documented otherwise, to me > that includes 32 vs 64 bit platforms, implying > that the modules should automatically use large > enough accumulators for the data type input. The 32/64-bitness of your platform has nothing to do with floating point. As a naive end user, I can, and do, download different binaries for different CPUs/Windows versions and will get different results http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070747.html Nothing discussed in this thread is platform-specific (modulo some minor details about the hardware FPU, but that should be taken as read). And compilers, apparently. The important point was that it would be best if all of the methods affected by summing 32 bit floats with 32 bit accumulators had the same Notes as numpy.mean(). We went through a lot of code yesterday, assuming that any numpy or Scipy.stats functions that use accumulators suffer the same issue, whether noted or not, and found it true. "Depending on the input data, this can cause the results to be inaccurate, especially for float32 (see example below). Specifying a higher-precision accumulator using the dtype keyword can alleviate this issue." seems rather un-Pythonic. - Ray

9 24

ImportError while building Numpy on Ubuntu 14.04
by Gabriel Altay 27 Jul '14

27 Jul '14

I'm attempting to build Numpy from source in order to do some development. I've cloned the github repo and installed the pre-reqs for Ubuntu http://www.scipy.org/scipylib/building/linux.html#debian-ubuntu However, when I do >>> python setup.py build I get Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 251, in <module> setup_package() File "setup.py", line 235, in setup_package from numpy.distutils.core import setup File "/home/galtay/github/numpy-env/numpy/numpy/distutils/__init__.py", line 37, in <module> from numpy.testing import Tester File "/home/galtay/github/numpy-env/numpy/numpy/testing/__init__.py", line 13, in <module> from .utils import * File "/home/galtay/github/numpy-env/numpy/numpy/testing/utils.py", line 17, in <module> from numpy.core import float32, empty, arange, array_repr, ndarray File "/home/galtay/github/numpy-env/numpy/numpy/core/__init__.py", line 6, in <module> from . import multiarray ImportError: cannot import name multiarray Any hints? I'm running Continuum Analytics Anaconda Python distribution thanks, -Gabriel

1 0

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by Eelco Hoogendoorn 26 Jul '14

26 Jul '14

Cool, sounds like great improvements. I can imagine that after some loop unrolling one becomes memory bound pretty soon. Is the summation guaranteed to traverse the data in its natural order? And do you happen to know what the rules for choosing accumulator dtypes are? -----Original Message----- From: "Julian Taylor" <jtaylor.debian(a)googlemail.com> Sent: ‎26-‎7-‎2014 00:58 To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays On 25.07.2014 23:51, Eelco Hoogendoorn wrote: > Ray: I'm not working with Hubble data, but yeah these are all issues > I've run into with my terrabytes of microscopy data as well. Given that > such raw data comes as uint16, its best to do your calculations as much > as possible in good old ints. What you compute is what you get, no > obscure shenanigans. integers are dangerous too, they overflow quickly and signed overflow is even undefined in C the standard. > > It just occurred to me that pairwise summation will lead to highly > branchy code, and you can forget about any vector extensions. Tradeoffs > indeed. Any such hierarchical summation is probably best done by > aggregating naively summed blocks. pairwise summation is usually implemented with a naive sum cutoff large enough so the recursion does not matter much. In numpy 1.9 this cutoff is 128 elements, but the inner loop is unrolled 8 times which makes it effectively 16 elements. the unrolling factor of 8 was intentionally chosen to allow using AVX in the inner loop without changing the summation ordering, but last I tested actually using AVX here only gave mediocre speedups (10%-20% on an i5). > ------------------------------------------------------------------------ > From: RayS <mailto:rays@blue-cove.com> > Sent: ‎25-‎7-‎2014 23:26 > To: Discussion of Numerical Python <mailto:numpy-discussion@scipy.org> > Subject: Re: [Numpy-discussion] numpy.mean still broken for > largefloat32arrays > > At 11:29 AM 7/25/2014, you wrote: >>On Fri, Jul 25, 2014 at 5:56 PM, RayS <rays(a)blue-cove.com> wrote: >> > The important point was that it would be best if all of the >> methods affected >> > by summing 32 bit floats with 32 bit accumulators had the same Notes as >> > numpy.mean(). We went through a lot of code yesterday, assuming that any >> > numpy or Scipy.stats functions that use accumulators suffer the same > issue, >> > whether noted or not, and found it true. >> >>Do you have a list of the functions that are affected? > > We only tested a few we used, but > scipy.stats.nanmean, or any .*mean() > numpy.sum, mean, average, std, var,... > > via something like: > > import numpy > import scipy.stats > print numpy.__version__ > print scipy.__version__ > onez = numpy.ones((2**25, 1), numpy.float32) > step = 2**10 > func = scipy.stats.nanmean > for s in range(2**24-step, 2**25, step): > if func(onez[:s+step])!=1.: > print '\nbroke', s, func(onez[:s+step]) > break > else: > print '\r',s, > >> That said, it does seem that np.mean could be implemented better than >>it is, even given float32's inherent limitations. If anyone wants to >>implement better algorithms for computing the mean, variance, sums, >>etc., then we would love to add them to numpy. > > Others have pointed out the possible tradeoffs in summation algos, > perhaps a method arg would be appropriate, "better" depending on your > desire for speed vs. accuracy. > > It just occurred to me that if the STSI folks (who count photons) > took the mean() or other such func of an image array from Hubble > sensors to find background value, they'd better always be using float64. > > - Ray > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
by Lars Buitinck 26 Jul '14

26 Jul '14

> Date: Fri, 25 Jul 2014 15:06:40 +0200 > From: Olivier Grisel <olivier.grisel(a)ensta.org> > Subject: Re: [Numpy-discussion] change default integer from int32 to > int64 on win64? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Content-Type: text/plain; charset=UTF-8 > > The dtype returned by np.where looks right (int64): > >>>> import platform >>>> platform.architecture() > ('64bit', 'WindowsPE') >>>> import numpy as np >>>> np.__version__ > '1.9.0b1' >>>> a = np.zeros(10) >>>> np.where(a == 0) > (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),) Strange. In [1] we had to cast the result of np.where because it was an array of long. I ran through the NumPy code, and I couldn't find the flaw, but neither could I find a point in the history where it was fixed. [1] https://github.com/scikit-learn/scikit-learn/commit/ebdeddbab1620c2473d04dc…

2 1

Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
by RayS 26 Jul '14

26 Jul '14

At 02:36 PM 7/25/2014, you wrote: >But it doesn't compensate for users to be aware of the problems. I >think the docstring and the description of the dtype argument is pretty clear. Most of the docs for the affected functions do not have a Note with the same warning as mean() - Ray

1 0