Mailman 3 January 2015 - NumPy-Discussion

[ANN] bcolz 0.7.3
by Valentin Haenel 04 Jan '15

04 Jan '15

====================== Announcing bcolz 0.7.3 ====================== What's new ========== This release includes the support for pickling persistent carray/ctable objects contributed by Matthew Rocklin. Also, the included version of Blosc is updated to ``v1.5.2``. Lastly, several minor issues and typos have been fixed, please see the release notes for details. ``bcolz`` is a renaming of the ``carray`` project. The new goals for the project are to create simple, yet flexible compressed containers, that can live either on-disk or in-memory, and with some high-performance iterators (like `iter()`, `where()`) for querying them. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… For more detailed info, see the release notes in: https://github.com/Blosc/bcolz/wiki/Release-Notes What it is ========== bcolz provides columnar and compressed data containers. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt ---- **Enjoy data!**

1 0

Regarding np.ma.masked_equal behavior
by Maniteja Nandana 04 Jan '15

04 Jan '15

Hello friends, This is an issue related to the working of *masked_equal* method. I was thinking if anyone related to an old ticket #1851 <https://github.com/numpy/numpy/issues/1851>, regarding the modification of *masked_equal *function effect on *fill_value *could clarify the situation, since right now, the documentation and implementation conflict. There is an issue raised regarding this #5408 <https://github.com/numpy/numpy/issues/5408>. Cheers*,* N.Maniteja _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

numpy.fromiter in numpypy
by Albert-Jan Roskam 04 Jan '15

04 Jan '15

Hi, I would like to use the numpy implementation for Pypy. In particular, I would like to use numpy.fromiter, which is available according to this overview: http://buildbot.pypy.org/numpy-status/latest.html. However, contrary to what this website says, this function is not yet available. Conclusion: the website is wrong. Or am I missing something? albertjan@debian:~$ sudo pypy $(which pip) install -U git+https://bitbucket.org/pypy/numpy.git albertjan@debian:~$ sudo pypy -c 'import numpy' # sudo: as per the installation instructions albertjan@debian:~$ pypy Python 2.7.8 (f5dcc2477b97386c11e4b67f08a2d00fbd2fce5d, Sep 19 2014, 10:37:41) [PyPy 2.4.0 with GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> import numpy as np >>>> np.__version__, sys.version ('1.9.0', '2.7.8 (f5dcc2477b97386c11e4b67f08a2d00fbd2fce5d, Sep 19 2014, 10:37:41)\n[PyPy 2.4.0 with GCC 4.8.2]') >>>> np.fromiter <function tmp at 0x00007fdc06e35bc8> >>>> np.fromiter((i for i in range(10)), np.float) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/pypy-2.4/site-packages/numpy/core/multiarray.py", line 55, in tmp raise NotImplementedError("%s not implemented yet" % func) NotImplementedError: fromiter not implemented yet The same also applies to numpy.fromfile Thanks in advance and happy 2015. Regards, Albert-Jan ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2 1

diag, diagonal, ravel and all that
by Charles R Harris 04 Jan '15

04 Jan '15

Hi All, The diag, diagonal, and ravel functions have recently been changed to preserve subtypes. However, this causes lots of backward compatibility problems for matrix users, in particular, scipy.sparse. One possibility for fixing this is to special case matrix and so that these functions continue to return 1-d arrays for matrix instances. This is kind of ugly as `a..ravel` will still return a matrix when a is a matrix, an ugly inconsistency. This may be a case where practicality beats beauty. Thoughts? Chuck

4 8

Optimizing multi-tensor contractions in numpy
by Daniel Smith 03 Jan '15

03 Jan '15

Hello everyone, I have been working on a chunk of code that basically sets out to provide a single function that can take an arbitrary einsum expression and computes it in the most optimal way. While np.einsum can compute arbitrary expressions, there are two drawbacks to using pure einsum: einsum does not consider building intermediate arrays for possible reductions in overall rank and is not currently capable of using a vendor BLAS. I have been working on a project that aims to solve both issues simultaneously: https://github.com/dgasmith/opt_einsum <https://github.com/dgasmith/opt_einsum> This program first builds the optimal way to contract the tensors together, or using my own nomenclature a “path.” This path is then iterated over and uses tensordot when possible and einsum for everything else. In test cases the worst case scenario adds a 20 microsecond overhead penalty and, in the best case scenario, it can reduce the overall rank of the tensor. The primary (if somewhat exaggerated) example is a 5-term N^8 index transformation that can be reduced to N^5; even when N is very small (N=10) there is a 2,000 fold speed increase over pure einsum or, if using tensordot, a 2,400 fold speed increase. This is somewhat similar to the new np.linalg.multi_dot function. If you are interested in this function please head over to the github repo and check it out. I believe the README is starting to become self-explanatory, but feel free to email me with any questions. This originally started because I was looking into using numpy to rapidly prototype quantum chemistry codes. The results of which can be found here: https://github.com/dgasmith/psi4numpy <https://github.com/dgasmith/psi4numpy> As such, I am very interested in implementing this into numpy. While I think opt_einsum is in a pretty good place, there is still quite a bit to do (see outstanding issues in the README). Even if this is not something that would fit into numpy I would still be very interested in your comments. Thank you for your time, -Daniel Smith

3 3

Correct C string handling in the NumPy C API?
by Sturla Molden 03 Jan '15

03 Jan '15

Here is an example: NPY_NO_EXPORT NpyIter_IterNextFunc * NpyIter_GetIterNext(NpyIter *iter, char **errmsg) { npy_uint32 itflags = NIT_ITFLAGS(iter); int ndim = NIT_NDIM(iter); int nop = NIT_NOP(iter); if (NIT_ITERSIZE(iter) < 0) { if (errmsg == NULL) { PyErr_SetString(PyExc_ValueError, "iterator is too large"); } else { *errmsg = "iterator is too large"; } return NULL; } After NpyIter_GetIterNext returns, *errmsg points to a local variable in a returned function. Either I am wrong about C, or this code has undefied behavior... My gutfeeling is that *errmsg = "iterator is too large"; puts the string "iterator is too large" on the stack and points *errmsg to the string. Shouldn't this really be strcpy(*errmsg, "iterator is too large"); and then *errmsg should point to a char buffer allocated before NpyIter_GetIterNext is called? Or will the statement *errmsg = "iterator is too large"; put the string on the stack in the calling C function? Before I open an issue I will ask if my understanding of C is correct or not. I am a bit confused here... Regards, Sturla

2 3

Pass 2d ndarray into C **double using ctypes
by Yuxiang Wang 03 Jan '15

03 Jan '15

Dear all, I am currently using a piece of C code, where one of the input argument of a function is **double. So, in numpy, I tried np.ctypeslib.ndpointer(ctypes.c_double), but obviously this wouldn't work because this is only *double, not **double. Then I tried np.ctypeslib.ndpointer(np.ctypeslib.ndpointer(ctypes.c_double)), but this didn't work either because it says "ArgumentError: argument 4: <class 'TypeError'>: array must have data type uint64 ". np.ctypeslib.ndpointer(ctypes.c_double, ndim=2) wound't work too, because **double is not the same with *double[]. Could anyone please give any thoughts to help? Thanks, Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj(a)virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/

3 11

Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
by Stephan Hoyer 03 Jan '15

03 Jan '15

I recently wrote function to manually broadcast an ndarray to a given shape according to numpy's broadcasting rules (using strides): https://github.com/xray/xray/commit/7aee4a3ed2dfd3b9aff7f3c5c6c68d51df2e3ff3 The same functionality can be done pretty straightforwardly with np.broadcast_arrays, but that function does both too much (I don't actually have a second array that needs to be broadcast) and not enough (I need to create a dummy array to broadcast against it). This approach is simpler, and also, according to my benchmarks, about 3x faster than np.broadcast_arrays: In [1]: import xray In [2]: import numpy as np In [3]: x = np.random.randn(4) In [4]: y = np.empty((2, 3, 4)) In [5]: %timeit xray.core.utils.as_shape(x, y.shape) 100000 loops, best of 3: 17 µs per loop In [6]: %timeit np.broadcast_arrays(x, y)[0] 10000 loops, best of 3: 47.4 µs per loop Would this be a welcome addition to numpy's lib.stride_tricks? If so, I will put together a PR. In my search, I turned up a Stack Overflow post looking for similar functionality: https://stackoverflow.com/questions/11622692/is-there-a-better-way-to-broad… Cheers, Stephan

8 24

Extracting required indices from the array of tuples
by Simen Langseth 02 Jan '15

02 Jan '15

import numpy as np from scipy import signal y = np.array([[2, 1, 2, 3, 2, 0, 1, 0], [2, 1, 2, 3, 2, 0, 1, 0]]) maximas = signal.argrelmax(y, axis=1) print maximas (array([0, 0, 1, 1], dtype=int64), array([3, 6, 3, 6], dtype=int64)) I want to extract only the first maxima of both rows, i.e., [3, 3] using the tuples (maximas). How would you do it?

2 2

npymath on Windows
by Matthew Brett 02 Jan '15

02 Jan '15

Hi, Sorry for this ignorant email, but we got confused trying to use 'libnpymath.a' from the mingw builds of numpy: We were trying to link against the mingw numpy 'libnpymath.a' using Visual Studio C, but this give undefined symbols from 'libnpymath.a' like this: npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _atanf referenced in function _npy_atanf npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _acosf referenced in function _npy_acosf npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _asinf referenced in function _npy_asinf (see : http://nipy.bic.berkeley.edu/builders/dipy-bdist32-33/builds/73/steps/shell…) npymath.lib from Christophe Gohlke's (MSVC compiled) numpies does not give such an error. Sure enough, 'npymath.lib' shows these lines from `dumpbin /all npymath.lib`: 00000281 REL32 00000000 4F asinf 00000291 REL32 00000000 51 acosf 000002A1 REL32 00000000 53 atanf whereas `dumpbin /all libnpymath.a` shows these kinds of lines: 000008E5 REL32 00000000 86 _asinf 000008F5 REL32 00000000 85 _acosf 00000905 REL32 00000000 84 _atanf As far as I can see, 'acosf' is defined in the msvc runtime library. I guess that '_acosf' is defined in some mingw runtime library? Is there any way of making a npymath library that will pick up the msvc math and so may work with both msvc and mingw? Sorry again if that's a dumb question, Matthew

4 7