Mailman 3 February 2014 - NumPy-Discussion

New (old) function proposal.
by Charles R Harris Feb. 19, 2014

Feb. 19, 2014

Hi All, There is an old ticket, #1499 <https://github.com/numpy/numpy/issues/1499>, that suggest adding a segment_axis function. def segment_axis(a, length, overlap=0, axis=None, end='cut', endvalue=0): """Generate a new array that chops the given array along the given axis into overlapping frames. Parameters ---------- a : array-like The array to segment length : int The length of each frame overlap : int, optional The number of array elements by which the frames should overlap axis : int, optional The axis to operate on; if None, act on the flattened array end : {'cut', 'wrap', 'end'}, optional What to do with the last frame, if the array is not evenly divisible into pieces. - 'cut' Simply discard the extra values - 'wrap' Copy values from the beginning of the array - 'pad' Pad with a constant value endvalue : object The value to use for end='pad' Examples -------- >>> segment_axis(arange(10), 4, 2) array([[0, 1, 2, 3], [2, 3, 4, 5], [4, 5, 6, 7], [6, 7, 8, 9]]) Is there and interest in having this function available? Chuck

7 8

except expression discussion on python-ideas
by Alexander Belopolsky Feb. 19, 2014

Feb. 19, 2014

I would like to invite numpy community to weigh in on the idea that is getting momentum at https://mail.python.org/pipermail/python-ideas/2014-February/025437.html The main motivation is to provide syntactic alternative to proliferation of default value options, so that x = getattr(u, 'answer', 42) can be written as x = y.answer except ... 42 For a dictionary d, x = d.get('answer', 42) can be written as x = d['answer'] except ... 42 For a list L, try: x = L[i] except IndexError: x= 42 can be written as x = L[i] except ... 42 The ellipsis in the above stands for syntax being debated. Effectively, Python is about to gain support for a new operator and operators are very precious for numpy. So, I think numpy community has a horse in that race.

3 2

Default dtype of genfromtxt
by Charles R Harris Feb. 18, 2014

Feb. 18, 2014

This is apropos issue #1860 <https://github.com/numpy/numpy/issues/1860>, where it is proposed that the default dtype of genfromtxt should be None rather than float. A decision is needed. Thoughts? Chuck

2 1

Rethinking multiple dimensional indexing with sequences?
by Sebastian Berg Feb. 18, 2014

Feb. 18, 2014

Hey all, currently in numpy this is possible: a = np.zeros((5, 5)) a[[0, slice(None, None)]] #this behaviour has its quirks, since the "correct" way is: a[(0, slice(None, None))] # or identically a[0, :] The problem with using an arbitrary sequence is, that an arbitrary sequence is also typically an "array like" so there is a lot of guessing involved: a[[0, slice(None, None)]] == a[(0, slice(None, None))] # but: a[[0, 1]] == a[np.array([0, 1])] Now also NumPy commonly uses lists here to build up indexing tuples (since they are mutable), however would it really be so bad if we had to do `arr[tuple(slice_list)]` in the end to resolve this issue? So the proposal would be to deprecate anything but (base class) tuples, or maybe at least only allow this weird logic for lists and not all sequences. I do not believe we can find a logic to decide what to do which will not be broken in some way... PS: The code implementing the "advanced index or nd-index" logic is here: https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/mappin… - Sebastian Another confusing example: In [9]: a = np.arange(10) In [10]: a[[(0, 1), (2, 3)] * 17] # a[np.array([(0, 1), (2, 3)] * 17)] Out[10]: array([[0, 1], <snip> [2, 3]]) In [11]: a[[(0, 1), (2, 3)]] # a[np.array([0, 1]), np.array([2, 3])] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) <ipython-input-11-57b93f64dfa6> in <module>() ----> 1 a[[(0, 1), (2, 3)]] IndexError: too many indices for array

2 2

allocated memory cache for numpy
by Julian Taylor Feb. 18, 2014

Feb. 18, 2014

hi, I noticed that during some simplistic benchmarks (e.g. https://github.com/numpy/numpy/issues/4310) a lot of time is spent in the kernel zeroing pages. This is because under linux glibc will always allocate large memory blocks with mmap. As these pages can come from other processes the kernel must zero them for security reasons. For memory within the numpy process this unnecessary and possibly a large overhead for the many temporaries numpy creates. The behavior of glibc can be tuned to change the threshold at which it starts using mmap but that would be a platform specific fix. I was thinking about adding a thread local cache of pointers to of allocated memory. When an array is created it tries to get its memory from the cache and when its deallocated it returns it to the cache. The threshold and cached memory block sizes could be adaptive depending on the application workload. For simplistic temporary heavy benchmarks this eliminates the time spent in the kernel (system with time). But I don't know how relevant this is for real world applications. Have you noticed large amounts of time spent in the kernel in your apps? I also found this paper which describes pretty much exactly what I'm proposing: pyhpc.org/workshop/papers/Doubling.pdf‎ Someone know why their changes were never incorporated in numpy? I couldn't find a reference in the list archive.

5 15

Suggestion: Port Theano RNG implementation to NumPy
by Frédéric Bastien Feb. 18, 2014

Feb. 18, 2014

Hi, In a ticket I did a coment and Charles suggested that I post it here: In Theano we have an C implementation of a faster RNG: MRG31k3p. It is faster on CPU, and we have a GPU implementation. It would be relatively easy to parallize on the CPU with OpenMP. If someone is interested to port this to numpy, their wouldn't be any dependency problem. No license problem as Theano license have the same license as NumPy. The speed difference is significant, but I don't recall numbers. Fred

4 7

Proposal to make power return float, and other such things.
by Charles R Harris Feb. 18, 2014

Feb. 18, 2014

This is apropos issue #899 <https://github.com/numpy/numpy/issues/899>, where it is suggested that power promote integers to float. That sounds reasonable to me, but such a change in behavior makes it a bit iffy. Thoughts? Chuck

9 19

Numpy type inheritance
by Charles R Harris Feb. 18, 2014

Feb. 18, 2014

Hi All, I'm looking for some expert explication of the problems in issues #1398<https://github.com/numpy/numpy/issues/1398>and #1397 <https://github.com/numpy/numpy/issues/1397>. The segfault in the second looks nasty. Chuck

1 0

ANN: numexpr 2.3.1 released
by Francesc Alted Feb. 18, 2014

Feb. 18, 2014

========================== Announcing Numexpr 2.3.1 ========================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== * Added support for shift-left (<<) and shift-right (>>) binary operators. See PR #131. Thanks to fish2000! * Removed the rpath flag for the GCC linker, because it is probably not necessary and it chokes to clang. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

bug with mmap'ed datetime64 arrays
by Charles G. Waldman Feb. 17, 2014

Feb. 17, 2014

test case: #!/usr/bin/env python import numpy as np a=np.array(['2014', '2015', '2016'], dtype='datetime64') x=np.datetime64('2015') print a>x np.save('test.npy', a) b = np.load('test.npy', mmap_mode='c') print b>x result: >>> [False False True] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/tmp/t.py", line 12, in <module> print b>x File "/usr/lib64/python2.7/site-packages/numpy/core/memmap.py", line 279, in __array_finalize__ if hasattr(obj, '_mmap') and np.may_share_memory(self, obj): File "/usr/lib64/python2.7/site-packages/numpy/lib/utils.py", line 298, in may_share_memory a_low, a_high = byte_bounds(a) File "/usr/lib64/python2.7/site-packages/numpy/lib/utils.py", line 258, in byte_bounds bytes_a = int(ai['typestr'][2:]) ValueError: invalid literal for int() with base 10: '8[Y]' fix: diff --git a/numpy/lib/utils.py b/numpy/lib/utils.py index 1f1cdfc..c73f2f1 100644 --- a/numpy/lib/utils.py +++ b/numpy/lib/utils.py @@ -210,7 +210,7 @@ def byte_bounds(a): a_data = ai['data'][0] astrides = ai['strides'] ashape = ai['shape'] - bytes_a = int(ai['typestr'][2:]) + bytes_a = a.dtype.itemsize a_low = a_high = a_data if astrides is None: # contiguous case will submit pull request via github

1 0