Mailman 3 March 2015 - NumPy-Discussion

Re: [Numpy-discussion] 3D array and the right hand rule
by Dieter Van Eessen 17 Mar '15

17 Mar '15

Hello, Sorry to disturb again, but the topic still bugs me somehow... I'll try to rephrase the question: - What's the influence of the type of N-array representation with respect to TENSOR-calculus? - Are multiple representations possible? - I assume that the order of the dimensions plays a major role in for example TENSOR product. Is this assumption correct? As I said before, my math skills are lacking in this area... I hope you consider this a valid question. kind regards, Dieter On Fri, Jan 30, 2015 at 2:32 AM, Alexander Belopolsky <ndarray(a)mac.com> wrote: > > On Mon, Jan 26, 2015 at 6:06 AM, Dieter Van Eessen < > dieter.van.eessen(a)gmail.com> wrote: > >> I've read that numpy.array isn't arranged according to the >> 'right-hand-rule' (right-hand-rule => thumb = +x; index finger = +y, bend >> middle finder = +z). This is also confirmed by an old message I dug up from >> the mailing list archives. (see message below) >> > > Dieter, > > It looks like you are confusing dimensionality of the array with the > dimensionality of a vector that it might store. If you are interested in > using numpy for 3D modeling, you will likely only encounter 1-dimensional > arrays (vectors) of size 3 and 2-dimensional arrays (matrices) of size 9 > or shape (3, 3). > > A 3-dimensional array is a stack of matrices and the 'right-hand-rule' > does not really apply. The notion of C/F-contiguous deals with the order > of axes (e.g. width first or depth first) while the right-hand-rule is > about the direction of the axes (if you "flip" the middle finger right hand > becomes left.) In the case of arrays this would probably correspond to > little-endian vs. big-endian: is a[0] stored at a higher or lower address > than a[1]. However, whatever the answer to this question is for a > particular system, it is the same for all axes in the array, so right-hand > - left-hand distinction does not apply. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- gtz, Dieter VE

2 1

ANN: HoloViews 1.0 released
by James A. Bednar 17 Mar '15

17 Mar '15

We are pleased to announce the first public release of HoloViews, a Python package for scientific and engineering data visualization: http://ioam.github.io/holoviews HoloViews provides composable, sliceable, declarative data structures for building even complex visualizations easily. It's designed to exploit the rich ecosystem of scientific Python tools already available, using Numpy for data storage, matplotlib and mpld3 as plotting backends, and integrating fully with IPython Notebook to make your data instantly visible. If you look at the website for just about any other visualization package, you'll see a long list of pretty pictures, each one of which has a page or two of code putting it together. There are pretty pictures in HoloViews too, but there is *no* hidden code -- *all* of the steps needed to build a given figure are shown right before the HoloViews plot, with just a few lines needed for nearly all of our examples, even complex multi-figure subplots and animations. This concise but flexible specification makes it practical to explore and analyze your data interactively, while leaving a full record for later reproducibility in the notebook. It may sound like magic, but it's not -- HoloViews simply lets you annotate your data with appropriate metadata, and then the data can display itself! HoloViews provides a set of general, compositional, multidimensional data structures suitable for both discrete and continuous real-world data, and pairs them with separate customizable plotting classes to visualize them without extensive coding. An large collection of continuously tested IPython Notebook tutorials accompanies HoloViews, showing you precisely the small number of steps required to generate any of the plots. Some of the most important features: - Freely available under a BSD license - Python 2 and 3 compatible - Minimal external dependencies -- easy to integrate into your workflow - Builds figures by slicing, sampling, and composing your data - Builds web-embeddable animations without any extra coding - Easily customizable without obscuring the underlying data objects - Includes interfaces to pandas and Seaborn - Winner of the 2015 UK Open Source Award For the rest, check out ioam.github.io/holoviews! Jean-Luc Stevens Philipp Rudiger James A. Bednar The University of Edinburgh School of Informatics -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

should views into structured arrays be reversible?
by Allan Haldane 17 Mar '15

17 Mar '15

Hello all, I've introduced PR 5548 <https://github.com/numpy/numpy/pull/5548> which, through more careful safety checks, allows views of object arrays. However, I had to make 'partial views' into structured arrays irreversible, and I want to check with the list that that's ok. With the PR, if you only view certain fields of an array you cannot take a 'reverse' view of the resulting object to get back the original array: >>> arr = np.array([(1,2),(4,5)], dtype=[('A', 'i'), ('B', 'i')]) >>> varr = arr.view({'names': ['A'], 'formats': ['i'], ... 'itemsize': arr.dtype.itemsize}) >>> varr.view(arr.dtype) TypeError: view would access data parent array doesn't own Ie., with this PR you can only take views into parts of an array that have fields. This was necessary in order to guarantee that we never interpret memory containing a python Object as another type, which could cause a segfault. I have a more extensive discussion & motivation in the PR, including an alternative idea. So does this limitation seem reasonable? Cheers, Allan

1 0

Fastest way to compute summary statistics for a specific axis
by Dave Hirschfeld 17 Mar '15

17 Mar '15

I have a number of large arrays for which I want to compute the mean and standard deviation over a particular axis - e.g. I want to compute the statistics for axis=1 as if the other axes were combined so that in the example below I get two values back In [1]: a = randn(30, 2, 10000) For the mean this can be done easily like: In [2]: a.mean(0).mean(-1) Out[2]: array([ 0.0007, -0.0009]) ...but this won't work for the std. Using some transformations we can come up with something which will work for either: In [3]: a.transpose(2,0,1).reshape(-1, 2).mean(axis=0) Out[3]: array([ 0.0007, -0.0009]) In [4]: a.transpose(1,0,2).reshape(2, -1).mean(axis=-1) Out[4]: array([ 0.0007, -0.0009]) If we look at the performance of these equivalent methods: In [5]: %timeit a.transpose(2,0,1).reshape(-1, 2).mean(axis=0) 100 loops, best of 3: 14.5 ms per loop In [6]: %timeit a.transpose(1,0,2).reshape(2, -1).mean(axis=-1) 100 loops, best of 3: 5.05 ms per loop we can see that the latter version is a clear winner. Investigating further, both methods appear to copy the data so the performance is likely down to better cache utilisation. In [7]: np.may_share_memory(a, a.transpose(2,0,1).reshape(-1, 2)) Out[7]: False In [8]: np.may_share_memory(a, a.transpose(1,0,2).reshape(2, -1)) Out[8]: False Both methods are however significantly slower than the initial attempt: In [9]: %timeit a.mean(0).mean(-1) 1000 loops, best of 3: 1.2 ms per loop Perhaps because it allocates a smaller temporary? For those who like a challenge: is there a faster way to achieve what I'm after? Cheers, Dave

4 4

numpy.stack -- which function, if any, deserves the name?
by Stephan Hoyer 16 Mar '15

16 Mar '15

In the past months there have been two proposals for new numpy functions using the name "stack": 1. np.stack for stacking like np.asarray(np.bmat(...)) http://thread.gmane.org/gmane.comp.python.numeric.general/58748/ https://github.com/numpy/numpy/pull/5057 2. np.stack for stacking along an arbitrary new axis (this was my proposal) http://thread.gmane.org/gmane.comp.python.numeric.general/59850/ https://github.com/numpy/numpy/pull/5605 Both functions generalize the notion of stacking arrays from the existing hstack, vstack and dstack, but in two very different ways. Both could be useful -- but we can only call one "stack". Which one deserves that name? The existing *stack functions use the word "stack" to refer to combining arrays in two similarly different ways: a. For ND -> ND stacking along an existing dimensions (like numpy.concatenate and proposal 1) b. For ND -> (N+1)D stacking along new dimensions (like proposal 2). I think it would be much cleaner API design if we had different words to denote these two different operations. Concatenate for "combine along an existing dimension" already exists, so my thought (when I wrote proposal 2), was that the verb "stack" could be reserved (going forward) for "combine along a new dimension." This also has the advantage of suggesting that "concatenate" and "stack" are the two fundamental operations for combining N-dimensional arrays. The documentation on this is currently quite confusing, mostly because no function like that in proposal 2 currently exists. Of course, the *stack functions have existed for quite some time, and in many cases vstack and hstack are indeed used for concatenate like functionality (e.g., whenever they are used for 2D arrays/matrices). So the case is not entirely clear-cut. (We'll never be able to remove this functionality from NumPy.) In any case, I would appreciate your thoughts. Best, Stephan

3 3

Fix masked arrays to properly edit views
by John Kirkham 14 Mar '15

14 Mar '15

The sample case of the issue ( https://github.com/numpy/numpy/issues/5558 ) is shown below. A proposal to address this behavior can be found here ( https://github.com/numpy/numpy/pull/5580 ). Please give me your feedback. I tried to change the mask of `a` through a subindexed view, but was unable. Using this setup I can reproduce this in the 1.9.1 version of NumPy. import numpy as np a = np.arange(6).reshape(2,3) a = np.ma.masked_array(a, mask=np.ma.getmaskarray(a), shrink=False) b = a[1:2,1:2] c = np.zeros(b.shape, b.dtype) c = np.ma.masked_array(c, mask=np.ma.getmaskarray(c), shrink=False) c[:] = np.ma.masked This yields what one would expect for `a`, `b`, and `c` (seen below). masked_array(data = [[0 1 2] [3 4 5]], mask = [[False False False] [False False False]], fill_value = 999999) masked_array(data = [[4]], mask = [[False]], fill_value = 999999) masked_array(data = [[--]], mask = [[ True]], fill_value = 999999) Now, it would seem reasonable that to copy data into `b` from `c` one can use `__setitem__` (seen below). b[:] = c This results in new data and mask for `b`. masked_array(data = [[--]], mask = [[ True]], fill_value = 999999) This should, in turn, change `a`. However, the mask of `a` remains unchanged (seen below). masked_array(data = [[0 1 2] [3 0 5]], mask = [[False False False] [False False False]], fill_value = 999999) Best, John

2 1

Error message
by Danny Kramer 14 Mar '15

14 Mar '15

Hi, I am getting the following error message: C:\Python27\lib\site-packages\numpy\lib\npyio.py:819: UserWarning: loadtxt: Empty input file: "[]" warnings.warn('loadtxt: Empty input file: "%s"' % fname) main loop list assignment index out of range 1. Why is it happening? 2. How can I fix it? Thanks a lot in advance. Danny

2 1

Numpy where
by Charles R Harris 13 Mar '15

13 Mar '15

Hi All, This is apropos gh-5582 <https://github.com/numpy/numpy/pull/5582> dealing with some corner cases of np.where. The following are the current behavior >>> import numpy >>> numpy.where(True) # case 1 ... (array([0]),) >>> numpy.where(True, None, None) # case 2 ... array(None, dtype=object) >>> numpy.ma.where(True) # case 3 ... (array([0]),) >>> numpy.ma.where(True, None, None) # case 4 ... (array([0]),) The question is, what exactly should be done in these cases? I'd be inclined to raise an error for cases 1 and 3. Case two looks correct to me if we agree that scalar inputs are acceptable. Case 4 looks wrong. Thoughts? Chuck

3 6

Custom __array_interface__ error
by Daniel Smith 13 Mar '15

13 Mar '15

Greetings everyone, I have a new project that deals with core and disk tensors wrapped into a single object so that the expressions are transparent to the user after the tensor is formed. I would like to add __array_interface__ to the core tensor and provide a reasonable error message if someone tries to call the __array_interface__ for a disk tensor. I may be missing something, but I do not see an obvious way to do this in the python layer. Currently I do something like: if ttype == “Core": self.__array_interface__ = self.tensor.ndarray_interface() else: self.__array_interface__ = {'typestr’: 'Only Core tensor types are supported.’} Which provides at least a readable error message if it is not a core tensor: TypeError: data type "Only Core tensor types are supported." not understood A easy solution I see is to change numpy C side __array_interface__ error message to throw custom strings. In numpy/core/src/multiarray/ctors.c:2100 we have the __array_interface__ conversion: if (!PyDict_Check(iface)) { Py_DECREF(iface); PyErr_SetString(PyExc_ValueError, "Invalid __array_interface__ value, must be a dict"); return NULL; } It could simply be changed to: if (!PyDict_Check(iface)) { if (PyString_Check(iface)){ PyErr_SetString(PyExc_ValueError, iface); } else{ PyErr_SetString(PyExc_ValueError, "Invalid __array_interface__ value, must be a dict”); } Py_DECREF(iface); return NULL; } Thoughts? Cheers, -Daniel Smith

3 2

random.RandomState and deepcopy
by Neal Becker 13 Mar '15

13 Mar '15

It is common that to guarantee good statistical independence between various random generators, a singleton instance of an RNG is shared between them. So I typically have various random generator objects, which (sometimes several levels objects deep) embed an instance of RandomState. Now I have a requirement to copy a generator object (without knowing exactly what that generator object is). My solution is to use deepcopy on the top-level object. But I need to overload __deepcopy__ on the singleton RandomState object. Unfortunately, RandomState doesn't allow customization of __deepcopy__ (or anything else). And it has no __dict__. My solution is: class shared_random_state (object): def __init__ (self, rs): self.rs = rs def __getattr__ (self, attr): return getattr (self.rs, attr) def __deepcopy__ (self): return self An example usage: rs = shared_random_state (RandomState(0)) from exponential import exponential e = exponential (rs, 1) where exponential is: class exponential (object): def __init__ (self, rs, mu): self.rs = rs self.mu = mu def __call__ (self, size=None): if size == None: return self.rs.exponential (self.mu, 1)[0] else: return self.rs.exponential (self.mu, size) def __repr__ (self): return 'exp(%s)' % self.mu I wonder if anyone has any other suggestions? Personally, I would prefer if numpy provided a more direct solution to this. Either by providing for overloading RandomState deepcopy, or by making the copy behavior switchable with a flag to the constructor. -- Those who fail to understand recursion are doomed to repeat it

2 3