NumPy-Discussion August 2014

numpy-discussion@python.org

55 participants
47 discussions

Unsubscribe
by Mochena, Mogus D. 28 Aug '14

28 Aug '14

Please unsubsribe me from this list. ________________________________ From: numpy-discussion-bounces(a)scipy.org [numpy-discussion-bounces(a)scipy.org] on behalf of Charles R Harris [charlesr.harris(a)gmail.com] Sent: Thursday, August 28, 2014 1:00 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] ANN: NumPy 1.9.0 release candidate 1 available On Wed, Aug 27, 2014 at 3:52 PM, Orion Poplawski <orion(a)cora.nwra.com<mailto:orion@cora.nwra.com>> wrote: On 08/27/2014 11:07 AM, Julian Taylor wrote: > Hello, > > Almost punctually for EuroScipy we have finally managed to release the > first release candidate of NumPy 1.9. > We intend to only fix bugs until the final release which we plan to do > in the next 1-2 weeks. I'm seeing the following errors from setup.py: non-existing path in 'numpy/f2py': 'docs' non-existing path in 'numpy/f2py': 'f2py.1' non-existing path in 'numpy/lib': 'benchmarks' Hmm, benchmarks is long gone, and the f2py files also no longer exist. Since this causes no problems on my system, I suspect something else. How are you doing the install? Chuck

1 0

Should concatenate broadcast shapes?
by Jaime Fernández del Río 27 Aug '14

27 Aug '14

After reading this stackoverflow question: http://stackoverflow.com/questions/25530223/append-a-list-at-the-end-of-eac… I was reminded that the `np.concatenate` family of functions do not broadcast the shapes of their inputs: >>> import numpy as np >>> a = np.arange(6).reshape(3, 2) >>> b = np.arange(6, 8) >>> np.concatenate((a, b), axis=1) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: all the input arrays must have same number of dimensions >>> np.concatenate((a, b[None]), axis=1) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: all the input array dimensions except for the concatenation axis must match exactly >>> np.concatenate((a, np.tile(b[None], (a.shape[0], 1))), axis=1) array([[0, 1, 6, 7], [2, 3, 6, 7], [4, 5, 6, 7]]) But there doesn't seem to be any fundamental reason why they shouldn't: >>> from numpy.lib.stride_tricks import as_strided >>> b_ = as_strided(b, (a.shape[0],)+b.shape, (0,)+b.strides) >>> np.concatenate((a, b_), axis=1) array([[0, 1, 6, 7], [2, 3, 6, 7], [4, 5, 6, 7]]) Is there any fundamental interface design reason why things are the way they are? Or is it simply that no one has implemented broadcasting for these functions? Without thinking much about it, I am +1 on doing this... At the least, it would probably be good to add a note to the docs explaining why broadcasting is not implemented. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 2

Convert 3d NumPy array into 2d
by phinn stuart 27 Aug '14

27 Aug '14

Hi everyone, how can I convert (1L, 480L, 1440L) shaped numpy array into (480L, 1440L)? Thanks in the advance. phinn

5 4

Bug in genfromtxt with usecols and converters
by Adrian Altenhoff 27 Aug '14

27 Aug '14

Hi, I tried to load data from a csv file into numpy using genfromtxt. I need only a subset of the columns and want to apply some conversions to the data. attached is a minimal script showing the error. In brief, I want to load columns 1,2 and 4. But in the converter function for the 4th column, I get the 3rd value. The issue does not occur if I also load the 3rd column. Did I somehow misunderstand how the function is supposed to work or is this indeed a bug? I'm using python 3.3.1 with numpy 1.8.1 Regards Adrian

2 5

Return item rather than scalar for user defined types
by Anthony Scopatz 26 Aug '14

26 Aug '14

Hello All, Yesterday I opened PR #4889 <https://github.com/numpy/numpy/pull/4998> to solve a problem I have been having w.r.t. xdress and Nathaniel asked me bring the issue up here. The PR itself is quite small (6 lines?) and is easy to review. The opening text of my PR is pasted below because I believe that is a pretty good description of the issue. But briefly, pulling user defined dtypes out of an array do not behave idiomatically because you get a numpy scalar rather than a more representative Python object. For user-defined dtypes - which are typically more complex and possibly stateful than the builtin dtypes, I believe that it makes much more sense to get actual Python representation back a la the getitem() function. In fact, I think that this case also applies to the object dtype. However, changing that usage would likely break downstream code and would be inconsistent with how other builtin types are returned. In future major versions of numpy it would be ideal if the dtypes themselves could flag how they wished to be returned - either as a scalar or as the Python item. Thoughts? Be Well Anthony This updates what is effectively the __getitem__() method. For arrays such that the dtype is a user defined type, you receive the return that dtype's getitem() rather than a numpy scalar of the dtype. This allow the custom type to present a single Python API as well as an associated dtype. It also prevents users from having to subclass ndarray to get the appropriate behaviour. For example, suppose that we have a dtype representing a C++ std::vector<int> and then we had a numpy array of this dtype. From Python, it might look like >>> arrarray([array([0, 0, 0, 0, 0], dtype=int32), array([0, 1, 2, 3, 4], dtype=int32), array([0, 2, 4, 6, 8], dtype=int32)], dtype='xd_vector_int') Without this PR, you'd have to do the following to access the most deeply nested elements: >>> arr.item(2)[4]8 This is because you cannot index a scalar: >>> arr[2][4]IndexError: invalid index to scalar variable With this PR, the idiomatic expression is now allowable because arr[2] is the associated Python type: >>> arr[2][4]8 This is a pretty big deal for xdress <http://xdress.org/> which creates many custom dtypes and provided a Python interface into those. See xdress/xdress#265 <https://github.com/xdress/xdress/pull/265> for what prompted this. Thanks for considering!

1 0

Changing the numpy array into required shape
by Cleo Drakos 23 Aug '14

23 Aug '14

Hello numpy users: I have 2d numpy array of 480 rows and 1440 columns as named by 'data' below: The first element belongs to (49.875S,179.875W), the second element belongs to (49.625S,179.625W),and the last element belongs to (49.875N,179.875E). import os, glob, gdal, numpy as np fname = '3B42RT.2014010606.7.bin' with open(fname, 'rb') as fi: fi.seek(2880,0) data = np.fromfile(fi,dtype=np.uint16,count=480*1440) data = data.byteswap() data = data.reshape(1440,480) How can I convert this numpy array so that its first element belongs to (49.875N,179.625W), i.e., upper left latitude and longitude respectively; and the last element belong to (49.625S,179.875E), i.e., lower right latitute and longitude respectively. I tried to rotate it, but I do not think it is correct. data = np.rot90(data,1) Have some of you experienced with this type of problem? The binary file I am using is here: ftp://trmmopen.gsfc.nasa.gov/pub/merged/3B42RT/3B42RT.2014010606.7.bin.gz cleo

3 2

Best way to broadcast a function from C
by James Crist 23 Aug '14

23 Aug '14

All, I have a C function func that takes in scalar arguments, and an array of fixed dimension that is modified in place to provide the output. The prototype is something like: `void func(double a, double b, double c, double *arr);` I've wrapped this in Cython and called it from python with no problem. What I'd like to do now though is get it set up to broadcast over the input arguments and return a 3 dimensional array of the results. By this I mean a = array([1, 2, 3]) b = array([2.0, 3.0, 4.0]) c = array([3, 4, 5]) func(a, b, c) -> a 3d array containing the results of func for (a, b, c) = (1, 2.0, 3), (2, 3.0, 4), (3, 4.0, 5) I'm not sure if this would qualify as a ufunc, as the result of one function call isn't a scalar but an array, but the effect I'm looking for is similar. Ideally it would handle datatype conversions (in the above `a` and `c` aren't double, but `func` takes in double). It would also be awesome to allow an argument to be a scalar and not an array, and have it be broadcast as if it were. I'm just wondering what the best way for me to hook my code up to the internals of numpy and get this kind of behavior in an efficient way. I've read the "writing your own ufunc" part of the docs, but am unsure if what I'm looking for qualifies. Note that I can change the inner workings of `func` if this is required to achieve this behavior. Thanks!

3 4

np.unique with structured arrays
by Nicolas P. Rougier 22 Aug '14

22 Aug '14

Hello, I've found a strange behavior or I'm missing something obvious (or np.unique is not supposed to work with structured arrays). I'm trying to extract unique values from a simple structured array but it does not seem to work as expected. Here is a minimal script showing the problem: import numpy as np V = np.zeros(4, dtype=[("v", np.float32, 3)]) V["v"] = [ [0.5, 0.0, 1.0], [0.5, -1.e-16, 1.0], # [0.5, +1.e-16, 1.0] works [0.5, 0.0, -1.0], [0.5, -1.e-16, -1.0]] # [0.5, +1.e-16, -1.0]] works V_ = np.zeros_like(V) V_["v"][:,0] = V["v"][:,0].round(decimals=3) V_["v"][:,1] = V["v"][:,1].round(decimals=3) V_["v"][:,2] = V["v"][:,2].round(decimals=3) print np.unique(V_) [([0.5, 0.0, 1.0],) ([0.5, 0.0, -1.0],) ([0.5, -0.0, 1.0],) ([0.5, -0.0, -1.0],)] While I would have expected: [([0.5, 0.0, 1.0],) ([0.5, 0.0, -1.0],)] Can anyone confirm ? Nicolas

3 4

Just FYI: numpy-vbench was moved to another box, benchmarks are re-estimating
by Yaroslav Halchenko 22 Aug '14

22 Aug '14

I have no stats on either anyone is looking at http://yarikoptic.github.io/numpy-vbench besides me at times, so I might be just crying into the wild: I have moved running of numpy-vbench on a bit newer/more powerful box, and that is why benchmark results are being reestimated (thus you might still find some spikes and more noise than before for a bit longer). It is a dual Quad-Core AMD Opteron(tm) Processor 2384 with 32GB of RAM, so if someone is keen on pushing the benchmarks limits -- be my guest ;) -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Research Scientist, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

1 0

Numpy/Fortran puzzle (?)
by Andrea Gavana 21 Aug '14

21 Aug '14

Hi All, I have the following (very ugly) line of code: all_results = np.asarray([transm_hist[date_idx, :, idx_main_set[date_idx] ]*main_flow[date_idx, 0:n_fluids] for date_idx in xrange(n_dates)]) where transm_hist.shape = (n_dates, n_fluids, n_nodes), main_flow.shape = (n_dates, n_fluids) and idx_main_set is an array containing integer indices with idx_main_set.shape = (n_dates, ) . The resulting variable all_results.shape = (n_dates, n_fluids) Since that line of code is relatively slow if done repeatedly, I thought I'd be smart to rewrite it in Fortran and then use f2py to wrap the subroutine. So I wrote this: subroutine matmul(transm_hist, idx_main_set, main_flow, all_results, & n_dates, n_fluids, n_nodes) implicit none integer ( kind = 4 ), intent(in) :: n_dates, n_fluids, n_nodes real ( kind = 4 ), intent(in) :: transm_hist(n_dates, n_fluids, n_nodes) real ( kind = 4 ), intent(in) :: main_flow(n_dates, n_fluids) integer ( kind = 4 ), intent(in) :: idx_main_set(n_dates) real ( kind = 4 ), intent(out):: all_results(n_dates, n_fluids) integer (kind = 4) i, node do i = 1, n_dates node = int(idx_main_set(i)) all_results(i, :) = transm_hist(i, 1:n_fluids, node)*main_flow(i, 1:n_fluids) enddo end Unfortunately, it appears that I am not getting out quite the same results... I know it's a bit of a stretch with so little information, but does anyone have a suggestion on where the culprit might be? Maybe the elementwise multiplication is done differently in Numpy and Fortran, or I am misunderstanding what the np.asarray is doing with the list comprehension above? I appreciate any suggestion, which can also be related to improvement in the code. Thank you in advance. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net

2 1

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion August 2014