Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great. OK: how do i iterate over an axis other than 0? I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead? thanks, alex.
for i in range(52): week_data = data[:,i,:] OR for week_data in data.transpose(1,0,2): ... Nadav -----הודעה מקורית----- מאת: numpy-discussion-bounces@scipy.org בשם a g נשלח: ד 30-אפריל-08 11:11 אל: numpy-discussion@scipy.org נושא: [Numpy-discussion] very simple iteration question. Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great. OK: how do i iterate over an axis other than 0? I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead? thanks, alex. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
2008/4/30 a g
Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great.
OK: how do i iterate over an axis other than 0?
I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead?
Well, there's always for i in xrange(A.shape[1]): do_something(A[:,i,...]) But that's kind of ugly in this day and age. I would be tempted by for a in np.rollaxis(A,1): do_something(a) It's worth mentioning that traversing an array along an axis not the first usually results in subarrays that are not contiguous in memory. While numpy makes working with these convenient, they may not be very efficient for cache reasons (for example, modern processors load from memory - an extremely slow operation - in blocks that may be as large as 64 bytes; if you only use eight of these before moving on, your code will use much more memory bandwidth than it needs to). If this is a concern, you can usually transparently rearrange your array's memory layout to avoid it. Anne
Hi Alex, a g wrote:
Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great.
You should look at any of the documents below and read up on array slicing. It is perhaps the most important and pervasive concept of Numpy and should be understood by all users. Numpy Tutorial: http://www.scipy.org/Tentative_NumPy_Tutorial Numpy for MATLAB users: http://www.scipy.org/NumPy_for_Matlab_Users Guide to Numpy
OK: how do i iterate over an axis other than 0?
I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead?
It is not clear to me whether you want to slice or iterate over an array. Assuming you are fixing the year and location, the following code iterates over data for fixed year and location. for week in xrange(0, 52): <do something with> data[year, week, loc] Slicing is more efficient and you should use it if you can. Fixing the year and location, the following computes the mean and standard deviation across all weeks. All of the statements below yield scalars. data[year, :, loc].mean() -- takes the mean of the data across weeks data[year, :, loc].std() -- takes the standard deviation of the data across weeks You should download IPython and type help(numpy.array) to see one set of functions you can call on the result of a slice (sum, min, etc.). Although I don't know what statistics you are computing for sure, the following code might be useful since it computes a statistic across all weeks for each year and location value. data.mean(axis=1) It yields a num_years by num_locations array mu where mu[y, l] is the average data value across all weeks for year y and loc l. I hope this helps. Damian
a g wrote:
OK: how do i iterate over an axis other than 0?
This ties in nicely with some of the discussion about interating over matrices. It ahs been suggested that it would be nice to have iterators for matrices, so you could do: for row in M.rows: ... and for column in M.cols: ... If so, then wouldn't it make sense to have built in iterators for nd-arrays as well? something like: for subarray in A.iter(axis=i): ... where axis would default to 0. This is certainly cleaner than: for j in range(A.shape[i]): subarray = A[:,:,j,:] Wait! I have no idea how to spell that generically -- i.e. the ith index, where i is a variable. There must be a way to build a slice object dynamically, but I don't know it. How often would i be a variable, rather than hard coded? I have no idea. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
2008/4/30 Christopher Barker
a g wrote:
OK: how do i iterate over an axis other than 0?
This ties in nicely with some of the discussion about interating over matrices. It ahs been suggested that it would be nice to have iterators for matrices, so you could do:
for row in M.rows: ...
and
for column in M.cols: ...
If so, then wouldn't it make sense to have built in iterators for nd-arrays as well? something like:
for subarray in A.iter(axis=i): ...
where axis would default to 0.
This is certainly cleaner than:
for j in range(A.shape[i]): subarray = A[:,:,j,:]
Wait! I have no idea how to spell that generically -- i.e. the ith index, where i is a variable. There must be a way to build a slice object dynamically, but I don't know it.
Slices can be built without too much trouble using slice(), but it's much easier to just write for subarray in np.rollaxis(A,i): ... rollaxis() just pulls the specified axis to the front. (It doesn't do what I thought it would, which is a cyclic permutation of the axes). Anne
Anne Archibald wrote:
it's much easier to just write
for subarray in np.rollaxis(A,i): ...
cool, thanks! So the answer to the OPs question:
OK: how do i iterate over an axis other than 0?
I have a 3D array of data[year, week, location]. I want to iterate over each year at each location ...
for loc in np.rollaxis(data, 2): for year in np.rollaxis(data, 0): # rollaxis not required here, but # for symmetry's sake... .... I think I still like the idea of an iterator (or maybe making rollaxis a method?), but this works pretty well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
I think I still like the idea of an iterator (or maybe making rollaxis a method?), but this works pretty well.
Generally, in object oriented programming, you expect a method like rollaxis to modify an object inplace. At least that would be my expectation. Gaël
On Wed, Apr 30, 2008 at 3:09 PM, Gael Varoquaux
On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
I think I still like the idea of an iterator (or maybe making rollaxis a method?), but this works pretty well.
Generally, in object oriented programming, you expect a method like rollaxis to modify an object inplace. At least that would be my expectation.
Gaël, I'm not sure where you learned this expectation, as I don't think (but I could be wrong -- so educate me!) it is universal for OOP nor encouraged for Python in particular. OOP promotes encapsulation by providing mechanisms for objects to publish an interface and thereby hide the details of their inner workings, but I don't think there is a common OOP philosophy that disallows methods returning views and requires all transformations be performed in-place. A design philosophy like you've espoused is tractable in languages that support overloading, like C++, but it is not tractable in Python (at least not cleanly within the design of the base language). How do you create a function that returns a "flat" iterator of a container generically? In C++, each container would overload the flat function. In Python, your only hope is to ensure that the flat function only requires a standardized interface to a container object, but that would likely depend on the standardized interface including some low level iteration method from which to build more complex iterators. Even in C++ where such a design philosophy is possible at some level, objects often expose const iterators of various sorts as methods. Finally, Python has plenty of counter-examples to the maxim: all the string methods because strings are designed to be immutable, set object methods (even for mutable sets), various iterators for dicts, etc. On this topic. I would love to see numpy evolve a moderately generic array interface so that we could write stand-alone functions that work generically with ndarrays, as well as "masked arrays" and "sparse arrays" so that there was "one dot-product to rule them all", so to speak. Right now, you can't use functions like numpy.dot on a numpy.ma.MaskedArray, for example. Regards, Alex
On Wed, 2008-04-30 at 21:09 +0200, Gael Varoquaux wrote:
On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
I think I still like the idea of an iterator (or maybe making rollaxis a method?), but this works pretty well.
Generally, in object oriented programming, you expect a method like rollaxis to modify an object inplace. At least that would be my expectation.
BTW. rollaxis isn't a method. I was completely unaware of this function. Learned something new today... BC
Gaël
participants (8)
-
a g -
Alexander Michael -
Anne Archibald -
Bryan Cole -
Christopher Barker -
Damian Eads -
Gael Varoquaux -
Nadav Horesh