Mailman 3 December 2010 - NumPy-Discussion

savetxt to a string?
by Neal Becker 22 Dec '10

22 Dec '10

Is the formatting of savetxt available with a string as the destination? If not, shouldn't this functionality be factored out of savetxt?

2 4

use of the PyArray_SETITEM numpy method
by Michel Dupront 22 Dec '10

22 Dec '10

Hello, Please somebody help me ! I am really confused with all these void that I see everywhere. The documentation says: PyObject * PyArray_SETITEM(PyObject * arr, void* itemptr, PyObject* obj) I created a 1D array of doubles with PyArray_SimpleNew. Now I am in a loop of index i to feed my array with values. What should I give for itemptr ? Thank you

2 1

take from structured array is faster than boolean indexing, but reshapes columns to 2D
by Christopher Mutel 22 Dec '10

22 Dec '10

Dear all- Structured arrays are great, but I am having problems filtering them efficiently. Reading through the mailing list, it seems like boolean arrays are the recommended approach to filtering arrays for arbitrary conditions, but my testing shows that a combination of take and where can be much faster when dealing with structured arrays: import timeit setup = "from numpy import random, where, zeros; r = random.random_integers(1e3, size=1e6); q = zeros((1e6), dtype=[('foo', 'u4'), ('bar', 'u4'), ('baz', 'u4')]); q['foo'] = r" statement1 = "s = q.take(where(q['foo'] < 500))" statement2 = "s = q[q['foo'] < 500]" t = timeit.Timer(statement1, setup) t.timeit(10) t = timeit.Timer(statement2, setup) t.timeit(10) Using the boolean array is about 4 times slower when dealing with large arrays. In my case, these operations are supposed to happen on a web server with a large number of requests, so the efficiency gain is important. However, the combination of take and where reshapes the columns of structured arrays to be 2-dimensional: q['foo'].shape >> (1000000,) s = q[q['foo'] < 500] s['foo'].shape >> (499102,) s = q.take(where(q['foo'] < 500)) s['foo'].shape >> (1, 499102) Is there a way to use this seemingly more efficient approach (take & where) and not have to manually reshape the columns? This seems ungainly for larger structured arrays. Or should I file this as a bug? Perhaps there are even more efficient approaches that I haven't thought of, but are obvious to others? Thanks in advance, Yours, -Chris -- ############################ Chris Mutel Ökologisches Systemdesign - Ecological Systems Design Institut f.Umweltingenieurwissenschaften - Institute for Environmental Engineering ETH Zürich - HIF C 42 - Schafmattstr. 6 8093 Zürich Telefon: +41 44 633 71 45 - Fax: +41 44 633 10 61 ############################

1 0

sample without replacement
by Alan G Isaac 22 Dec '10

22 Dec '10

I want to sample *without* replacement from a vector (as with Python's random.sample). I don't see a direct replacement for this, and I don't want to carry two PRNG's around. Is the best way something like this? permutation(myvector)[:samplesize] Thanks, Alan Isaac

6 9

Reversing an array in-place
by Justin Peel 22 Dec '10

22 Dec '10

I noticed that there is currently no way to reverse a numpy array in-place. The current way to reverse a numpy array is using slicing, ala arr[::-1]. This is okay for small matrices, but for really large ones, this can be prohibitive. Not only that, but an in-place reverse is much faster than slicing. It seems like a reverse method could be added to arrays that would reverse the array along a given axis fairly easily. Is there any opposition to this? Also, there is consideration for simply marking a given axis as being reverse similar to how transposes are taken. However, I see this as a problem for a method like reshape to deal with and therefore think that it is better to just add a reverse method. What are your opinions? I'm quite willing to make such a method if it will be accepted. Justin Peel

3 3

Links to doc guidelines broken
by Fernando Perez 21 Dec '10

21 Dec '10

Howdy, In the ipython doc guide (and many other places) we point to the numpy coding guidelines (especially for documentation), but today while conducting a sprint at the Scipy India conference, I noticed this link is now dead: http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines It seems the docs got moved over to github, which is fine: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt but it would be really nice if whoever went through the Trac wiki deleting stuff could have left a link pointing to the new proper location of this document. For those who knew what they were looking for it's easy enough to find it againg with a bit of googling around, but newcomers who may be trying to read these guidelines and simply get Trac's version of a 404 are likely to be left confused. I realize that in moving to a new infrastructure broken links are hard to avoid, but for documents as widely used as the numpy coding guidelines, perhaps leaving a link to the new location in the old location would be a good idea... I fixed the page with a github link, but there may be other important pages needing a similar treatment, and I think as a policy it's generally a good idea to leave proper redirects when important pages are deleted. Thanks f

2 1

Short circuiting the all() and any() methods/functions
by Justin Peel 21 Dec '10

21 Dec '10

It has come to my attention that the all() and any() methods/functions do not short circuit. It takes nearly as much time to call any() on an array which has 1 as the first entry as it does to call it on an array of the same size full of zeros. The cause of the problem is that all() and any() just call reduce() with the appropriate operator. Is anyone opposed to changing the implementations of these functions so that they short-circuit? By the way, Python already short circuits all() and any() correctly so it certainly makes sense to enact this change. I'm willing to head this up if there isn't any opposition to it. Justin Peel

3 3

creating zonal statistics from two arrays
by Gregory, Matthew 21 Dec '10

21 Dec '10

Hi all, Likely a very newbie type of question. I'm using numpy with GDAL to calculate zonal statistics on images. The basic approach is that I have a zone raster and a value raster which are aligned spatially and I am storing each zone's corresponding values in a dictionary, then calculating the statistics on that population. (I'm well aware that this approach may have memory issues with large rasters ...) GDAL ReadAsArray gives you a chunk of raster data as a numpy array. Currently I'm iterating over rows and columns of that chunk, but I'm guessing there's a better (and more numpy-like) way. zone_stats = {} zone_block = zone_band.ReadAsArray(x_off, y_off, x_size, y_size) value_block = value_band.ReadAsArray(x_off, y_off, x_size, y_size) for row in xrange(y_size): for col in xrange(x_size): zone = zone_block[row][col] value = value_block[row][col] try: zone_stats[zone].append(value) except KeyError: zone_stats[zone] = [value] # Then calculate stats per zone ... Thanks for all suggestions on how to make this better, especially if the initial approach I'm taking is flawed. matt

3 2

same name and title in structured arrays
by Kathleen M Tacina 20 Dec '10

20 Dec '10

Hi, I've been finding numpy/scipy/matplotlib a very useful tool for data analysis. However, a recent change has caused me some problems. Numpy used to allow the name and title of a column of a structured array or recarray to be the same (at least in the svn version as of early last winter). Now, it seems that this is not allowed; see below. Python 2.6.5 (r265:79063, Apr 27 2010, 12:20:23) [GCC 4.2.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '2.0.0.dev-799179d' >>> data = np.ndarray((5,1),dtype=[(('T','T'),float)]) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: title already used as a name or title. >>> data = np.ndarray((5,1),dtype=[(('T at 0.25-in in F','T'),float)]) >>> Would it be possible to change the tests to allow the name and title to be the same for the same component? I can work around this new limitation for new data files, but I'm having trouble reading data files I created last winter. (But even for new stuff, it would be nice if it name=title was allowed. I like using both names and titles so that I can record interesting information like point location and units in the titles. Sometimes, though, there isn't anything else interesting to say about a column (e.g., 'point','date', 'time'), and I'd like to (1) have both a title and a name to be consistent with more interesting columns and (2) have the title be equal to the name.) Thanks for any help you can give me with this! Kathy Tacina

2 1

Efficient Matrix-matrix product of hermitian matrices, zhemm (blas) and numpy
by qubax＠gmx.at 19 Dec '10

19 Dec '10

I need to calculate several products of matrices where at least one of them is always hermitian. The function zhemm (in blas, level 3) seems to directly do that in an efficient manner. However ... how can i access that function and dirctly apply it on numpy arrays? If you know alternatives that are equivalent or even faster, please let me know. Any help is highly appreciated. Q -- The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity.

1 0