Mailman 3 August 2008 - NumPy-Discussion

patch for new mgrid / ogrid functionality
by David M. Kaplan 13 Sep '08

13 Sep '08

Hi, A while back, I sent some changes to index_tricks.py that would allow mgrid and ogrid to mesh things other than slices. For example: >>> mgrid[['a','b'],[float,int],:3] [array([[['a', 'a', 'a'], ['a', 'a', 'a']], [['b', 'b', 'b'], ['b', 'b', 'b']]], dtype='|S1'), array([[[<type 'float'>, <type 'float'>, <type 'float'>], [<type 'int'>, <type 'int'>, <type 'int'>]], [[<type 'float'>, <type 'float'>, <type 'float'>], [<type 'int'>, <type 'int'>, <type 'int'>]]], dtype=object), array([[[0, 1, 2], [0, 1, 2]], [[0, 1, 2], [0, 1, 2]]])] At the time, there wasn't much follow-up, but I am hoping that there is still interest in this functionality, as I have gone ahead and finished the patch including documentation changes and updates to test_index_tricks.py. Attached is a patch set to the latest subversion of the numpy trunk. I don't think I am allowed to commit the changes myself - correct me if I am wrong. This functionality seems like a nice addition to me as it allows one to mesh things that are not uniformly spaced and potentially not even numbers. The changes don't affect functionality that existed previously except for one minor change - instead of returning a numpy array of arrays, mgrid/ogrid now return a list of arrays. However, this is unlikely to be a problem as the majority of users generally unpack the results of mgrid/ogrid so that each matrix can be used individually. Comments welcome. Cheers, David -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html **********************************

3 4

Updated Numpy reference guide
by Pauli Virtanen 04 Sep '08

04 Sep '08

Hi all, I finished the first iteration of incorporating material from Travis Oliphant's "Guide to Numpy" to the Sphinxy reference guide we were constructing in the Doc marathon. Result is here: (the PDF is a bit ugly, though, some content is almost randomly scattered there) http://www.iki.fi/pav/tmp/numpy-refguide/index.xhtml http://www.iki.fi/pav/tmp/numpy-refguide/NumPy.pdf Source is here: (Stéfan, if it looks ok to you, could you pull and check if it builds for you when you have time?) https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide What I did with the "Guide to Numpy" material was: - Collapsed each of the reference Chapters 3, 6, 8, 9 (ndarrays, scalars, dtypes, ufuncs) with the more introductory material in Chapter 2. - As this was supposed to be a reference guide, I tried to compress the text from Chapter 2 as much as possible, by sticking to definitions and dropping some more tutorial-oriented parts. This may have reduced readability at some points... - I added some small bits or rewrote parts in the above sections in places where I thought it would improve the result. - I did not include material that I thought was better to be put into appropriate docstrings in Numpy. What to do with class docstrings and obscure __xxx__ attributes was not so clear a decision, so what I did for these varies. - The sections about Ufuncs and array indexing are taken almost verbatim from the "Guide to Numpy". The ndarray, scalar and dtype sections somewhat follow the structure of the Guide, but the text is more heavily edited from the original. Some things to do: - Descriptions about constructing items with __new__ methods should probably still be clarified; I just replaced references to __new__ with references to the corresponding classes. - What to do with the material from numpy.doc.* should be decided, as the text there doesn't look like it should go into a reference manual. Some questions: - Is this good enough to go into Numpy SVN at some point? Or should we redo it and base the work closer to the original "Guide to Numpy"? - Does it build for you? (I'd recommend using the development 0.5 version of Sphinx, so that you get the nifty Inter-Sphinx links to the Python documentation.) We are unfortunately beating the Sphinx with a big stick to make it place the documentation of each function or class into a separate file, and to convert the Numpy docstring format to something the Sphinx can fathom. There's also some magic in place to make toctrees:: of function listings more pleasant to the eye. Any comments of what should be improved are welcome. (Even better: clone the bzr branch, make the changes yourself, and put the result somewhere available! E.g. as a bzr bundle or a branch on the launchpad.) -- Pauli Virtanen

2 1

at my wits end over an error message...
by Alan Jackson 03 Sep '08

03 Sep '08

I been beating myself up over this bit of code for far too long now - I know I must be missing something really simple, but what is it? TYPICALLY_UINT_COLUMNS = ['Track', 'Bin', 'code', 'horizon'] ...... dtypes = [ ] for i in range(0, len(self.var_list)) : if TYPICALLY_UINT_COLUMNS.count(self.var_list[i]) > 0: dtypes.append((self.var_list[i], '<i8')) else : dtypes.append((self.var_list[i], '<f8')) print "dtypes = ", dtypes print outdata[0] outdata = np.array(outdata, dtype=dtypes) dtypes = [('Track', '<i8'), ('Bin', '<i8'), ('Depth', '<f8'), ('extrema', '<f8'), ('code', '<i8'), ('horizon', '<i8')] (345, 615, 4826.0, 470.588012695, 47, 636) Exception occurred in traits notification handler. Please check the log file for details. Exception occurred in traits notification handler for object: <__main__.MaplEdit object at 0x8bb54dc>, trait: Run, old value: <undefined>, new value: 0 Traceback (most recent call last): File "/usr/local/lib/python2.5/site-packages/enthought.traits-2.0.4-py2.5-linux-i686.egg/enthought/traits/trait_notifiers.py", line 325, in call_1 self.handler( object ) File "TrimMapl_1.py", line 98, in _Run_fired outdata = np.array(outdata, dtype=dtypes) TypeError: expected a readable buffer object Thanks, Alan -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan(a)ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | -----------------------------------------------------------------------

3 3

floating point char - bug?
by Christian K. 31 Aug '08

31 Aug '08

Hi, I just came across somethin I never noticed before. I cannot say whether this is due to an update of numpy but it is possible - I am running 1.1.1 on __german__ windows. Here is the observation: a = N.linspace(0,1,5) a array([ 0. , 0.25, 0.5 , 0.75, 1. ]) a.astype(float) array([ 0. , 0.25, 0.5 , 0.75, 1. ]) a[0].astype(float) 0.0 a[1].astype(float) 0,25 As you see in the last line, suddenly numpy picks up the german locale setting and converts the floating point into a comma. It does not affect the '0.0' in the line above and I believe to have seen some other numbers ending with '.0' where the point has not been replaced. I guess this is a bug. In fact I do not like the idea that repr() of a numpy float honours the locale settings. Reagrds, Christian

2 2

sort documentation
by Alan G Isaac 31 Aug '08

31 Aug '08

I find this confusing: numpy.sort(a, axis=-1, kind='quicksort', order=None) Return copy of 'a' sorted along the given axis. Perform an inplace sort along the given axis using the algorithm specified by the kind keyword. I suppose the last bit is supposed to refer to the ``sort`` method rather than the function, but I do not see any signal that this is the case. Cheers, Alan Isaac

5 7

Advice on converting iterator into array efficiently
by Alan Jackson 31 Aug '08

31 Aug '08

Looking for advice on a good way to handle this problem. I'm dealing with large tables (Gigabyte large). I would like to efficiently subset values from one column based on the values in another column, and get arrays out of the operation. For example, say I have 2 columns, "energy" and "collection". Collection is basically an index that flags values that go together, so all the energy values with a collection value of 18 belong together. I'd like to be able to set up an iterator on collection that would hand me an array of energy on each iteration : if table is all my data, then something like for c in table['collection'] : e = c['energy'] ... do array operations on e I've been playing with pytables, and they help, but I can't quite seem to get there. I can get an iterator for energy within a collection, but I can't figure out an efficient way to get an array out. What I have so far is for h in np.unique(table.col('collection')) : rows = table.where('collection == c') for row in rows : print c,' : ', row['energy'] but I really want to convert rows['energy'] to an array. I've thought about building a nasty set of pointers and whatnot - I did it once in perl - but I'm hoping to avoid that. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan(a)ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | -----------------------------------------------------------------------

3 5

huge array calculation speed
by Dan Lussier 30 Aug '08

30 Aug '08

Hello, I am relatively new to numpy and am having trouble with the speed of a specific array based calculation that I'm trying to do. What I'm trying to do is to calculate the total total potential energy and coordination number of each atom within a relatively large simulation. Each atom is at a position (x,y,z) given by a row in a large array (approximately 1e6 by 3) and presently I have no information about its nearest neighbours so each its position must be checked against all others before cutting the list down prior to calculating the energy. My current numpy code is below and in it I have tried to push as much of the work for this computation into compiled C (ideally) of numpy. However it is still taking a very long time at approx 1 second per row. At this speed even some simple speed ups like weave won't really help. Are there any other numpy ways that it would be possible to calculate this, or should I start looking at going to C/C++? I am also left wondering if there is something wrong with my installation of numpy (i.e. not making use of lapack/ATLAS). Other than that if it is relevant - I am running 32 bit x86 Centos 5.1 linux on a dual Xeon 3.0 GHz Dell tower with 4 GB of memory. Any suggestions of things to try would be great. Dan ======================================================================== ======================= def epairANDcoord(xyz,cutoff=4.0): box_length = 160.0 cut2 = cutoff**2 ljTotal = numpy.zeros(xyz.shape[0]) coord = numpy.zeros(xyz.shape[0]) i = 0 for r0 in xyz: r2 = r0-xyz # application of minimum image convention so each atom interacts with nearest periodic image w/i box_length/2 r2 -= box_length*numpy.rint(r2/box_length) r2 = numpy.power(r2,2).sum(axis=1) r2 = numpy.extract(r2<cut2, r2) coord[i] = r2.shape[0]-1 r2 = numpy.extract(r2 != 0, r2) # avoid problems of inf when considering own atom. r2 = numpy.divide(1.0,r2) ljTotal[i] = numpy.multiply(2,(numpy.power(r2,6)- numpy.power(r2,3))).sum(axis=0) # ljTotal[i] = ljPerAtom.sum(axis=0) i += 1 del r2 # if i>25: break # used to cut down the length of calculation as needed return ljTotal,coord

11 13

How do I do this?
by Christopher Barker 30 Aug '08

30 Aug '08

HI all, I need to do something I thought would be simple -- set all the values of an array to some minimum. so I did this: >>> min_value = 2 >>> a = np.array((1, 2, 3, 4, 5,)) >>> np.maximum(a, min_value) array([2, 2, 3, 4, 5]) all was well... then I realized that a could have negative numbers in in, and I really wanted the absolute value to be greater than than minimum: >>> a = np.array((1, 2, 3, 4, -5,)) >>> np.maximum(a, min_value) array([2, 2, 3, 4, 2]) oops! so I added a sign() and abs(): >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 2, 2, 3, 4, -5]) all was well. However it turns out a could contain a zero: >>> a = np.array((0, 1, 2, 3, 4, -5,)) >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 0, 2, 2, 3, 4, -5]) Darn! I want that zero to become a 2, but sign(0) = 0, so that doesn't work. How can I do this without another line of code special casing the 0, which isn't that big I deal, but it seems kind of ugly... >>> a[a==0] = min_value >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 2, 2, 2, 3, 4, -5]) thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

4 16

default index inconsistency?
by Alan G Isaac 30 Aug '08

30 Aug '08

I think the question of the timing of removing the remaining default index inconsistencies in NumPy was raised by not answered? Anyway, what is the answer? I just ran into this again, this time with ``argsort``, which still has ``axis=-1`` as the default. But maybe it is intentional to keep this? E.g., a.sort(axis=None) is refused. I would have expected it to return the same as np.sort(a.flat), possibly (?) reshaped. Alan Isaac

1 0

np.test() Bus error on OSX. Reopened Ticket 816
by Christopher Burns 29 Aug '08

29 Aug '08

I reopened Ticket 816: http://scipy.org/scipy/numpy/ticket/816 Running numpy.test() causes a bus error on OSX. -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

2 1