Mailman 3 March 2010 - NumPy-Discussion

Is this a bug in numpy.ma.reduce?
by David Goldsmith March 8, 2010

March 8, 2010

Hi! Sorry for the cross-post, but my own investigation has led me to suspect that mine is actually a numpy problem, not a matplotlib problem. I'm getting the following traceback from a call to matplotlib.imshow: Traceback (most recent call last): File "C:\Users\Fermat\Documents\Fractals\Python\Source\Zodiac\aquarius_test.py", line 108, in <module> ax.imshow(part2plot, cmap_name, extent = extent) File "C:\Python254\lib\site-packages\matplotlib\axes.py", line 6261, in imshow im.autoscale_None() File "C:\Python254\lib\site-packages\matplotlib\cm.py", line 236, in autoscale_None self.norm.autoscale_None(self._A) File "C:\Python254\lib\site-packages\matplotlib\colors.py", line 792, in autoscale_None if self.vmin is None: self.vmin = ma.minimum(A) File "C:\Python254\Lib\site-packages\numpy\ma\core.py", line 5555, in __call__ return self.reduce(a) File "C:\Python254\Lib\site-packages\numpy\ma\core.py", line 5570, in reduce t = self.ufunc.reduce(target, **kargs) ValueError: zero-size array to ufunc.reduce without identity Script terminated. Based on examination of the code, the last self is an instance of ma._extrema_operation (or one of its subclasses) - is there a reason why this class is unable to deal with a "zero-size array to ufunc.reduce without identity," (i.e., was it thought that it would - or should - never get one) or was this merely an oversight? Either way, there's other instances on the lists of this error cropping up, so this circumstance should probably be handled more robustly. In the meantime, workaround? DG

5 14

Why is the shape of a singleton array the empty tuple?
by David Goldsmith March 8, 2010

March 8, 2010

>>> x = numpy.array(3) >>> x array(3) >>> x.shape () My question is: why? DG

3 6

Iterative Matrix Multiplication
by Ian Mallett March 6, 2010

March 6, 2010

Hi, I have a list of vec3 lists (e.g. [[1,2,3],[4,5,6],[7,8,9],...]). To every single one of the vec3 sublists, I am currently applying transformations. I need to optimize this with numpy. To get proper results, as far as I can tell, the vec3 lists must be expressed as vec4s: [[1,2,3],[4,5,6],[7,8,9],...] -> [[1,2,3,1],[4,5,6,1],[7,8,9,1],...]. Each of these needs to be multiplied by either a translation matrix, or a rotation and translation matrix. I don't really know how to approach the problem . . . Thanks, Ian

5 21

Re: [Numpy-discussion] Building Numpy Windows Superpack
by Patrick Marsh March 6, 2010

March 6, 2010

Greetings I sent this last night but the buildout.txt file was too large so my email was held pending moderation. When I got up this morning I realized that there was a good chance that the moderation queue is overwhelmed and not looked at much so I decided to post links to the out files and resubmit. I apologize if any of you get this twice... Okay, I'm about out of ideas. Hopefully someone on here has an idea as to what might be going on. 1. I am still unable to build the windows superpack using pavement.py, even after making sure I have all the necessary dependencies. However, the good news is that pavement.py now recognizes where the Atlas binaries David provided are located. The output from "paver bdist_wininst" is located here http://patricktmarsh.com/numpy/20100302.paveout.txt. 2. Since I couldn't get pavement.py to work, I decided to try and build Numpy with sse3 support using "python25 setup.py build -c mingw32 bdist_wininst > buildout.txt". (Note, I've also done this for building with Python 2.6.) This works and I'm able to build the windows installer for both Python 2.5 and 2.6. (The output of the build from Python 2.5 is here http://patricktmarsh.com/numpy/20100302.buildout.txt. However, when I try to run the test suite (using "python25 -c 'import numpy; print numpy.__version__; numpy.test();' > testout.txt"), Python 2.5 runs, with failures & errors, whereas Python 2.6 freezes and eventually (Python itself) quits. Since I'm unable to generate the test suite log for Numpy on Python 2.6, I'm working on the (incorrect?) assumption that the freezing when using Python 2.6 corresponds with the failures/errors when using Python 2.5. The Python 2.5 numpy test suite log is located here: http://patricktmarsh.com/numpy/20100302.testout.txt. Most of the errors with my locally build numpy version comes from the test suite being unable to call "matrix". One, of many, examples is attached below. I don't know where else to look anymore. I get the same results when building from trunk and building from the 1.4.x tag (which I downloaded this morning). I've gotten these same results when building on a different laptop (which was a 32bit version of Windows 7 Professional). I should disclose that I'm currently running a 64bit version of Windows 7 Professional. I'm running EPDv5.1.1 32bit for Python 2.5.4 (shipped numpy test suite works fine). I'm running EPDv6.1.1 32bit for Python 2.6.4 (shipped numpy test suite works fine). I've achieved the same errors when using a generic 32bit Python install downloaded from the official Python website. I'm hoping that I've looked at this for so long that I'm missing something obvious. Any thoughts/suggestions at this point would be appreciated. ====================================================================== ERROR: Test whether matrix.sum(axis=1) preserves orientation. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\lib\site-packages\numpy\core\tests\test_defmatrix.py", line 56, in test_sum M = matrix([[1,2,0,0], NameError: global name 'matrix' is not defined ====================================================================== Patrick -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com

5 11

Loading bit strings
by Dan Lenski March 5, 2010

March 5, 2010

Is there a good way in NumPy to convert from a bit string to a boolean array? For example, if I have a 2-byte string s='\xfd\x32', I want to get a 16-length boolean array out of it. Here's what I came up with: A = fromstring(s, dtype=uint8) out = empty(A.size * 8, dtype=bool) for bit in range(0,8): out[bit::8] = A&(1<<bit) I just can't shake the feeling that there may be a better way to do this, though... Dan

3 2

Why does np.nan{min, max} clobber my array mask?
by David Carmean March 5, 2010

March 5, 2010

I'm just starting to work with masked arrays and I've found some behavior that definitely does not follow the Principle of Least Surprise: I've generated a 2-d array from a list of lists, where the elements are floats with a good number of NaNs. Inspections shows the expected numbers for ma.count() and ma.count_masked(). However, as soon as I run np.nanmin() or np.nanmax() over it, all of the mask elements are reset to False. (Pdb) flat = flatten(uut) # my own utility function (Pdb) len ( [ x for x in flat if x+0 == x ] ) # only way I could figure to detect 4086 (Pdb) len ( [ x for x in flat if x+0 != x ] ) # 1458 NaNs in the set. 1458 (Pdb) msk = ma.masked_invalid(uut) (Pdb) msk.shape (99, 56) (Pdb) ma.count(msk) 4086 (Pdb) ma.count_masked(msk) 1458 (Pdb) msk.hardmask False (Pdb) msk.harden_mask() # harden the mask first, for demo masked_array(data =.... (Pdb) msk.hardmask True (Pdb) rslt_hm = np.nanmin(msk, axis=1) (Pdb) rslt_hm.shape (99,) (Pdb) ma.count_masked(rslt_hm) 0 (Pdb) ma.count(rslt_hm) 99 # Is my original still OK? msk masked_array(data = ... ... [False False False ..., True True True]], fill_value = 1e+20) (Pdb) msk.soften_mask() # now re-soften the mask: masked_array(data = .... (Pdb) rslt_softmask = np.nanmin(msk, axis=1) (Pdb) rslt_softmask.shape (99,) (Pdb) msk.mask.any() False # BAM! note: 'control' is a hardmasked control copy: (Pdb) control.mask.any() True As the above shows, I discovered that I can work around this by setting the hardmask property, but ... there is no mention of such a side-effect in the docs (including the brand-new reference book). Have I found a bug? This is 1.4.0 running under 64-bit Windows 7 ( Python(x,y) distribution).

3 5

memory usage of numpy arrays
by kaby March 5, 2010

March 5, 2010

Hi. I am using numpy arrays and when constructing an array I get a "cannot allocate memory for thread-local data: ABORT" The array i'm constructing is zeros((numVars, 2, numVars, 2), dtype=float) Where numVars is at about 2000. I was expecting the memory usage to be 2000*2000*2*2*8Bytes=128.000.000Bytes=128MBytes So why is that happening? What am I missing? Thanks for any help. -- View this message in context: http://old.nabble.com/memory-usage-of-numpy-arrays-tp27788048p27788048.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 1

how to efficiently build an array of x, y, z points
by Brennan Williams March 4, 2010

March 4, 2010

I'm reading a file which contains a grid definition. Each cell in the grid, apart from having an i,j,k index also has 8 x,y,z coordinates. I'm reading each set of coordinates into a numpy array. I then want to add/append those coordinates to what will be my large "points" array. Due to the orientation/order of the 8 corners of each hexahedral cell I may have to reorder them before adding them to my large points array (not sure about that yet). Should I create a numpy array with nothing in it and then .append to it? But this is probably expensive isn't it as it creates a new copy of the array each time? Or should I create a zero or empty array of sufficient size and then put each set of 8 coordinates into the correct position in that big array? I don't know exactly how big the array will be (some cells are inactive and therefore don't have a geometry defined) but I do know what its maximum size is (ni*nj*nk,3). Thanks Brennan

4 7

dtype for a single char
by sam tygier March 4, 2010

March 4, 2010

Hello today i was caught out by trying to use 'a' as a dtype for a single character. a simple example would be: >>> array([('a',1),('b',2),('c',3)], dtype=[("letter", "a"), ("number", "i")]) array([('', 1), ('', 2), ('', 3)], dtype=[('letter', '|S0'), ('number', '<i4')]) the fix seems to be using 'a1' instead >>> array([('a',1),('b',2),('c',3)], dtype=[("letter", "a1"), ("number", "i")]) array([('a', 1), ('b', 2), ('c', 3)], dtype=[('letter', '|S1'), ('number', '<i4')]) this seems odd to me, as other types eg 'i' and 'f' can be used on their own. is there a reason for this? thanks Sam

3 5

combinatorics
by Ernest Adrogué March 4, 2010

March 4, 2010

Hello everybody, Suppose I want to find all 2-digit numbers whose first digit is either 4 or 5, the second digit being 7, 8 or 9. Is there a Numpy/Scipy function to calculate that kind of combinations? I came up with this function, the problem is it uses recursion: def g(sets): if len(sets) < 2: return sets return [[i] + j for i in sets[0] for j in f(sets[1:])] In [157]: g([[4,5],[7,8,9]]) Out[157]: [[4, 7], [4, 8], [4, 9], [5, 7], [5, 8], [5, 9]] Is important that it works with more than two sets too. Any idea is appreciated. Ernest

2 3