Mailman 3 January 2009 - NumPy-Discussion

Examples for numpy.genfromtxt
by Nils Wagner 21 Jan '09

21 Jan '09

Hi all, Where can I find some sophisticated examples for the usage of numpy.genfromtxt ? Nils

2 1

Testing for close to zero?
by Jonathan Taylor 20 Jan '09

20 Jan '09

Hi, When solving a quadratic equation I get that alpha = -3.78336776728e-31 which I believe to be far below machine precision: finfo(float).eps 2.2204460492503131e-16 But an if statement like: if alpha == 0: ... does not catch this. Is there a better way to check for things that are essentially zero or should I really be using if np.abs(alpha) < finfo(float).eps: ... ? Thanks for any help. Jonathan.

3 9

Re: [Numpy-discussion] ANN: Numexpr 1.1, an efficient array evaluator
by Francesc Alted 20 Jan '09

20 Jan '09

A Friday 16 January 2009, jh(a)physics.ucf.edu escrigué: > Hi Francesc, > > > Numexpr is a fast numerical expression evaluator for NumPy. With > > it, expressions that operate on arrays (like "3*a+4*b") are > > accelerated and use less memory than doing the same calculation in > > Python. > > Please pardon my ignorance as I know this project has been around for > a while. It this looks very exciting, but either it's cumbersome, or > I'm not understanding exactly what's being fixed. If you can > accelerate evaluation, why not just integrate the faster math into > numpy, rather than having two packages? Or is this something that is > only an advantage when the expression is given as a string (and why > is that the case)? It would be helpful if you could put the answer > on your web page and in your standard release blurb in some compact > form. I guess what I'm really looking for when I read one of those is > a quick answer to the question "should I look into this?". Well, there is a link in the project page to the "Overview" section of the wiki, but perhaps is a bit hidden. I've added some blurb as you suggested in the main page an another link to the "Overview" wiki page. Hope that, by reading the new blurb, you can see why it accelerates expression evaluation with regard to NumPy. If not, tell me and will try to come with something more comprehensible. > Right > now, I'm not quite sure whether the problem you are solving is merely > the case of expressions-in-strings, and there is no advantage for > expressions-in-code, or whether your expressions-in-strings are > faster than numpy's expressions-in-code. In either case, it would > appear this would be a good addition to the numpy core, and it's past > 1.0, so why keep it separate? Even if there is value in having a > non-numpy version, is there not also value in accelerating numpy by > default? Having the expression encapsulated in a string has the advantage that you exactly know the part of the code that you want to parse and accelerate. Making NumPy to understand parts of the Python code that can be accelerated sounds more like a true JIT for Python, and this is something that is not trivial at all (although, with the advent of PyPy there are appearing some efforts in this direction [1]). [1] http://www.enthought.com/~ischnell/paper.html Cheers, -- Francesc Alted

4 6

How to make "lazy" derived arrays in a recarray view of a memmap on large files
by Kim Hansen 19 Jan '09

19 Jan '09

Hi numpy forum I need to efficiently handle some large (300 MB) recordlike binary files, where some data fields are less than a byte and thus cannot be mapped in a record dtype immediately. I would like to be able to access these derived arrays in a memory efficient manner but I cannot figure out how to acheive this. My application of the derived arrays would never be to do operation on the entire array, rather iterate over some selected elements and do somthing about it - operations which seems well suited for doing on demand I wrote a related post yesterday, which I have not received any response on. I am now posting again using another example and perhaps more clear example which I beleive describes my problem spot on from numpy import * # Python.exe memory use here: 8.14 MB desc = dtype([("baconandeggs", "<u1"), ("spam","<u1"), ("parrots","<u1")]) index = memmap("g:/id-2008-10-25-17-ver4.idx", dtype = desc, mode="r").view(recarray) # The index file is very large, contains 292 MB of data # Python.exe memory use: 8.16 MB, only 20 kB extra for memmap mapped to recarray # The following instant operation takes a few secs working on 3*10^8 elements # How can I derive new array in a lazy/ondemand/memmap manner? index.bacon = index.baconandeggs >> 4 # python.exe memory use: 595 MB! Not surprising but how to do better?? # Another derived array, which is resource demanding index.eggs = index.baconandeggs & 0x0F # python.exe memory usage is now 731 MB! What I'd like to do is implement a class, LazyBaconEggsSpamParrots, which encapsulates the derived arrays such that I could do besp = LazyBaconEggsSpamParrots("baconeggsspamparrots.idx") for b in besp.bacon: #Iterate lazy spam(b) #Only derive the 1000 needed elements, don't do all 1000000 dosomething(besp.bacon[1000000:1001000]) I envision the class would look something like this class LazyBaconEggsSpamParrots(object): def __init__(self, filename): desc = dtype([("baconandeggs", "<u1"), ("spam","<u1"), ("parrots","<u1")]) self._data = memmap(filename, dtype=desc, mode='r').view(recarray) # Expose the one-to-one data directly self.spam = self._data.spam self.parrots = self._data.parrots # This would work but costs way too much memory # self.bacon = self._data.baconandeggs >> 4 # self.eggs = self._data.baconandeggs & 0x0F def __getattr__(self, attr_name): if attr_name == "bacon": # return bacon in an on demand manner, but how? elif attr_name == "eggs": # return eggs in an on demand manner, but how? else: # If the name is not a data attribute treat it as a normal # non-existing attribute - raise AttributeError raise AttributeError but how to do the lazy part of it? -- Kim

2 2

question about memory ownership
by Darren Dale 19 Jan '09

19 Jan '09

Hello, Would someone be so kind as to explain how to create an ndarray subclass that owns its own memory? I think RealisticInfoArray at http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclas… not own its own memory, do you have to call ndarray.__new__ directly, or is there another way? Thanks, Darren

2 5

Help with interpolating missing values from a 3D scanner
by David Bolme 19 Jan '09

19 Jan '09

I am working on a face recognition using 3D data from a special 3D imaging system. For those interested the data comes from the FRGC 2004 dataset. The problem I am having is that for some pixels the scanner fails to capture depth information. The result is that the image has missing values. There are small regions on the face such as eyebrows and eyes that are missing the depth information. I would like to fill in these region by interpolating from nearby pixels but I am not sure of the best way to do that. I currently have two arrays: * floating point array with depth information (missing data is encoded as a large negative number -999999.0) * boolean array that is a missing data mask I have some ideas on how to solve this problem but I am hoping that someone on this list may have more experience with this type of missing data problem. * Are there any utilities in scipy/numpy designed for this type of missing data problem? * If not does any one have suggestions on how I should proceed?

4 5

efficient usage of tensordot
by Gideon Simpson 18 Jan '09

18 Jan '09

This is related to a question I posted earlier. Suppose I have array A with dimensions n x m x l and array x with dimensions m x l. Interpret this as an array of l nxm matrices and and array of l m dimensional vectors. I wish to compute the matrix- vector product A[:,:,k] x[:,k] for each k = 0,... l -1. I discovered that I could accomplish this with the command np.diagonal(np.tensordot(A, k, axes=(1,0)), axis1= 1, axis2 = 2) The tensordot command gives me A_{ijk}x_{jl} = C_{ikl} And the diagonal command grabs the entries in array C where k=l. Is this the "optimal" way to make this calculation in numpy? It certainly makes for nice, clean code, but is it the fastest I can get? -gideon

2 1

tensor contractions
by Gideon Simpson 17 Jan '09

17 Jan '09

Suppose I have a 3d array, A, with dimensions 2 x 2 x N, and a 2d 2 x N array, u. I interpret A as N 2x2 matrices and u as N 2d vectors. Suppose I want to apply the mth matrix to the mth vector, i.e. A[, , m] u[, m] = v[, m] Aside from doing A[0,0,:] u[0,:] + A[0,1,:] u[1,:] = v[0,:] and A[1,0,:] u[0,:] + A[1,1,:] u[1,:] = v[1,:] is there a smart way to perform this computation? -gideon

1 0

Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64)
by George Goussard 17 Jan '09

17 Jan '09

Hello. I have been battling with the following error for the past week. The output from the terminal is: Traceback (most recent call last): File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\backends\backend_qt4agg.py", line 86, in paintEvent FigureCanvasAgg.draw(self) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\backends\backend_agg.py", line 261, in draw self.figure.draw(self.renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\figure.py", line 765, in draw legend.draw(renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 197, in draw self._update_positions(renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 513, in _update_positions l,b,w,h = get_tbounds(self.texts[-1]) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 499, in get_tbounds bboxa = bbox.inverse_transformed(self.get_transform()) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\transforms.py", line 478, in inverse_transformed return Bbox(transform.inverted().transform(self.get_points())) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\transforms.py", line 1338, in inverted self._inverted = Affine2D(inv(mtx)) File "C:\development\Python\2_5_2\Lib\site-packages\numpy\linalg\linalg.py", line 350, in inv return wrap(solve(a, identity(a.shape[0], dtype=a.dtype))) File "C:\development\Python\2_5_2\Lib\site-packages\numpy\linalg\linalg.py", line 249, in solve raise LinAlgError, 'Singular matrix' numpy.linalg.linalg.LinAlgError: Singular matrix Initially MPL plots a graph but when you try to interact with the widget(for example resize) then the output is displayed and the MPL figure is not updated. Everything works with Windows 32-bit. Linux 32-bit and 64-bit are working correctly. Any ideas would be helpful. Thanks. George.

6 9

numpy.testing.asserts and masked array
by josef.pktd＠gmail.com 17 Jan '09

17 Jan '09

I have a regression result with masked arrays that produces a masked array output, estm5.yhat, and I want to test equality to the benchmark case, estm1.yhat, with the asserts in numpy.testing, but I am getting strange results. checking for equality directly returns True, but the asserts in numpy.testing think that they are not equal. It seems the asserts use one of the masked (random) values in the comparison. Whats the trick to assert_almost_equal for masked arrays? Josef >>> np.all(estm5.yhat == estm1.yhat) True and they look the same: >>> estm5.yhat.T masked_array(data = [[-- 3.72175928422 4.44050187015 3.86191008624 7.54173762712 3.52929761101 5.00953694359 5.19214323965 4.71500145194 8.22336216275 7.8884375775 5.24556560163 6.81814135969 7.65416592877 3.7608331097]], mask = [[ True False False False False False False False False False False False False False False]], fill_value=1e+020) >>> estm1.yhat.T masked_array(data = [[-- 3.72175928422 4.44050187015 3.86191008624 7.54173762712 3.52929761101 5.00953694359 5.19214323965 4.71500145194 8.22336216275 7.8884375775 5.24556560163 6.81814135969 7.65416592877 3.7608331097]], mask = [[ True False False False False False False False False False False False False False False]], fill_value=1e+020) using any np.testing.assert returns False >>> np.testing.assert_array_almost_equal(estm5.yhat, estm1.yhat) Traceback (most recent call last): File "<pyshell#253>", line 1, in <module> np.testing.assert_array_almost_equal(estm5.yhat, estm1.yhat) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 310, in assert_array_almost_equal header='Arrays are not almost equal') File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 299, in assert_array_compare raise ValueError(msg) ValueError: Arrays are not almost equal x: array([[ 6.1104881 ], [ 3.72175928], [ 4.44050187],... y: array([[ NaN], [ 3.72175928], [ 4.44050187],... >>> np.testing.assert_almost_equal(estm5.yhat, estm1.yhat) Traceback (most recent call last): File "<pyshell#254>", line 1, in <module> np.testing.assert_almost_equal(estm5.yhat, estm1.yhat) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 208, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 310, in assert_array_almost_equal header='Arrays are not almost equal') File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 299, in assert_array_compare raise ValueError(msg) ValueError: Arrays are not almost equal x: array([[ 6.1104881 ], [ 3.72175928], [ 4.44050187],... y: array([[ NaN], [ 3.72175928], [ 4.44050187],... >>> np.testing.assert_equal(estm5.yhat, estm1.yhat) Traceback (most recent call last): File "<pyshell#255>", line 1, in <module> np.testing.assert_equal(estm5.yhat, estm1.yhat) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 180, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 303, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 295, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 100.0%) x: array([[ 6.1104881 ], [ 3.72175928], [ 4.44050187],... y: array([[ NaN], [ 3.72175928], [ 4.44050187],... >>> estm5.yhat == estm1.yhat masked_array(data = [[--] [True] [True] [True] [True] [True] [True] [True] [True] [True] [True] [True] [True] [True] [True]], mask = [[ True] [False] [False] [False] [False] [False] [False] [False] [False] [False] [False] [False] [False] [False] [False]], fill_value=1e+020)

2 2