Mailman 3 January 2011 - NumPy-Discussion

numpy.append & numpy.where vs list.append and brute iterative for loop
by Dewald Pieterse Jan. 28, 2011

Jan. 28, 2011

I am processing two csv files against another, my first implementation used python list of lists and list.append to generate a new list while looping all the data including the non-relevant data (can't determine location of specific data element in a list of list). So I re-implented the exact same code but using numpy.array's (2d arrays) using numpy.where to prevent looping over an entire dataset needlessly but the numpy.array based code is about 7.6 times slower? relevant list of list code: > starttime = time.clock() > #NI_data_list room_eqp_list > NI_data_list_new = [] > for NI_row in NI_data_list: > treelevel = NI_row[0] > elevation = NI_row[1] > locater = NI_row[2] > area = NI_row[3] > NIroom = NI_row[4] > #Write appropriate equipment models and drawing into new list > if NIroom != '': > #Write appropriate equipment models and drawing into new list > for row in room_eqp_list: > eqp_room = row[0] > if len(eqp_room) == 5: > eqp_drawing = row[1] > if NIroom == eqp_room: > newrow = > [int(treelevel)+1,elevation,locater,area,NIroom,eqp_drawing] > NI_data_list_new.append(newrow) > #Write appropriate piping info into the new list > for prow in unique_piping_list: > pipe_room = prow[0] > if len(pipe_room) == 5: > pipe_drawing = prow[1] > if pipe_room == NIroom: > piperow = > [int(treelevel)+1,elevation,locater,area,NIroom,pipe_drawing] > NI_data_list_new.append(piperow) > #Write appropriate equipment models and drawing into new list > if (locater != '' and NIroom == ''): > #Write appropriate equipment models and drawing into new list > for row in room_eqp_list: > eqp_locater = row[0] > if len(eqp_locater) == 4: > eqp_drawing = row[1] > if locater == eqp_locater: > newrow = > [int(treelevel)+1,elevation,eqp_locater,area,'',eqp_drawing] > NI_data_list_new.append(newrow) > #Write appropriate piping info into the new list > for prow in unique_piping_list: > pipe_locater = prow[0] > if len(pipe_locater) == 4: > pipe_drawing = prow[1] > if pipe_locater == locater: > piperow = > [int(treelevel)+1,elevation,pipe_locater,area,'',pipe_drawing] > NI_data_list_new.append(piperow) > #Rewrite NI_data to new list > if NIroom == '': > NI_data_list_new.append(NI_row) > > print (time.clock()-starttime) > relevant numpy.array code: > NI_data_write_url = reports_dir + 'NI_data_room2.csv' > NI_data_list_file = open(NI_data_write_url, 'wb') > NI_data_list_writer = csv.writer(NI_data_list_file, delimiter=',', > quotechar='"') > starttime = time.clock() > #NI_data_list room_eqp_list > NI_data_list_new = numpy.array([['TreeDepth', 'Elevation', > 'BuildingLocater', 'Area', 'Room', 'Item']]) > for NI_row in NI_data_list: > treelevel = NI_row[0] > elevation = NI_row[1] > locater = NI_row[2] > area = NI_row[3] > NIroom = NI_row[4] > #Write appropriate equipment models and drawing into new array > if NIroom != '': > #Write appropriate equipment models and drawing into new array > (rowtest, columntest) = numpy.where(room_eqp_list==NIroom) > for row_iter in rowtest: > eqp_room = room_eqp_list[row_iter,0] > if len(eqp_room) == 5: > eqp_drawing = room_eqp_list[row_iter,1] > if NIroom == eqp_room: > newrow = > numpy.array([[int(treelevel)+1,elevation,locater,area,NIroom,eqp_drawing]]) > NI_data_list_new = numpy.append(NI_data_list_new, > newrow, 0) > > #Write appropriate piping info into the new array > (rowtest, columntest) = > numpy.where(unique_room_piping_list==NIroom) > for row_iter in rowtest: #unique_room_piping_list > pipe_room = unique_room_piping_list[row_iter,0] > if len(pipe_room) == 5: > pipe_drawing = unique_room_piping_list[row_iter,1] > if pipe_room == NIroom: > piperow = > numpy.array([[int(treelevel)+1,elevation,locater,area,NIroom,pipe_drawing]]) > NI_data_list_new = numpy.append(NI_data_list_new, > piperow, 0) > #Write appropriate equipment models and drawing into new array > if (locater != '' and NIroom == ''): > #Write appropriate equipment models and drawing into new array > (rowtest, columntest) = numpy.where(room_eqp_list==locater) > for row_iter in rowtest: > eqp_locater = room_eqp_list[row_iter,0] > if len(eqp_locater) == 4: > eqp_drawing = room_eqp_list[row_iter,1] > if locater == eqp_locater: > newrow = > numpy.array([[int(treelevel)+1,elevation,eqp_locater,area,'',eqp_drawing]]) > NI_data_list_new = numpy.append(NI_data_list_new, > newrow, 0) > #Write appropriate piping info into the new array > (rowtest, columntest) = numpy.where(unique_room_eqp_list==locater) > for row_iter in rowtest: > pipe_locater = unique_room_piping_list[row_iter,0] > if len(pipe_locater) == 4: > pipe_drawing = unique_room_piping_list[row_iter,1] > if pipe_locater == locater: > piperow = > numpy.array([[int(treelevel)+1,elevation,pipe_locater,area,'',pipe_drawing]]) > NI_data_list_new = numpy.append(NI_data_list_new, > piperow, 0) > #Rewrite NI_data to new list > if NIroom == '': > NI_data_list_new = numpy.append(NI_data_list_new,[NI_row],0) > > print (time.clock()-starttime) > some relevant output > >>> print NI_data_list_new > [['TreeDepth' 'Elevation' 'BuildingLocater' 'Area' 'Room' 'Item'] > ['0' '' '1000' '' '' ''] > ['1' '' '1000' '' '' 'docname Rev 0'] > ..., > ['5' '6' '1164' '4' '' 'eqp11 RB, R. surname, 24-NOV-08'] > ['4' '6' '1164' '4' '' 'anotherdoc Rev A'] > ['0' '' '' '' '' '']] > Is numpy.append so slow? or is the culprit numpy.where? Dewald Pieterse "A democracy is nothing more than mob rule, where fifty-one percent of the people take away the rights of the other forty-nine." ~ Thomas Jefferson

3 8

Wiener filter / decorrelation
by Daniele Nicolodi Jan. 27, 2011

Jan. 27, 2011

Hello, I'm trying to write a numerical implementation of Wiener filtering / decorrelation (extraction of a signal from noisy time series). What I'm trying to do is the construction of the time domain filter from a measurement of the power spectrum of the noise and the shape of the signal. However I'm encountering some problems that are beyond my knowledge of the matter. Can someone suggest me a reference text book, or other resource? Thank you in advance. Cheers, -- Daniele

5 8

sort descending with NaNs
by Fabrizio Pollastri Jan. 27, 2011

Jan. 27, 2011

Hello, when one has to find a given number of highest values in an array containing NaNs, the sort function (always ascending) is uncomfortable. Since numpy >= 1.4.0 NaNs are sorted to the end, so the searched values are just before the first NaN in a unpredictable position and one has to do another search for the first NaN position. Sorting descending will solve the problem, but there is no option with numpy sort. There is any other trick to avoid this second search? TIA, Fabrizio Pollastri

2 2

Error in tanh for large complex argument
by Mark Bakker Jan. 27, 2011

Jan. 27, 2011

Hello list, When computing tanh for large complex argument I get unexpected nans: tanh works fine for large real values: In [84]: tanh(1000) Out[84]: 1.0 Not for large complex values: In [85]: tanh(1000+0j) Out[85]: (nan+nan*j) While the correct answer is: In [86]: (1.0-exp(-2.0*(1000+0j)))/(1.0+exp(-2.0*(1000+0j))) Out[86]: 1.0 Bug? Thanks, Mark

4 3

How can I install numpy on Python 3.1 in Ubuntu?
by cool-RR Jan. 27, 2011

Jan. 27, 2011

Hello folks, I have Ubuntu 10.10 server on EC2. I installed Python 3.1, and now I want to install NumPy on it. How do I do it? I tried `easy_install-3.1 numpy` but got this error: [...] RefactoringTool: Refactored /tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/core/defchararray.py RefactoringTool: Files that were modified: RefactoringTool: /tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/compat/py3k.py RefactoringTool: /tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/core/defchararray.py Running from numpy source directory.Traceback (most recent call last): File "/usr/local/bin/easy_install-3.1", line 9, in <module> load_entry_point('distribute==0.6.14', 'console_scripts', 'easy_install-3.1')() File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 1855, in main with_ei_usage(lambda: File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage return f() File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 1859, in <lambda> distclass=DistributionWithoutHelpCommands, **kw File "/usr/lib/python3.1/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python3.1/distutils/dist.py", line 919, in run_commands self.run_command(cmd) File "/usr/lib/python3.1/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 342, in run self.easy_install(spec, not self.no_deps) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 582, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 612, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 802, in install_eggs return self.build_and_install(setup_script, setup_base) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 1079, in build_and_install self.run_setup(setup_script, setup_base, args) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/command/easy_install.py", line 1068, in run_setup run_setup(setup_script, args) File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/sandbox.py", line 30, in run_setup lambda: exec(compile(open( File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/sandbox.py", line 71, in run return func() File "/usr/local/lib/python3.1/dist-packages/distribute-0.6.14-py3.1.egg/setuptools/sandbox.py", line 33, in <lambda> {'__file__':setup_script, '__name__':'__main__'}) File "setup.py", line 211, in <module> File "setup.py", line 204, in setup_package File "/tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/distutils/core.py", line 152, in setup File "setup.py", line 151, in configuration File "/tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 972, in add_subpackage File "/tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 941, in get_subpackage File "/tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py File "numpy/setup.py", line 5, in configuration File "/tmp/easy_install-MiUli2/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 713, in __init__ ValueError: 'build/py3k/numpy' is not a directory What can I do? Thanks, Ram.

2 2

Einstein summation convention
by George Nurser Jan. 27, 2011

Jan. 27, 2011

Hi Mark, I was very interested to see that you had written an implementation of the Einstein summation convention for numpy. I'd thought about this last year, and wrote some notes on what I thought might be a reasonable interface. Unfortunately I was not in a position to actually implement it myself, so I left it. But I'll set them out here in case they are useful to you, or anyone else. The discussion about the dot product earlier last year e.g. http://mail.scipy.org/pipermail/numpy-discussion/2010-April/050160.html, http://mail.scipy.org/pipermail/numpy-discussion/2010-April/050202.html and the more recent work on tensor manipulation http://mail.scipy.org/pipermail/numpy-discussion/2010-June/050945.html and named arrays, http://projects.scipy.org/numpy/wiki/NdarrayWithNamedAxes suggested to me that there is a lot of interest in writing tensor and vector products more explicitly. Rather than defining a new binary operator for matrix multiplication, perhaps the most pressing need is for an easy, intuitive, notation to define the axes that are being summed over. My initial thought was that perhaps one could do something like a[:,:*]*b[:*,:] = dot(a,b) where the * would denote the axis that was being summed over. So a[:,:*]*b[:,:*] = dot(a,b.T) and perhaps allow more lables, e.g. & a[:&,:*]*b[:&,:*] = tensordot(a,b,axes=([0,1],[0,1])) I'm not sure whether this is possible to implement, though. Even if it was, I suppose it would require major changes to the python slicing mechanism. A rather more cumbersome, though more powerful idea, might be to somehow parse operations written in terms of the Einstein summation convention. So in this case a function einst might be implemented such that e.g. supposing a.shape = (M,N), b.shape = (N,P), c.shape = (M,N) d.shape = (M,), e.shape = (M,), f.shape = (M,M), g.shape = (M,N,N,P) from numpy import einst as E The dot product would be taken where the same letter is found in lowerscript and upperscript, but no sum would be taken where the same letter was in lowerscript both times. So E(d,'i',e,'I') = sum_i d_{i}*b_{i} = dot(a,b) E(a,'ik',b,'Kj') = sum_k a_{ik}*b_{kj} = dot(a,b) E(f,'ik',a,'IK') = sum_{ik} f_{ik}*a_{ik} = tensordot(a,b,axes=([0,1],[0,1])) E(f,'ki',a,'IK') = sum_{ik} f_{ki}*a_{ik} = tensordot(a,b,axes=([1,0],[0,1])) Contraction, diagonalization and outer product emerge naturally E(f,'iI') = sum_i a_{ii} = f.trace() E(f,'ii') = a_{ii} = f.diagonal() E(d,'i',e,'k') = outer(d,e) Multiplication of more than two tensors could be performed: E(d,'i',f,'Ik',e,'K') = sum_{ik} d_{i}*f_{ik}*e_{k} = dot(d,dot(f,e)) = d.dot(f.dot(e)) E(a,'ik',g,'IKlm',b,'LM') = sum_{iklm} a_{ik}*g_{iklm}*b{lm} = tensordot(a,tensordot(g,b,axes=([2,3],[0,1])),axes=([0,1],[0,1])) Multiplication term by term without summation is implied by repeated letters of the same case. Multiplication over unequal axis lengths would raise an error. E(d,'j',e,'j') = d_{j}*e_{j} = d*e E(a,'ij',c,'ij') = a_{ij}*c_{ij} = a*c E(a,'ij',f,'ij') undefined, since a.shape[0] != f.shape[0] E(f,'ij',f,'ji') = f_{ij}*f_{ji} = f*f.T Broadcasting would be explicit E(f,'ij',d,'j') =a_{ij}*d_j = a*d E(f,'ij',d,'i') =a_{ij}*d_i = a*d[:,None] If a definite order of precedence of sums were established (e.g. doing the rightmost multiplication first) dotting and term by term multiplication could be mixed E(d,'I',d,'i',e,'i') = sum_i d_i*d_i*e_i I hope this is of some use. --George Nurser

1 0

einsum
by Mark Wiebe Jan. 27, 2011

Jan. 27, 2011

I wrote a new function, einsum, which implements Einstein summation notation, and I'd like comments/thoughts from people who might be interested in this kind of thing. In testing it, it is also faster than many of NumPy's built-in functions, except for dot and inner. At the bottom of this email you can find the documentation blurb I wrote for it, and here are some timings: In [1]: import numpy as np In [2]: a = np.arange(25).reshape(5,5) In [3]: timeit np.einsum('ii', a) 100000 loops, best of 3: 3.45 us per loop In [4]: timeit np.trace(a) 100000 loops, best of 3: 9.8 us per loop In [5]: timeit np.einsum('ii->i', a) 1000000 loops, best of 3: 1.19 us per loop In [6]: timeit np.diag(a) 100000 loops, best of 3: 7 us per loop In [7]: b = np.arange(30).reshape(5,6) In [8]: timeit np.einsum('ij,jk', a, b) 10000 loops, best of 3: 11.4 us per loop In [9]: timeit np.dot(a, b) 100000 loops, best of 3: 2.8 us per loop In [10]: a = np.arange(10000.) In [11]: timeit np.einsum('i->', a) 10000 loops, best of 3: 22.1 us per loop In [12]: timeit np.sum(a) 10000 loops, best of 3: 25.5 us per loop -Mark The documentation: einsum(subscripts, *operands, out=None, dtype=None, order='K', casting='safe') Evaluates the Einstein summation convention on the operands. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. The best way to understand this function is to try the examples below, which show how many common NumPy functions can be implemented as calls to einsum. The subscripts string is a comma-separated list of subscript labels, where each label refers to a dimension of the corresponding operand. Repeated subscripts labels in one operand take the diagonal. For example, ``np.einsum('ii', a)`` is equivalent to ``np.trace(a)``. Whenever a label is repeated, it is summed, so ``np.einsum('i,i', a, b)`` is equivalent to ``np.inner(a,b)``. If a label appears only once, it is not summed, so ``np.einsum('i', a)`` produces a view of ``a`` with no changes. The order of labels in the output is by default alphabetical. This means that ``np.einsum('ij', a)`` doesn't affect a 2D array, while ``np.einsum('ji', a)`` takes its transpose. The output can be controlled by specifying output subscript labels as well. This specifies the label order, and allows summing to be disallowed or forced when desired. The call ``np.einsum('i->', a)`` is equivalent to ``np.sum(a, axis=-1)``, and ``np.einsum('ii->i', a)`` is equivalent to ``np.diag(a)``. It is also possible to control how broadcasting occurs using an ellipsis. To take the trace along the first and last axes, you can do ``np.einsum('i...i', a)``, or to do a matrix-matrix product with the left-most indices instead of rightmost, you can do ``np.einsum('ij...,jk...->ik...', a, b)``. When there is only one operand, no axes are summed, and no output parameter is provided, a view into the operand is returned instead of a new array. Thus, taking the diagonal as ``np.einsum('ii->i', a)`` produces a view. Parameters ---------- subscripts : string Specifies the subscripts for summation. operands : list of array_like These are the arrays for the operation. out : None or array If provided, the calculation is done into this array. dtype : None or data type If provided, forces the calculation to use the data type specified. Note that you may have to also give a more liberal ``casting`` parameter to allow the conversions. order : 'C', 'F', 'A', or 'K' Controls the memory layout of the output. 'C' means it should be Fortran contiguous. 'F' means it should be Fortran contiguous, 'A' means it should be 'F' if the inputs are all 'F', 'C' otherwise. 'K' means it should be as close to the layout as the inputs as is possible, including arbitrarily permuted axes. casting : 'no', 'equiv', 'safe', 'same_kind', 'unsafe' Controls what kind of data casting may occur. Setting this to 'unsafe' is not recommended, as it can adversely affect accumulations. 'no' means the data types should not be cast at all. 'equiv' means only byte-order changes are allowed. 'safe' means only casts which can preserve values are allowed. 'same_kind' means only safe casts or casts within a kind, like float64 to float32, are allowed. 'unsafe' means any data conversions may be done. Returns ------- output : ndarray The calculation based on the Einstein summation convention. See Also -------- dot, inner, outer, tensordot Examples -------- >>> a = np.arange(25).reshape(5,5) >>> b = np.arange(5) >>> c = np.arange(6).reshape(2,3) >>> np.einsum('ii', a) 60 >>> np.trace(a) 60 >>> np.einsum('ii->i', a) array([ 0, 6, 12, 18, 24]) >>> np.diag(a) array([ 0, 6, 12, 18, 24]) >>> np.einsum('ij,j', a, b) array([ 30, 80, 130, 180, 230]) >>> np.dot(a, b) array([ 30, 80, 130, 180, 230]) >>> np.einsum('ji', c) array([[0, 3], [1, 4], [2, 5]]) >>> c.T array([[0, 3], [1, 4], [2, 5]]) >>> np.einsum(',', 3, c) array([[ 0, 3, 6], [ 9, 12, 15]]) >>> np.multiply(3, c) array([[ 0, 3, 6], [ 9, 12, 15]]) >>> np.einsum('i,i', b, b) 30 >>> np.inner(b,b) 30 >>> np.einsum('i,j', np.arange(2)+1, b) array([[0, 1, 2, 3, 4], [0, 2, 4, 6, 8]]) >>> np.outer(np.arange(2)+1, b) array([[0, 1, 2, 3, 4], [0, 2, 4, 6, 8]]) >>> np.einsum('i...->', a) array([50, 55, 60, 65, 70]) >>> np.sum(a, axis=0) array([50, 55, 60, 65, 70]) >>> a = np.arange(60.).reshape(3,4,5) >>> b = np.arange(24.).reshape(4,3,2) >>> np.einsum('ijk,jil->kl', a, b) array([[ 4400., 4730.], [ 4532., 4874.], [ 4664., 5018.], [ 4796., 5162.], [ 4928., 5306.]]) >>> np.tensordot(a,b, axes=([1,0],[0,1])) array([[ 4400., 4730.], [ 4532., 4874.], [ 4664., 5018.], [ 4796., 5162.], [ 4928., 5306.]])

9 20

3d plane to point cloud fitting using SVD
by Peter Schmidtke Jan. 26, 2011

Jan. 26, 2011

Dear Numpy Users, I want to fit a 3d plane into a 3d point cloud and I saw that one could use svd for this purpose. So as I am very fond of numpy I saw that svd was implementented in the linalg module. Currently I have a numpy array called xyz with n lines (number of points) and 3 columns (x,y,z). I calculated the centroid as : xyz0=npy.mean(xyz, axis=0) #calculate the centroid Next I shift the centroid of the point cloud to the origin with. M=xyz-xyz0 next I saw by matlab analogy (http://www.mathworks.co.jp/matlabcentral/newsreader/view_thread/262996) that I can write this : u,s,vh=numpy.linalg.linalg.svd(M) Then in the matlab analog they use the last column of vh to get the a,b,c coefficients for the equation a,b,c=vh[:, -1] in numpy The problem is that the equation ax+by+cz=0 does not represent the plan through my point cloud at all. What am I doing wrong, how can I get the a,b and c coefficients? Thanks in advance. -- Peter Schmidtke ---------------------- PhD Student at the Molecular Modeling and Bioinformatics Group Dep. Physical Chemistry Faculty of Pharmacy University of Barcelona

3 2

tril, triu, document/ implementation conflict
by eat Jan. 26, 2011

Jan. 26, 2011

Hi, I just noticed a document/ implementation conflict with tril and triu. According tril documentation it should return of same shape and data-type as called. But this is not the case at least with dtype bool. The input shape is referred as (M, N) in tril and triu, but as (N, M) in tri. Inconsistent? Also I'm not very happy with the performance, at least dtype bool can be accelerated as follows. In []: M= ones((2000, 3000), dtype= bool) In []: timeit triu(M) 10 loops, best of 3: 173 ms per loop In []: timeit triu_(M) 10 loops, best of 3: 107 ms per loop In []: M= asarray(M, dtype= int) In []: timeit triu(M) 10 loops, best of 3: 160 ms per loop In []: timeit triu_(M) 10 loops, best of 3: 163 ms per loop In []: M= asarray(M, dtype= float) In []: timeit triu(M) 10 loops, best of 3: 195 ms per loop In []: timeit triu_(M) 10 loops, best of 3: 157 ms per loop I have attached a crude 'fix' incase someone is interested. Regards, eat

2 2

bug in genfromtxt with missing values?
by Skipper Seabold Jan. 25, 2011

Jan. 25, 2011

Am I misreading the docs or missing something? Consider the following adapted from here: http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html from StringIO import StringIO import numpy as np data = "1, 2, 3\n4, ,5" np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values=" ", filling_values=0) array([(1.0, 2.0, 3.0), (4.0, nan, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')]) np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values={'b':" "}, filling_values={'b' : 0}) array([(1.0, 2.0, 3.0), (4.0, 0.0, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')]) Unless I use the dict for missing_values, it doesn't fill them in. Without named columns np.genfromtxt(StringIO(data), delimiter=",", missing_values=" ", filling_values=0) array([[ 1., 2., 3.], [ 4., nan, 5.]]) np.genfromtxt(StringIO(data), delimiter=",", missing_values={1 :" "}, filling_values={1 :0}) array([[ 1., 2., 3.], [ 4., 0., 5.]]) Skipper

3 4