Mailman 3 March 2011 - NumPy-Discussion

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

SVD errors
by mtrumpis＠berkeley.edu 04 Jun '13

04 Jun '13

Hello list.. I've run into two SVD errors over the last few days. Both errors are identical in numpy/scipy. I've submitted a ticket for the 1st problem (numpy ticket #990). Summary is: some builds of the lapack_lite module linking against system LAPACK (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not converge" exception on my matrix. This error does occur using Mac's Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on said Ubuntu) Just today I've come across a negative singular value cropping up in an SVD of a different matrix. This error does occur on my ATLAS LAPACK based numpy, as well as on the Ubuntu setup. And once again, it does not happen in Octave/Matlab. I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running. Here are some npy files for the two different cases: https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy Mike

6 7

example reading binary Fortran file
by David Froger 13 Feb '13

13 Feb '13

Hy, My question is about reading Fortran binary file (oh no this question again...) Until now, I was using the unpack module like that : def lread(f,fourBeginning,fourEnd,*tuple): from struct import unpack """Reading a Fortran binary file in litte-endian""" if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) After googling, I read that fopen and npfille was deprecated, and we should use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the cookbook, the mailling list and google, I don't succed in making a simple example. Considering the simple Fortran code below what is the Python script to read the four arrrays? What about if my pc is little endian and the file big endian? I think it will be a good idea to put the Fortran writting-arrays code and the Python reading-array script in the cookbook and maybe a page to help people comming from Fortran to start with Python ? Best, David Froger program makeArray implicit none integer,parameter:: nx=10,ny=20 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j open(11,file='uxuyp.bin',form='unformatted') do i = 1,nx do j = 1,ny ux(i,j) = real(i*j) uy(i,j) = real(i)/real(j) p (i,j) = real(i) + real(j) enddo enddo write(11) ux,uy write(11) p close(11) end program makeArray

8 22

dtype '|S0' not understood
by David Warde-Farley 07 Oct '12

07 Oct '12

Howdy, It seems it's possible using e.g. In [25]: dtype([('foo', str)])Out[25]: dtype([('foo', '|S0')]) to get yourself a zero-length string. However dtype('|S0') results in a TypeError: data type not understood. I understand the stupidity of creating a 0-length string field but it's conceivable that it's accidental. For example, it could lead to a situation where you've created that field, are missing all the data you had meant to put in it, serialize with np.save, and upon np.load aren't able to get _any_ of your data back because the dtype descriptor is considered bogus (can you guess why I thought of this scenario?). It seems that either dtype(str) should do something more sensible than zero-length string, or it should be possible to create it with dtype('| S0'). Which should it be? David

3 3

3D array -> 2D array issue...
by Fred 06 Sep '11

06 Sep '11

Hi all, Let's say C1 is a 3D array, and q0 and k are 2D array. dim C1 = nx*ny*nz dim q0 = nx*ny = dim k I have to do the following: q0[0, 0] = C1[0, 0, k[0, 0]] q0[1, 1] = C1[1, 1, k[1, 1]] ... q0[i, j] = C1[i, j, k[i, j]] ... I tried q0 = C1[:, :, k] but this obviously does not work. How could I do this ala NumPy? TIA Cheers, -- Fred

5 11

segfault on complex array on solaris x86
by John Hunter 17 Aug '11

17 Aug '11

johnh@udesktop253:~> gcc --version gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath) Copyright (C) 2004 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. johnh@udesktop253:~> uname -a SunOS udesktop253 5.10 Generic_142910-17 i86pc i386 i86pc johnh@udesktop253:~> cat test.py import numpy as np print np.__version__ fs = 1000 t = np.linspace(0, 0.3, 301) A = np.array([2, 8]).reshape(-1, 1) f = np.array([150, 140]).reshape(-1, 1) xn = (A * np.exp(2j * np.pi * f * t)).sum(axis=0) johnh@udesktop253:~> python test.py 2.0.0.dev-9451260 Segmentation Fault (core dumped) johnh@udesktop253:~> johnh@udesktop253:~> sudo pstack /var/core/core.python.957 core '/var/core/core.python.957' of 9397: python test.py febf1928 cexp (0, 0, 0, 0, 8060ab0, 84321ac) + 1b0 fe9657e0 npy_cexp (80458e0, 0, 0, 0, 0, 84e2530) + 30 fe95064f nc_exp (8045920, 84e72a0, 8045978, 8045920, 10, 10) + 3f fe937d5b PyUFunc_D_D (84e2530, 84e20f4, 84e25b0, fe950610, 1, 0) + 5b fe95e818 PyUFunc_GenericFunction (81e96e0, 807deac, 0, 80460b8, 2, 2) + 448 fe95fb10 ufunc_generic_call (81e96e0, 807deac, 0, fe98a820) + 70 feeb2d78 PyObject_Call (81e96e0, 807deac, 0, 80a24ec, 8061c08, 0) + 28 fef11900 PyEval_EvalFrame (80a2394, 81645a0, 8079824, 8079824) + 146c fef17708 PyEval_EvalCodeEx (81645a0, 8079824, 8079824, 0, 0, 0) + 620 fef178af PyEval_EvalCode (81645a0, 8079824, 8079824, 8061488, fef3d9ee, 0) + 2f fef3d095 PyRun_FileExFlags (feb91c98, 804687b, 101, 8079824, 8079824, 1) + 75 fef3d9ee PyRun_SimpleFileExFlags (feb91c98, 804687b, 1, 80465a8, fef454a1, 804687b) + 172 fef3e4fd PyRun_AnyFileExFlags (feb91c98, 804687b, 1, 80465a8) + 61 fef454a1 Py_Main (1, 80466b8, feb1cf35, fea935a1, 29, feb96750) + 9d9 08050862 main (2, 80466b8, 80466c4) + 22 08050758 _start (2, 8046874, 804687b, 0, 8046883, 80468ad) + 60

2 3

loadtxt/savetxt tickets
by Charles R Harris 19 Jun '11

19 Jun '11

Hi All, Could someone with an interest in loadtxt/savetxt look through the associated tickets? A search on the tickets using either of those keys will return fairly lengthy lists. Chuck

10 52

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 16 May '11

16 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

1.6.0b1 half float buffer bug?
by Eli Stevens (Gmail) 02 Apr '11

02 Apr '11

I'm working on getting support for 16-bit floats into cython and have run into what seems like a numpy bug (I'm a new contributor to both projects, so bear with me ;). https://github.com/wickedgrey/numpy/commit/29f9f1b709cc2c346b8514859c58a761… Adding NPY_HALF to the switch statement on line 350 of numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it needs to be spelled NPY_FLOAT16 or something similar), but I have no idea how to test the change. Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request? Thanks, Eli

5 22

1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc
by Orion Poplawski 02 Apr '11

02 Apr '11

I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. It appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc (see below for examples). This seems wrong. Thoughts? /usr/lib64/python2.7/site-packages/doc/cython/MANIFEST /usr/lib64/python2.7/site-packages/doc/cython/Makefile /usr/lib64/python2.7/site-packages/doc/cython/README.txt /usr/lib64/python2.7/site-packages/doc/cython/c_numpy.pxd /usr/lib64/python2.7/site-packages/doc/cython/c_python.pxd /usr/lib64/python2.7/site-packages/doc/cython/numpyx.pyx /usr/lib64/python2.7/site-packages/doc/cython/run_test.py /usr/lib64/python2.7/site-packages/doc/cython/run_test.pyc /usr/lib64/python2.7/site-packages/doc/cython/run_test.pyo /usr/lib64/python2.7/site-packages/doc/cython/setup.py /usr/lib64/python2.7/site-packages/doc/cython/setup.pyc /usr/lib64/python2.7/site-packages/doc/cython/setup.pyo /usr/lib64/python2.7/site-packages/doc/pyrex/MANIFEST /usr/lib64/python2.7/site-packages/doc/pyrex/Makefile /usr/lib64/python2.7/site-packages/doc/pyrex/README.txt /usr/lib64/python2.7/site-packages/doc/pyrex/c_numpy.pxd /usr/lib64/python2.7/site-packages/doc/pyrex/c_python.pxd /usr/lib64/python2.7/site-packages/doc/pyrex/notes /usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.c /usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.pyx /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.py /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyc /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyo /usr/lib64/python2.7/site-packages/doc/pyrex/setup.py /usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyc /usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyo /usr/lib64/python2.7/site-packages/doc/swig/Array.i /usr/lib64/python2.7/site-packages/doc/swig/Array1.cxx /usr/lib64/python2.7/site-packages/doc/swig/Array1.h -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com

3 12

NumPy-Discussion March 2011