example reading binary Fortran file
Hy, My question is about reading Fortran binary file (oh no this question again...) Until now, I was using the unpack module like that : def lread(f,fourBeginning,fourEnd,*tuple): from struct import unpack """Reading a Fortran binary file in litte-endian""" if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) After googling, I read that fopen and npfille was deprecated, and we should use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the cookbook, the mailling list and google, I don't succed in making a simple example. Considering the simple Fortran code below what is the Python script to read the four arrrays? What about if my pc is little endian and the file big endian? I think it will be a good idea to put the Fortran writting-arrays code and the Python reading-array script in the cookbook and maybe a page to help people comming from Fortran to start with Python ? Best, David Froger program makeArray implicit none integer,parameter:: nx=10,ny=20 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j open(11,file='uxuyp.bin',form='unformatted') do i = 1,nx do j = 1,ny ux(i,j) = real(i*j) uy(i,j) = real(i)/real(j) p (i,j) = real(i) + real(j) enddo enddo write(11) ux,uy write(11) p close(11) end program makeArray
On 1/30/2009 5:03 PM, David Froger wrote:
I think it will be a good idea to put the Fortran writting-arrays code and the Python reading-array script in the cookbook and maybe a page to help people comming from Fortran to start with Python ?
If you want to be completely safe, read the file in Fortran, then send it as an array to Python (use f2py). Aside from that, assuming your compiler only writes the raw bytes in Fortran order to the file: If you have real(4),dimension(nx,ny):: ux in Fortran, and writes ux to disk. It could be retrieved like this in NumPy: ux = np.fromfile(nx*ny, dtype=np.float32).view((nx,ny), order='F') assuming real(kind=4) is equivalent to np.float32. Sturla Molden
On 1/30/2009 5:27 PM, Sturla Molden wrote:
On 1/30/2009 5:23 PM, Sturla Molden wrote:
ux = np.fromfile(nx*ny, dtype=np.float32).view((nx,ny), order='F')
oops.. this should be
ux = np.fromfile(file, count=nx*ny, dtype=np.float32).view((nx,ny), order='F')
fu*k ux = np.fromfile(file, count=nx*ny, dtype=np.float32).reshape((nx,ny), order='F') Sorry for the previous typos, it's Friday and soon weekend... Sturla Molden
If you want to be completely safe, read the file in Fortran, then send it as an array to Python (use f2py). Aside from that, assuming your compiler only writes the raw bytes in Fortran order to the file:
Careful -- the last time I read a Fortran-=written binary file, I found that the various structures (is that what you call them in Fortran?) were padded with I think 4 bytes. I did it by reading through the file bit by bit and parsing it out with the struct module. Not very efficient, but easy to control. In that case, there were a bunch of tings written, where a few values described the size of the next array, then the array, etc. If you've got a single array, it'll be easier. You might start that way, and then, when you've got it figured out, translate to fromfile(). -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Thank Sturla and Christopher, yes, with the Fortran code : !========================================= program makeArray implicit none integer,parameter:: nx=2,ny=5 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j open(11,file='uxuyp.bin',form='unformatted') do i = 1,nx do j = 1,ny ux(i,j) = 100 + j+(i-1)*10 uy(i,j) = 200. + j+(i-1)*10 p (i,j) = 300. + j+(i-1)*10 enddo enddo write(11) ux,uy write(11) p close(11) write(*,*) 'ux=' do i = 1,nx write(*,*) ( ux(i,j) , j =1,ny) enddo write(*,*) write(*,*) 'uy=' do i = 1,nx write(*,*) ( uy(i,j) , j =1,ny) enddo write(*,*) write(*,*) 'p=' do i = 1,nx write(*,*) ( p(i,j) , j =1,ny) enddo write(*,*) end program makeArray !========================================= the size of the 'uxuyp.bin' file is: 4bytes + ux_bytes + uy_bytes + 4bytes + 4bytes + p_bytes + 4bytes = 4 + nx*ny*4 + nx*ny*4 + 4 + 4 +nx*ny*4 + 4 = 136 bytes the arrays are : ux= 101.00000 102.00000 103.00000 104.00000 105.00000 111.00000 112.00000 113.00000 114.00000 115.00000 uy= 201.00000 202.00000 203.00000 204.00000 205.00000 211.00000 212.00000 213.00000 214.00000 215.00000 p= 301.00000 302.00000 303.00000 304.00000 305.00000 311.00000 312.00000 313.00000 314.00000 315.00000 and with the Python script : #=============================================== import numpy as np nx,ny = 2,5 fourBytes = np.fromfile('uxuyp.bin', count=1, dtype=np.float32) ux = np.fromfile('uxuyp.bin', count=nx*ny, dtype=np.float32).reshape((nx,ny), order='F') print ux #=============================================== I get : [[ 1.12103877e-43 1.11000000e+02 1.12000000e+02 1.13000000e+02 1.14000000e+02] [ 1.01000000e+02 1.02000000e+02 1.03000000e+02 1.04000000e+02 1.05000000e+02]] this function do the trick, but is it optimized? #=============================================== def lread(f,fourBeginning,fourEnd,*tuple): from struct import unpack """Reading a Fortran binary file in litte-endian""" if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) #===============================================
David Froger wrote:
import numpy as np
nx,ny = 2,5
fourBytes = np.fromfile('uxuyp.bin', count=1, dtype=np.float32) ux = np.fromfile('uxuyp.bin', count=nx*ny, dtype=np.float32).reshape((nx,ny), order='F')
print ux #===============================================
I get :
[[ 1.12103877e-43 1.11000000e+02 1.12000000e+02 1.13000000e+02 1.14000000e+02] [ 1.01000000e+02 1.02000000e+02 1.03000000e+02 1.04000000e+02 1.05000000e+02]]
this function do the trick, but is it optimized?
#=============================================== def lread(f,fourBeginning,fourEnd, *tuple): from struct import unpack """Reading a Fortran binary file in litte-endian"""
if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) #===============================================
I'm not sure about whether or not its optimized, but I can tell you that the "mystery" 4 bytes are the number of bytes it that wrote out followed by that number of bytes of data. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma
Careful -- the last time I read a Fortran-=written binary file, I found that the various structures (is that what you call them in Fortran?) were padded with I think 4 bytes.
That is precisely why I suggested using f2py. If you let Fortran read the file (be careful to the same compiler!), it is easy to pass the data on to Python. Otherwise, you will have to figure out how your Fortran program writes the file. I.e. what padding, metainformation, etc. that are used. If you switch Fortran compiler, or even compiler version from the same vendor, you must start over again. On the other hand, an f2py'd solution just requires a recompile. For my own work, I just makes sure NEVER to do any I/O in Fortran! It is asking for trouble. I leave the I/O to Python or C, where it belongs. That way I know what data are written and what data are read. S. M.
ok for f2py!
Otherwise, you will have to figure out how your Fortran program writes the file. I.e. what padding, metainformation, etc. that are used. If you switch Fortran compiler, or even compiler version from the same vendor, you must start over again.
In my experience, I never had this kind of problem. I just have to convert files between big/little endian with uswap (http://linux.die.net/man/1/uswap), but I never see, in my experience, a Fortran program writting data differently given the compilator.
For my own work, I just makes sure NEVER to do any I/O in Fortran! It is asking for trouble. I leave the I/O to Python or C, where it belongs. That way I know what data are written and what data are read.
Unfortunately., binary files are mandatory in the contex I work. I use a scientific code written in Fortran to compute fluid dynamics. Typically the simulation is run on supercalculator and generate giga and giga of datum, so we must use the binary format, which recquire less memory for the stockage. Then, I like to post-trait the datum using Python and Gnuplot.py. That's why I'm looking for a performantant, easy and 'standard' way to read binary Fortran files. (I think many people have the same need). David
A Friday 30 January 2009, David Froger escrigué:
ok for f2py!
Otherwise, you will have to figure out how your Fortran program writes the file. I.e. what padding, metainformation, etc. that are used. If you switch Fortran compiler, or even compiler version from the same vendor, you must start over again.
In my experience, I never had this kind of problem. I just have to convert files between big/little endian with uswap (http://linux.die.net/man/1/uswap), but I never see, in my experience, a Fortran program writting data differently given the compilator.
For my own work, I just makes sure NEVER to do any I/O in Fortran! It is asking for trouble. I leave the I/O to Python or C, where it belongs. That way I know what data are written and what data are read.
Unfortunately., binary files are mandatory in the contex I work. I use a scientific code written in Fortran to compute fluid dynamics. Typically the simulation is run on supercalculator and generate giga and giga of datum, so we must use the binary format, which recquire less memory for the stockage. Then, I like to post-trait the datum using Python and Gnuplot.py.
That's why I'm looking for a performantant, easy and 'standard' way to read binary Fortran files. (I think many people have the same need).
If you need to compact your datafiles to a maximum, you may want to write your data with the HDF5 library [1] that, besides using a binary format, it allows on-the-flight compression. HDF5 is a fairly growing standard in scientific computing and it has wrappers for the most important languages like C, Fortran, Java and, of course, Python ;-) In particular, the available HDF5 interfaces to Python allows to read/write HDF5 native files very easily. Also, many computational environments, like Matlab, Octave or IDL do support HDF5 files. [1] http://www.hdfgroup.org/HDF5/ Cheers, -- Francesc Alted
Thanks a lot Fransesc and Neil, yours messages really help me. I'll look at these solutions attentively. Here is what I write recently, but I begin to understand it's effectively not portable... def fread(fileObject,*arrayAttributs): """ Reading in a binary (=unformatted) Fortran file Let's call 'record' the list of arrays written with one write in the Fortran file. Call one fread per write. Parameter : * fileObject, eg: fileObject = open("data.bin",'rb') * arrayAttributs = ( (shape1,dtype1,readorskip1), (shape2,dtype2,readorskip2) ...) * shape: eg: (100,200) * dtype: eg: '<f4' (little endian, simple precision), '>f8' (big endian, double precision) * readorskip = [0|1] * 1: the array is read and return * 0: the size of the array is skipped to read the array after, the array isn't returned Exemples (with write ux,uy,p in the Fortran code) * f = open("uxuyp.bin",'rb') nx,ny = 100,200 p = readFortran( f, ((nx,ny),'<f8',0), ((nx,ny),'<f8',0), ((nx,ny),'<f8',1) ) f.close """ import numpy,os,utils # compute the recordSize to be read recordBytes = 8 for (shape,dtype,read) in arrayAttributs: dtype = numpy.dtype(dtype) # number of elements to be read in this array count = 1 for size in shape: count *= size # size of the record recordBytes += count*dtype.itemsize # the fileSize fileSize = os.stat(fileObject.name)[6] # compare recordSize and fileSize if recordBytes > fileSize: import logging logging.error('To much data to be read in %r',fileObject.name) logging.error('File Size: %r',fileSize) logging.error('To be read: %r',recordBytes) NoneList = [] for (shape,dtype,read) in arrayAttributs: if read: NoneList.append(None) return NoneList # skip the four bytes in the beginning of the record fileObject.seek(4,1) # read the arrays in record arrays = [] for (shape,dtype,read) in arrayAttributs: # number of elements to be read in this array count=1 for size in shape: count *= size if read: array = numpy.fromfile(fileObject, count=count, dtype=dtype).reshape(shape, order='F') arrays.append(array) else: dtype = numpy.dtype(dtype) arrayBytes = count*dtype.itemsize fileObject.seek(arrayBytes,1) # skip the four bytes at the end of the record fileObject.seek(4,1)
Hy Neil Martinsen-Burrell, I'm trying the FortranFile class, http://www.scipy.org/Cookbook/FortranIO/FortranFile It looks like there are some bug in the last revision (7): * There are errors cause by lines 60,61,63 in * There are indentation errors on lines 97 and 113.
On 2009-05-28 09:32 , David Froger wrote:
Hy Neil Martinsen-Burrell,
I'm trying the FortranFile class, http://www.scipy.org/Cookbook/FortranIO/FortranFile
It looks like there are some bug in the last revision (7):
* There are errors cause by lines 60,61,63 in
* There are indentation errors on lines 97 and 113.
There seem to have been some problems in putting the file on the wiki ("Proxy-Connection: keep-alive\nCache-Control: max-age=0" seems to come from an HTML communication). I've attached my current version of the file to this email. Let me know if you have problems with this. I will try to get the working version up on the wiki. Peace, -Neil
Thank you very much :-) 2009/5/28 Neil Martinsen-Burrell <nmb@wartburg.edu>
On 2009-05-28 09:32 , David Froger wrote:
Hy Neil Martinsen-Burrell,
I'm trying the FortranFile class, http://www.scipy.org/Cookbook/FortranIO/FortranFile
It looks like there are some bug in the last revision (7):
* There are errors cause by lines 60,61,63 in
* There are indentation errors on lines 97 and 113.
There seem to have been some problems in putting the file on the wiki ("Proxy-Connection: keep-alive\nCache-Control: max-age=0" seems to come from an HTML communication). I've attached my current version of the file to this email. Let me know if you have problems with this. I will try to get the working version up on the wiki. Peace,
-Neil
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Sorry, I still don't understand how to use FortranFile ... ============ The fortran code ============ program writeArray implicit none integer,parameter:: nx=2,ny=5 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j do i = 1,nx do j = 1,ny ux(i,j) = 100. + j+(i-1.)*10. uy(i,j) = 200. + j+(i-1.)*10. p(i,j) = 300. + j+(i-1.)*10. enddo enddo open(11,file='uxuyp.bin',form='unformatted') write(11) ux,uy write(11) p close(11) end program writeArray ============= The Python script ============= from fortranfile import FortranFile f = FortranFile('uxuyp.bin') x = f.readReals() ============= The output ============= Traceback (most recent call last): File "readArray.py", line 5, in <module> x = f.readReals() File "/home/users/redone/file2/froger/travail/codes/lib/Tests/fortranread/fortranfile.py", line 181, in readReals data_str = self.readRecord() File "/home/users/redone/file2/froger/travail/codes/lib/Tests/fortranread/fortranfile.py", line 128, in readRecord raise IOError('Could not read enough data') IOError: Could not read enough dat => How to read the file 'uxuyp.bin' ? 2009/5/28 David Froger <david.froger.info@gmail.com>
Thank you very much :-)
2009/5/28 Neil Martinsen-Burrell <nmb@wartburg.edu>
On 2009-05-28 09:32 , David Froger wrote:
Hy Neil Martinsen-Burrell,
I'm trying the FortranFile class, http://www.scipy.org/Cookbook/FortranIO/FortranFile
It looks like there are some bug in the last revision (7):
* There are errors cause by lines 60,61,63 in
* There are indentation errors on lines 97 and 113.
There seem to have been some problems in putting the file on the wiki ("Proxy-Connection: keep-alive\nCache-Control: max-age=0" seems to come from an HTML communication). I've attached my current version of the file to this email. Let me know if you have problems with this. I will try to get the working version up on the wiki. Peace,
-Neil
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
I think the FortranFile class is not intended to read arrays written with the syntax 'write(11) array1, array2, array3' (correct me if I'm wrong). This is the use in the laboratory where I'm currently completing a phd. I'm going to dive into struc, FotranFile etc.. to propose something convenient for people who have to read unformatted binary fortran file very often. 2009/5/28 David Froger <david.froger.info@gmail.com>
Sorry, I still don't understand how to use FortranFile ...
============ The fortran code ============
program writeArray
implicit none integer,parameter:: nx=2,ny=5 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j
do i = 1,nx do j = 1,ny ux(i,j) = 100. + j+(i-1.)*10. uy(i,j) = 200. + j+(i-1.)*10. p(i,j) = 300. + j+(i-1.)*10. enddo enddo
open(11,file='uxuyp.bin',form='unformatted') write(11) ux,uy write(11) p close(11)
end program writeArray
============= The Python script =============
from fortranfile import FortranFile
f = FortranFile('uxuyp.bin')
x = f.readReals()
============= The output =============
Traceback (most recent call last): File "readArray.py", line 5, in <module> x = f.readReals() File "/home/users/redone/file2/froger/travail/codes/lib/Tests/fortranread/fortranfile.py", line 181, in readReals data_str = self.readRecord() File "/home/users/redone/file2/froger/travail/codes/lib/Tests/fortranread/fortranfile.py", line 128, in readRecord raise IOError('Could not read enough data') IOError: Could not read enough dat
=> How to read the file 'uxuyp.bin' ?
2009/5/28 David Froger <david.froger.info@gmail.com>
Thank you very much :-)
2009/5/28 Neil Martinsen-Burrell <nmb@wartburg.edu>
On 2009-05-28 09:32 , David Froger wrote:
Hy Neil Martinsen-Burrell,
I'm trying the FortranFile class, http://www.scipy.org/Cookbook/FortranIO/FortranFile
It looks like there are some bug in the last revision (7):
* There are errors cause by lines 60,61,63 in
* There are indentation errors on lines 97 and 113.
There seem to have been some problems in putting the file on the wiki ("Proxy-Connection: keep-alive\nCache-Control: max-age=0" seems to come from an HTML communication). I've attached my current version of the file to this email. Let me know if you have problems with this. I will try to get the working version up on the wiki. Peace,
-Neil
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 2009-05-29 10:12 , David Froger wrote:
I think the FortranFile class is not intended to read arrays written with the syntax 'write(11) array1, array2, array3' (correct me if I'm wrong). This is the use in the laboratory where I'm currently completing a phd.
You're half wrong. FortranFile can read arrays written as above, but it sees them as a single real array. So, with the attached Fortran program:: In [1]: from fortranfile import FortranFile In [2]: f = FortranFile('uxuyp.bin', endian='<') # Original bug was incorrect byte order In [3]: u = f.readReals() In [4]: u.shape Out[4]: (20,) In [5]: u Out[5]: array([ 101., 111., 102., 112., 103., 113., 104., 114., 105., 115., 201., 211., 202., 212., 203., 213., 204., 214., 205., 215.], dtype=float32) In [6]: ux = u[:10].reshape(2,5); uy = u[10:].reshape(2,5) In [7]: p = f.readReals().reshape(2,5) In [8]: ux, uy, p Out[8]: (array([[ 101., 111., 102., 112., 103.], [ 113., 104., 114., 105., 115.]], dtype=float32), array([[ 201., 211., 202., 212., 203.], [ 213., 204., 214., 205., 215.]], dtype=float32), array([[ 301., 311., 302., 312., 303.], [ 313., 304., 314., 305., 315.]], dtype=float32)) What doesn't currently work is to have arrays of mixed types in the same write statement, e.g. integer :: index(10) real :: x(10,10) ... write(13) x, index To address the original problem, I've changed the code to default to the native byte-ordering (f.ENDIAN='@') and to be more informative about what happened in the error. In the latest version (attached): In [1]: from fortranfile import FortranFile In [2]: f = FortranFile('uxuyp.bin', endian='>') # incorrect endian-ness In [3]: u = f.readReals() IOError: Could not read enough data. Wanted 1342177280 bytes, got 132 and hopefully when people see crazy big numbers like 1.34e9 they will think of byte order problems.
I'm going to dive into struc, FotranFile etc.. to propose something convenient for people who have to read unformatted binary fortran file very often.
Awesome! The thoughts banging around in my head right now are that some sort of mini-language that encapsulates the content of the declarations and the write statements should allow one to tease out exactly which struct call will unpack the right information. f2py has some fortran parsing capabilities, so you might be able to use the fortran itself as the mini-language. Something like spec = fortranfile.OutputSpecification(\ """real(4),dimension(2,5):: ux,uy write(11) ux,uy""") ux, uy = fortranfile.FortranFile('uxuyp.bin').readSpec(spec) Best of luck. Peace, -Neil
You're half wrong. FortranFile can read arrays written as above, but it sees them as a single real array. So, with the attached Fortran program::
In [1]: from fortranfile import FortranFile
In [2]: f = FortranFile('uxuyp.bin', endian='<') # Original bug was incorrect byte order
In [3]: u = f.readReals()
In [4]: u.shape Out[4]: (20,)
In [5]: u Out[5]: array([ 101., 111., 102., 112., 103., 113., 104., 114., 105., 115., 201., 211., 202., 212., 203., 213., 204., 214., 205., 215.], dtype=float32)
In [6]: ux = u[:10].reshape(2,5); uy = u[10:].reshape(2,5)
In [7]: p = f.readReals().reshape(2,5)
In [8]: ux, uy, p Out[8]: (array([[ 101., 111., 102., 112., 103.], [ 113., 104., 114., 105., 115.]], dtype=float32), array([[ 201., 211., 202., 212., 203.], [ 213., 204., 214., 205., 215.]], dtype=float32), array([[ 301., 311., 302., 312., 303.], [ 313., 304., 314., 305., 315.]], dtype=float32))
ok! That's exactlly what I was looking for, thank you. Awesome! The thoughts banging around in my head right now are that some
sort of mini-language that encapsulates the content of the declarations and the write statements should allow one to tease out exactly which struct call will unpack the right information. f2py has some fortran parsing capabilities, so you might be able to use the fortran itself as the mini-language. Something like
spec = fortranfile.OutputSpecification(\ """real(4),dimension(2,5):: ux,uy write(11) ux,uy""") ux, uy = fortranfile.FortranFile('uxuyp.bin').readSpec(spec)
whouhou, I'm really enthusiastic, I love this solution!!! I begin to code it... I'll give news around 1 june, (somethings to finish before). One more time, thanks for this help! best, David
Neil Martinsen-Burrell <nmb <at> wartburg.edu> writes:
On 2009-05-29 10:12 , David Froger wrote:
I think the FortranFile class is not intended to read arrays written with the syntax 'write(11) array1, array2, array3' (correct me if I'm wrong). This is the use in the laboratory where I'm currently completing a phd.
You're half wrong. FortranFile can read arrays written as above, but it sees them as a single real array. So, with the attached Fortran program::
In [1]: from fortranfile import FortranFile
In [2]: f = FortranFile('uxuyp.bin', endian='<') # Original bug was incorrect byte order
In [3]: u = f.readReals()
In [4]: u.shape Out[4]: (20,)
In [5]: u Out[5]: array([ 101., 111., 102., 112., 103., 113., 104., 114., 105., 115., 201., 211., 202., 212., 203., 213., 204., 214., 205., 215.], dtype=float32)
In [6]: ux = u[:10].reshape(2,5); uy = u[10:].reshape(2,5)
In [7]: p = f.readReals().reshape(2,5)
In [8]: ux, uy, p Out[8]: (array([[ 101., 111., 102., 112., 103.], [ 113., 104., 114., 105., 115.]], dtype=float32), array([[ 201., 211., 202., 212., 203.], [ 213., 204., 214., 205., 215.]], dtype=float32), array([[ 301., 311., 302., 312., 303.], [ 313., 304., 314., 305., 315.]], dtype=float32))
What doesn't currently work is to have arrays of mixed types in the same write statement, e.g.
integer :: index(10) real :: x(10,10) ... write(13) x, index
To address the original problem, I've changed the code to default to the native byte-ordering (f.ENDIAN='@') and to be more informative about what happened in the error. In the latest version (attached):
In [1]: from fortranfile import FortranFile
In [2]: f = FortranFile('uxuyp.bin', endian='>') # incorrect endian-ness
In [3]: u = f.readReals()
IOError: Could not read enough data. Wanted 1342177280 bytes, got 132
Hello, I am trying to read a fortran unformatted binary file with FortranFile as follows but I get an error. -----------------------------------------
from FortranFile import FortranFile f = FortranFile("vor_465.float",endian="<") u = f.readReals() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "FortranFile.py", line 193, in readReals data_str = self.readRecord() File "FortranFile.py", line 140, in readRecord data_str = self._read_exactly(l) File "FortranFile.py", line 124, in _read_exactly ' Wanted %d bytes, got %d.' % (num_bytes, l)) IOError: Could not read enough data. Wanted 1124054321 bytes, got 536870908.
My file, "vor_465.float" has the following size: ------------------------------------------ [pradeep@laptop ~]$ls -l vor_465.float -rwxr-xr-x 1 pradeep staff 536870912 Feb 13 *** ----------------------------------------- I am sure this file has data in the right format as when I read it using fortran using the following command: -------------------------------------------------- open (in_file_id,FILE="vor_465.float",form='unformatted', access='direct',recl=4*512*512*512) read (in_file_id,rec=1) buffer -------------------------------------------------- it works completely fine. This data contains single precision float values for a scalar over a cube with 512 points in each direction. Any ideas? Pradeep
Pradeep Jha <pradeep.kumar.jha <at> gmail.com> writes:
Hello,
I am trying to read a fortran unformatted binary file with FortranFile as follows but I get an error. -----------------------------------------
from FortranFile import FortranFile f = FortranFile("vor_465.float",endian="<") u = f.readReals() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "FortranFile.py", line 193, in readReals data_str = self.readRecord() File "FortranFile.py", line 140, in readRecord data_str = self._read_exactly(l) File "FortranFile.py", line 124, in _read_exactly ' Wanted %d bytes, got %d.' % (num_bytes, l)) IOError: Could not read enough data. Wanted 1124054321 bytes, got 536870908.
My file, "vor_465.float" has the following size: ------------------------------------------ [pradeep <at> laptop ~]$ls -l vor_465.float -rwxr-xr-x 1 pradeep staff 536870912 Feb 13 *** -----------------------------------------
I am sure this file has data in the right format as when I read it using fortran using the following command:
-------------------------------------------------- open (in_file_id,FILE="vor_465.float",form='unformatted', access='direct',recl=4*512*512*512) read (in_file_id,rec=1) buffer --------------------------------------------------
it works completely fine. This data contains single precision float values for a scalar over a cube with 512 points in each direction.
Any ideas? Pradeep
Also, when I am trying to write an unformatted binary file with FortranFile I am getting the following error: --------------------------------------------------- [pradeep@laptop ~]$python Python 2.7.3 (v2.7.3:70274d53c1dd, Apr 9 2012, 20:52:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
from FortranFile import FortranFile f = FortranFile("filename", endian="<") array = [1,2,3,4,5,6,6,7] f.writeReals(array) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "FortranFile.py", line 214, in writeReals self._write_check(length_bytes) File "FortranFile.py", line 135, in _write_check number_of_bytes)) IOError: File not open for writing
I dont know how to make the file available for writing. I checked the file permissions and they are completely fine. Please help, Pradeep
The only time I've done this, I used numpy.fromfile exactly as follows. The file had a header followed by a number of records (one float followed by 128 complex numbers), requiring separate calls of numpy.fromfile to read each part. The only strange part about this was that the Fortran code was only supposed to be outputting 3 header fields but was adding an extra integer field for some unknown reason. I used a binary file viewer to work out the actual format of the file. To get at just the complex data in the records, I viewed the data as a recarray. Hopefully it's reasonably self-explanatory: --- import numpy as np import matplotlib.pyplot as plt import matplotlib.cm as cm f = open("field.dat",'rb') # Read header a = np.fromfile(f, np.dtype([('f1','<i4'), ('f2','<i4'), ('f3','<f8'), ('f4','<i4')]), count = 1) # Read data records b = np.fromfile(f, np.dtype([('f1','<f8'), ('f2','128<c16')])) bv = b.view(np.recarray) plt.imshow(np.abs(np.fft.fftshift(np.fft.fft(bv.f2, axis=1), axes=[1])), origin='lower', interpolation='nearest', cmap=cm.Oranges) plt.show() --- Gary David Froger wrote:
Hy,
My question is about reading Fortran binary file (oh no this question again...)
<snip>
After googling, I read that fopen and npfille was deprecated, and we should use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the cookbook, the mailling list and google, I don't succed in making a simple example. Considering the simple Fortran code below what is the Python script to read the four arrrays? What about if my pc is little endian and the file big endian?
participants (8)
-
Christopher Barker -
David Froger -
Francesc Alted -
Gary Ruben -
Neil Martinsen-Burrell -
Pradeep Jha -
Ryan May -
Sturla Molden