problem with a binary file on OS X 10.6
Hello, I have a problem reading a binary file on OS X 10.6. On my desktop mac (OS X 10.4.11, Python 2.5.4, Numpy 1.3.0), the command CELLS, NSIZE, NE = fromfile(fp, int32, 3), where fp is the filename, correctly prints "21 20 300". However, when I try the above on my laptop using Snow Leopard (Python 2.6.6, Numpy 1.5.0), I get the numbers "352321536 335544320 738263040". This results in a failure when trying to read the data that comes afterwards (Numpy fails with "array too big"). I assume this error may have something to do with 10.6 being 64-bit and 10.4 being 32-bit; however, the Python 2.6.6. distribution is also 32-bit. Any thoughts on this? Thanks,Olli Sipilä
On Thu, Nov 4, 2010 at 12:15 PM, Olli Sipilä
Hello, I have a problem reading a binary file on OS X 10.6. On my desktop mac (OS X 10.4.11, Python 2.5.4, Numpy 1.3.0), the command CELLS, NSIZE, NE = fromfile(fp, int32, 3), where fp is the filename, correctly prints "21 20 300". However, when I try the above on my laptop using Snow Leopard (Python 2.6.6, Numpy 1.5.0), I get the numbers "352321536 335544320 738263040". This results in a failure when trying to read the data that comes afterwards (Numpy fails with "array too big"). I assume this error may have something to do with 10.6 being 64-bit and 10.4 being 32-bit; however, the Python 2.6.6. distribution is also 32-bit. Any thoughts on this? Thanks, Olli Sipilä
Snow Leopard (10.6) is Intel only, so your laptop is using an Intel CPU which is little-endian. Is your Tiger (10.4) machine a PowerPC CPU? That would be big-endian, and therefore would write the floats to disk reversed compared to what a little-endian machine would expect. Peter
A Thursday 04 November 2010 13:15:00 Olli Sipilä escrigué:
Hello, I have a problem reading a binary file on OS X 10.6. On my desktop mac (OS X 10.4.11, Python 2.5.4, Numpy 1.3.0), the command CELLS, NSIZE, NE = fromfile(fp, int32, 3), where fp is the filename, correctly prints "21 20 300". However, when I try the above on my laptop using Snow Leopard (Python 2.6.6, Numpy 1.5.0), I get the numbers "352321536 335544320 738263040". This results in a failure when trying to read the data that comes afterwards (Numpy fails with "array too big"). I assume this error may have something to do with 10.6 being 64-bit and 10.4 being 32-bit; however, the Python 2.6.6. distribution is also 32-bit. Any thoughts on this? Thanks,Olli
Yeah. Most probably your Mac has a PowerPC processor, which has a different byte-ordering than Intel. Look at this:
a = np.int32(21) a.byteswap() 352321536 a = np.int32(20) a.byteswap() 335544320 a = np.int32(300) a.byteswap() 738263040
To solve this, just apply byteswap once more:
a = np.int32(300) a.byteswap().byteswap() 300
and you are done. -- Francesc Alted
A Thursday 04 November 2010 13:37:33 Pauli Virtanen escrigué:
Thu, 04 Nov 2010 13:33:48 +0100, Francesc Alted wrote: [clip]
To solve this, just apply byteswap once more:
a = np.int32(300) a.byteswap().byteswap()
300
and you are done.
Or directly specify big-endian byte order when reading
fromfile(fp, '>i4', 3)
Much, much better :-) -- Francesc Alted
On 11/4/10 5:15 AM, Olli Sipilä wrote:
I have a problem reading a binary file on OS X 10.6. On my desktop mac (OS X 10.4.11, Python 2.5.4, Numpy 1.3.0), the command
CELLS, NSIZE, NE = fromfile(fp, int32, 3), where fp is the filename,
correctly prints "21 20 300". However, when I try the above on my laptop using Snow Leopard (Python 2.6.6, Numpy 1.5.0), I get the numbers "352321536 335544320 738263040". This results in a failure when trying to read the data that comes afterwards (Numpy fails with "array too big"). I assume this error may have something to do with 10.6 being 64-bit and 10.4 being 32-bit; however, the Python 2.6.6. distribution is also 32-bit. Any thoughts on this?
you've specified int32, so 32-64 bit should not be the issue. however, it's likely that endian issues are -- is one of your machine PPC, and one Intel? yup, that's it: In [29]: a = np.array((21, 20, 300), dtype=np.int32) In [30]: a Out[30]: array([ 21, 20, 300]) In [31]: a.byteswap() Out[31]: array([352321536, 335544320, 738263040]) So, you can either call bytswap yourself, or, better yet specify the endianness in your dtype specifier, somethign like (untested) dtype='>i4' for big endian and dtype='>i4' for little endian. Intel is little endian, PPC is bigendian (I think -- do test to make sure!) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (5)
-
Christopher Barker -
Francesc Alted -
Olli Sipilä -
Pauli Virtanen -
Peter