[Numpy-discussion] Reading from binary file with memmap, with offset

Andreas Hilboll lists at hilboll.de
Thu Oct 10 08:43:25 EDT 2013


Hi,

I have a problem using memmap correctly. I need to read a data file
which consists of an ASCII header and appended binary single precision
floating point values. memmap complains that the "Size of available data
is not a multiple of the data-type size." But as far as I can tell, the
size *doas* match the data-type size.

The problem is illustrated by the following code:

---8<-------

url = "http://www.iup.uni-bremen.de/~hilboll/download/20120204.XD4_N2"
localfile = "np_memmap.dat"

import os
import urllib

import numpy as np


# download data file
if not os.access(localfile, os.R_OK):
  urllib.urlretrieve(url, localfile)

with open(localfile, "r") as fd:
  # read offset from first line of file
  offset = int(fd.readlines()[0].split()[-2])
  # jump to begin of data block
  fd.seek(offset)
  # read until EOF
  blob = fd.read()
  print("Size of data blob [bytes]: {}".format(len(blob)))
  print("This can actually be divided by 4: {} / 4.0 = {}".format(
            len(blob), len(blob) / 4.0))
  # go back to begin of data block
  fd.seek(offset)
  print("But it cannot be loaded as np.memmap with dtype float32:")
  np.memmap(fd, dtype="float32", mode="r")

---8<-------

Any help is greatly appreciated :)

-- Andreas.



More information about the NumPy-Discussion mailing list