[Numpy-discussion] Reading from binary file with memmap, with offset

Joe Kington joferkington at gmail.com
Thu Oct 10 09:07:58 EDT 2013


You just need to supply the "offset" kwarg to "memmap".

for example:

with open(localfile, "r") as fd:
  # read offset from first line of file
  offset = int(next(fd).split()[-2])
  np.memmap(fd, dtype="float32", mode="r", offset=offset)

Also, there's no need to do things like "offset =
int(fd.readlines()[0].split()[-2])"

Just do "offset = int(next(fd).split()[-2])" instead.  Readlines reads the
entire file into memory. You only want the first line.

Hope that helps!
-Joe


On Thu, Oct 10, 2013 at 7:43 AM, Andreas Hilboll <lists at hilboll.de> wrote:

> Hi,
>
> I have a problem using memmap correctly. I need to read a data file
> which consists of an ASCII header and appended binary single precision
> floating point values. memmap complains that the "Size of available data
> is not a multiple of the data-type size." But as far as I can tell, the
> size *doas* match the data-type size.
>
> The problem is illustrated by the following code:
>
> ---8<-------
>
> url = "http://www.iup.uni-bremen.de/~hilboll/download/20120204.XD4_N2"
> localfile = "np_memmap.dat"
>
> import os
> import urllib
>
> import numpy as np
>
>
> # download data file
> if not os.access(localfile, os.R_OK):
>   urllib.urlretrieve(url, localfile)
>
> with open(localfile, "r") as fd:
>   # read offset from first line of file
>   offset = int(fd.readlines()[0].split()[-2])
>   # jump to begin of data block
>   fd.seek(offset)
>   # read until EOF
>   blob = fd.read()
>   print("Size of data blob [bytes]: {}".format(len(blob)))
>   print("This can actually be divided by 4: {} / 4.0 = {}".format(
>             len(blob), len(blob) / 4.0))
>   # go back to begin of data block
>   fd.seek(offset)
>   print("But it cannot be loaded as np.memmap with dtype float32:")
>   np.memmap(fd, dtype="float32", mode="r")
>
> ---8<-------
>
> Any help is greatly appreciated :)
>
> -- Andreas.
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131010/1797d962/attachment.html>


More information about the NumPy-Discussion mailing list