Binary file output using python

[Nick Craig-Wood]

Peter Otten <__peter__ at web.de> wrote:
>  Chi Yin Cheung wrote:
> 
> > Is there a way in python to output binary files? I need to python to
> > write out a stream of 5 million floating point numbers, separated by
> > some separator, but it seems that all python supports natively is string
> > information output, which is extremely space inefficient.
> > 
> > I'd tried using the pickle module, but it crashed whenever I tried using
> > it due to the large amount of data involved.
> 
>  A minimalistic alternative is array.tofile()/fromfile(), but pickle should
>  handle a list, say, of 5 million floating point numbers just fine. What
>  exactly are you doing to provoke a crash, and what does it look like?
>  Please give minimal code and the traceback.

cPickle worked fine when I tried it...

  >>> L=map(float, range(5000000))
  >>> import cPickle
  >>> out=file("z", "wb")
  >>> cPickle.dump(L, out, -1)
  >>> out.close()
  >>> inp=file("z", "rb")
  >>> K=cPickle.load(inp)
  >>> inp.close()
  >>> import os
  >>> os.system("ls -l z")
  -rw-r--r-- 1 ncw ncw 45010006 Apr 19 18:43 z
  0
  >>> 

Indicating each float took 9 bytes to store, which is 1 byte more than
a 64 bit float would normally take.

The pickle dump / load each took about 2 seconds.

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick