Writing some floats in a file in an efficient way
bartc
bc at freeuk.com
Wed Feb 21 09:02:38 EST 2018
On 21/02/2018 13:27, ast wrote:
> Hello
>
> I would like to write a huge file of double precision
> floats, 8 bytes each, using IEEE754 standard. Since
> the file is big, it has to be done in an efficient
> way.
Time efficient or space efficient?
If the latter, how many floats are we talking about?
> I tried pickle module but unfortunately it writes
> 12 bytes per float instead of just 8.
>
> Example:
>
> import pickle
>
> f = open("data.bin", "wb")
> mypickler = pickle.Pickler(f)
>
> mypickler.dump(0.0)
> mypickler.dump(-0.0)
I don't know about pickle, but before looking at other approaches, have
you tried outputting the numbers more than one at a time?
I found that list or tuple of 10 floats produced a file of 98 or 97
bytes, a somewhat smaller overhead. 10,000, about 90KB, a 10% overhead.
--
bartc
More information about the Python-list
mailing list