Reading bz2 file into numpy array
__peter__ at web.de
Tue Nov 23 11:18:52 CET 2010
> On Mon, 22 Nov 2010 11:37:22 +0100, Peter Otten wrote:
>>> is there a convenient way to read bz2 files into a numpy array?
>> f = bz2.BZ2File(filename)
>> data = numpy.fromstring(f.read(), numpy.float32)
> That's going to hurt if the file is large.
Yes, but memory usage will peak at about 2*sizeof(data), and most scripts
need more data than just a single numpy.array.
In short: the OP is unlikely to run into the problem.
> You might be better off either extracting to a temporary file, or creating
> a pipe with numpy.fromfile() reading the pipe and either a thread or
> subprocess decompressing the data into the pipe.
I like to keep it simple, so if available RAM turns out to be the limiting
factor I think extracting the data into a temporary file is a good backup
More information about the Python-list