Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison
sturlamolden at yahoo.no
Sun Jan 14 07:37:04 CET 2007
> Thanks for your note. I have 1Gig of RAM. Also, Matlab has no problem
> in reading the file into memory. I am just running Istvan's code that
> was posted earlier.
You have a CSV file of about 520 MiB, which is read into memory. Then
you have a list of list of floats, created by list comprehension, which
is larger than 274 MiB. Additionally you try to allocate a NumPy array
slightly larger than 274 MiB. Now your process is already exceeding 1
GiB, and you are probably running other processes too. That is why you
run out of memory.
So you have three options:
1. Buy more RAM.
2. Low-level code a csv-reader in C.
3. Read the data in chunks. That would mean something like this:
import time, csv, random
def make_data(rows=6E6, cols=6):
fp = open('data.txt', 'wt')
counter = range(cols)
for row in xrange( int(rows) ):
vals = map(str, [ random.random() for x in counter ] )
fp.write( '%s\n' % ','.join( vals ) )
start = time.clock()
arrlist = None
r = 0
CHUNK_SIZE_HINT = 4096 * 4 # seems to be good
fid = file('data.txt')
chunk = fid.readlines(CHUNK_SIZE_HINT)
if not chunk: break
reader = csv.reader(chunk)
data = [ map(float, row) for row in reader ]
arrlist = [ numpy.array(data,dtype=float), arrlist ]
r += arrlist.shape
print 'Created list of chunks, elapsed time so far: ', time.clock()
print 'Joining list...'
data = numpy.empty((r,arrlist.shape),dtype=float)
r1 = r
r0 = r1 - arrlist.shape
data[r0:r1,:] = arrlist
r1 = r0
arrlist = arrlist
print 'Elapsed time:', time.clock() - start
This can process a CSV file of 6 million rows in about 150 seconds on
my laptop. A CSV file of 1 million rows takes about 25 seconds.
Just reading the 6 million row CSV file ( using fid.readlines() ) takes
about 40 seconds on my laptop. Python lists are not particularly
efficient. You can probably reduce the time to ~60 seconds by writing a
new CSV reader for NumPy arrays in a C extension.
More information about the Python-list