[SciPy-user] MemoryError transforming COO matrix to a CSR matrix

[Dinesh B Vadhia]

Hello!  This is a resend.  

I get a MemoryError when transforming a coo_matrix to a csr_matrix.  The coo_matrix is loaded with about 32m integers (in fact, just binary 1's) which at 4 bytes per int works out to about 122Mb for the matirix.  As I have 2Gb of RAM on my Windows machine this should be ample for transforming A to a csr_matrix.  Here is the error message followed by the code:

Traceback (most recent call last):
  File "... \... .py", line 310, in <module>
    A = sparse.csr_matrix(A)
  File "C:\Python25\lib\site-packages\scipy\sparse\sparse.py", line 1162, in __init__
    temp = s.tocsr()
  File "C:\Python25\lib\site-packages\scipy\sparse\sparse.py", line 2175, in tocsr
    return csr_matrix((data, colind, indptr), self.shape, check=False)
  File "C:\Python25\lib\site-packages\scipy\sparse\sparse.py", line 1197, in __init__
    self.data = asarray(s, dtype=self.dtype)
  File "C:\Python25\lib\site-packages\numpy\core\numeric.py", line 132, in asarray
    return array(a, dtype, copy=False, order=order)
MemoryError


# imports
import numpy
import scipy
from scipy import sparse 

# constants
nnz = 31398038
I = 20000
J = 80000
dataFile = aFilename

# Initialize A as a coo_matrix with dimensions(I, J)
> A = sparse.coo_matrix(None, dims=(I, J), dtype=int) 

# Populate matrix A by first loading data into a coo_matrix using the coo_matrix(V, (I,J)), dims) method
> ij = numpy.array(numpy.empty((nnz, 2), dtype=int))
> f = open(dataFile, 'rb')
> ij = pickle.load(f)
> row = ij[:,0]
> column = ij[:,1]
> data = scipy.ones(ij.shape[0], dtype=int)

# Load data into A, convert A to csr_matrix
> A = sparse.coo_matrix((data, (row, column)), dims=(I,J))
> A = sparse.csr_matrix(A)                                                        # the MemoryError occurs here

Any ideas?

Dinesh

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20080207/914a5dd9/attachment.html>