[SciPy-User] scipy.io.loadmat throws TypeError with large files

[Richard Llewellyn]

Hi,

I get this or similar (different integer than 75724 in error) exceptions
when loading a sparse matrix (CSC) saved with savemat, all default options.

>>> m = loadmat('my_large_mat.mat')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio.py",
line 176, in loadmat
    matfile_dict = MR.get_variables(variable_names)
  File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py",
line 274, in get_variables
    hdr, next_position = self.read_var_header()
  File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py",
line 236, in read_var_header
    raise TypeError('Expecting miMATRIX type here, got %d' %  mdtype)
TypeError: Expecting miMATRIX type here, got 75724


here the matrix was:

> matrix
<400000x4176 sparse matrix of type '<class 'numpy.uint8'>'
with 934099575 stored elements in Compressed Sparse Column format>

and looks fine before saving.

It looks as if this only occurs when the saved matrix file size is > 4GB --
at least I haven't seen it with files in the 3GB range.

64 bit Linux.

Not a crisis, as I am chunking anyway, so I can just chunk smaller, but
when I get more RAM would be nice to bump it up to 8 GB files or so.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20130807/c4684697/attachment.html>