Re: [SciPy-dev] Please stress-test SVN version pf matlab reader
Hi Matthew, further - if I set both squeeze_me and struct_as_record to True (does that make sense?), I get into some rather nasty situations, where I can't use the variables in the matfile: For example: In [40]: m = sio.loadmat('C-RA_Adapt.mat',squeeze_me=True,struct_as_record=True) In [41]: h = m['history'] In [42]: h1 = h[0] In [43]: q = h1['q'] Yes - I know it's a bit convoluted, but that's how it is. Now - I can't do anything with 'q': In [44]: q[0] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /Users/arokem/Projects/SchizoSpread/OrientationTuningBehavior/DATA/<ipython console> in <module>() IndexError: 0-d arrays can't be indexed In [45]: q.dtype Out[45]: dtype('object') There's supposed to be an array called 'x' in there: In [46]: q['x'] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/arokem/Projects/SchizoSpread/OrientationTuningBehavior/DATA/<ipython console> in <module>() ValueError: field named x not found. Am I doing something unreasonable? Cheers, Ariel On Thu, Jan 7, 2010 at 7:26 PM, Ariel Rokem <arokem@berkeley.edu> wrote:
Hi Matthew,
I updated my scipy from the repo and I am still getting the following behavior (also reported a couple of days ago on scipy-user), when dealing with the attached file:
In [6]: m = sio.loadmat('/Users/arokem/Projects/SchizoSpread/OrientationTuningBehavior/DATA/C-RA_Adapt.mat')
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio.py:99: FutureWarning: Using struct_as_record default value (False) This will change to True in future versions return MatFile5Reader(byte_stream, **kwargs)
So far, so good - this is to be expected. But, for obvious reasons, I want to set the kwarg squeeze_me to True:
In [7]: m = sio.loadmat('/Users/arokem/Projects/SchizoSpread/OrientationTuningBehavior/DATA/C-RA_Adapt.mat',squeeze_me=True) --------------------------------------------------------------------------- TypeError Traceback (most recent call last)
/Users/arokem/<ipython console> in <module>()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio.pyc in loadmat(file_name, mdict, appendmat, **kwargs) 132 ''' 133 MR = mat_reader_factory(file_name, appendmat, **kwargs) --> 134 matfile_dict = MR.get_variables() 135 if mdict is not None: 136 mdict.update(matfile_dict)
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5.pyc in get_variables(self, variable_names) 411 continue 412 try: --> 413 res = self.read_var_array(hdr, process) 414 except MatReadError, err: 415 warnings.warn(
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5.pyc in read_var_array(self, header, process) 380 `process`. 381 ''' --> 382 return self._matrix_reader.array_from_header(header, process) 383 384 def get_variables(self, variable_names=None):
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:4718)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:4367)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in scipy.io.matlab.mio5_utils.VarReader5.read_struct (scipy/io/matlab/mio5_utils.c:6771)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix (scipy/io/matlab/mio5_utils.c:3995)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:4610)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio_utils.so in scipy.io.matlab.mio_utils.process_element (scipy/io/matlab/mio_utils.c:1272)()
/Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/io/matlab/mio_utils.so in scipy.io.matlab.mio_utils.process_element (scipy/io/matlab/mio_utils.c:1149)()
TypeError: Cannot convert mat_struct to numpy.ndarray
This doesn't happen when I set struct_as_record to True as well, but then I need to deal with all these record arrays that I get. And yes - it does seem to work faster, when it works, though I haven't done any thorough testing of that.
Any ideas?
Cheers,
Ariel
-- Ariel Rokem Helen Wills Neuroscience Institute University of California, Berkeley http://argentum.ucbso.berkeley.edu/ariel
Hi
further - if I set both squeeze_me and struct_as_record to True (does that make sense?), I get into some rather nasty situations, where I can't use the variables in the matfile:
Sorry - I think I missed your first email, and can't see the attachment - can you send by private mail? I've fixed your first error I think - thanks for the report - please let me know if current SVN does work for mat_struct problem... See you, Matthew
Matthew Brett wrote:
Hi
further - if I set both squeeze_me and struct_as_record to True (does that make sense?), I get into some rather nasty situations, where I can't use the variables in the matfile:
Sorry - I think I missed your first email, and can't see the attachment - can you send by private mail?
I've fixed your first error I think - thanks for the report - please let me know if current SVN does work for mat_struct problem...
See you,
Matthew
Matthew, I updated from svn, built, and installed, and I am having a similar problem. A sample file is http://currents.soest.hawaii.edu/clivar/ladcp/I5S_2009/002.mat Here is what happens: In [16]:from scipy.io import loadmat In [17]:a = loadmat('002.mat', struct_as_record=True, squeeze_me=True) In [18]:a['p']['lon'] Out[18]:array(array(30.355230445182066), dtype=object) n [20]:scipy.version.version Out[20]:'0.8.0.dev6182' I think this is the same problem that Ariel noted. If I don't use squeeze_me but explicitly de-reference each of the two levels of arrays of shape (1,1), then I can get at the underlying variables. Eric
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Hi Ariel,
further - if I set both squeeze_me and struct_as_record to True (does that make sense?), I get into some rather nasty situations, where I can't use the variables in the matfile:
For example:
In [40]: m = sio.loadmat('C-RA_Adapt.mat',squeeze_me=True,struct_as_record=True)
In [41]: h = m['history']
In [42]: h1 = h[0]
In [43]: q = h1['q'] ... There's supposed to be an array called 'x' in there:
In [46]: q['x'] --------------------------------------------------------------------------- ValueError Traceback (most recent call last)
Looking further into this - rather late at night - so I might be wrong - I think this is structural (forgive the pun) to structured arrays. We have to load the arrays as structured arrays with named fields and object dtype per field, because matlab allows any contents of the fields (fields need not have the same class of contents for each struct in the struct array). When we squeeze structured arrays, we get 0d structured arrays, with object fields. We can't now index these, but It is only indexing into the structured array, that dereferences the object: In [3]: st = np.zeros((1,), dtype=[('f0', 'O'), ('f1', 'O')]) In [4]: st[0]['f0'] Out[4]: 0 In [5]: st['f0'] Out[5]: array([0], dtype=object) Now, when we squeeze to 0d, we can't index any more: In [6]: sst = np.squeeze(st) In [7]: sst.shape Out[7]: () In [8]: sst['f0'] Out[8]: array(0, dtype=object) So we can't dereference, except explicitly like this: In [9]: sst['f0'].item() Out[9]: 0 I think. Well, more, I hope that someone who knows better than I do can think of a way round this... Best, Matthew
participants (3)
-
Ariel Rokem -
Eric Firing -
Matthew Brett