<div dir="ltr"><div>hi</div><div><br></div><div>To respond to the different open points / questions in previous emails:<br></div><div><br></div><div>> <span style="font-size:12.8px">Ouch - I guess this means it's not possible to use h5py or pytables?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">It is, but you have to compile them with a version of HDF5 includes that match MATLAB's, so that the API matches.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">></span><span style="font-size:12.8px"> </span><span style="font-size:12.8px">what exactly do you mean "anything touching HDF5"? What series of steps causes this segfault? </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In fact, the h5py/pytables modules load symbols from the HDF5 lib, which as a first sanity check, verifies that the header version matches the library version. If not, abort(). MATLAB uses an older version of HDF5 than Anaconda's defaults hence the problem. Compiling these libs with a header matching MATLAB's version works works OK.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">> </span><span style="font-size:12.8px">anything that is matlab >= 2006b</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">They call it mat file version 7.3. Version 7 or less is fine, libmatio (and scipy) can handle it. </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">></span><span style="font-size:12.8px"> </span><span style="font-size:12.8px">I don't think Mathworks </span><span style="font-size:12.8px">has documented their 7.3 format.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Yep, IIRC they put weird stuff in the HDF5 which isn't trivially readable by h5py, but Octave people must have reverse engineered it, so would be a good reference point.</span></div><div><span style="font-size:12.8px"><br></span></div><div>> <span style="font-size:12.8px">What operating systems have you tested these on? </span><span style="font-size:12.8px"> </span><span style="font-size:12.8px">I wouldn't be surprised if you found that all of the above issues happened only on Linux</span></div><div><br></div><div>Windows, Mac & Linux. My experience is that toolchains, compiler flags and library versions are more consistent on Windows & Mac, leading to fewer surprises.</div><div><br></div><div>> Too bad about lin alg</div><div><br></div><div>The main issue is that MKL used by MATLAB uses the same symbol names but not the same ABI, so NumPy's extension modules' references resolve to MKL symbols and not its own and put the args on the stack in the wrong order. Monkey patching around it would be not difficult by intercepting numpy.linalg calls and routing them to MATLAB's MEX Lapack routines via ctypes.</div><div><br></div><div>It would also be possible to build NumPy against MATLAB's MKL BLAS & LAPACK routines, though my suggestion on the numpy mailing list about this got no replies.</div><div><br></div><div>Anaconda with MKL though appears to work fine, since the relevant Python modules (numpy..) are compiled against MKL in that case.</div><div><br></div><span class="im" style="font-size:12.8px">>> - Scipy.io.{save,load}mat segfault<br></span><span style="font-size:12.8px">> That's interesting - it's probably possible to fix that, because that </span><span style="font-size:12.8px">stuff doesn't call any external libraries. Do you have any more</span><br style="font-size:12.8px"><div><span style="font-size:12.8px">details?</span> </div><div><br></div><div>I thought I had seen a symbol in the stacktrace pointing to libmatio I believe, but I just read mio5.py and it's all in Python, so I'll have to go back and check, maybe it was a h5py load.</div><div><br></div><div>> <span style="font-size:12.8px">I wish that they told us what they are doing (rather than providing an opaque executable).</span></div><div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">To be fair, it was in their release notes ;)</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Cheers,</span></div><div><span style="font-size:12.8px">Marmaduke</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 13, 2016 at 9:10 PM, Matthew Brett <span dir="ltr"><<a href="mailto:matthew.brett@gmail.com" target="_blank">matthew.brett@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Dimitri,<br>
<span class=""><br>
On Mon, Jun 13, 2016 at 11:15 AM, Dimitri Papadopoulos Orfanos<br>
<<a href="mailto:dimitri.papadopoulos@cea.fr">dimitri.papadopoulos@cea.fr</a>> wrote:<br>
> Hi Matthew,<br>
><br>
> There are a few links referring to the Matlab 7.3 format:<br>
> <a href="http://scipy-cookbook.readthedocs.io/items/Reading_mat_files.html#matlab-7-3-and-greater" rel="noreferrer" target="_blank">http://scipy-cookbook.readthedocs.io/items/Reading_mat_files.html#matlab-7-3-and-greater</a><br>
<br>
</span>Yes, it's not hard to open the files with h5py or similar, the work<br>
will be to see how the file structure relates to the saved variables.<br>
It's quite possible that it is fairly easy to work out - I haven't<br>
looked.<br>
<span class=""><br>
> <a href="http://stackoverflow.com/questions/4950630/matlab-differences-between-mat-versions" rel="noreferrer" target="_blank">http://stackoverflow.com/questions/4950630/matlab-differences-between-mat-versions</a><br>
><br>
> There is also this PDF from Matlab which does not contain even once the<br>
> "HDF" string but does seem to refer to the HDF5-based 7.3 format:<br>
> <a href="http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf" rel="noreferrer" target="_blank">http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf</a><br>
<br>
</span>That's the doc that describes the earlier mat file formats (versions 4<br>
and 5). <a href="http://scipy.io" rel="noreferrer" target="_blank">scipy.io</a> implements all the stuff described in that doc.<br>
Quoting from the introduction there:<br>
<br>
"This document describes the internal format of MATLAB Level 4 and<br>
Level 5 MAT files."<br>
<br>
See you,<br>
<br>
Matthew<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br>
Neuroimaging mailing list<br>
<a href="mailto:Neuroimaging@python.org">Neuroimaging@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/neuroimaging</a><br>
</div></div></blockquote></div><br></div>