[Neuroimaging] Using Python libs from MATLAB

[Marmaduke Woodman]

hi Nathaniel

Thanks for the details on linking. I hadn't had time to test all bugs on
all platforms.

On Mac, everything seems ok, including h5py and non-MKL linalg.

On Linux, with h5py I see HDF5 library warning that lib and header versions
don't match, with a stack trace including

[  1] 0x00007f841ce30fc0
/lib/libc.so.6+00217024 abort+00000384
[  2] 0x00007f840b9c8d75
/soft/matlab2015a/bin/glnxa64/libhdf5.so.8+00224629 H5check_version+00000325
[  3] 0x00007f83de4c367f
/home/duke/mc/envs/test/lib/python3.5/site-packages/h5py/h5f.so+00034431
PyInit_h5f+00006815

ldd shows h5py's dll linked to conda's hdf5, so this perhaps confirms your
point about.

cheers,
Marmaduke

On Tue, Jun 14, 2016 at 9:53 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Jun 14, 2016 12:36 PM, "Marmaduke Woodman" <
> marmaduke.woodman at univ-amu.fr> wrote:
> >
> >
> > On Tue, Jun 14, 2016 at 5:03 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> My prediction is that on Windows and OS X, you actually can use
> inconsistent versions of hdf5 and blas, and it will work. Can you confirm
> this explicitly? It's possible that I'm wrong, but I would be surprised
> because it's actually impossible to get symbol collisions on those
> operating systems unless you really try hard to configure things in an
> extra broken way.
> >
> > I can confirm that h5py's call into hdf5 results in an abort due to
> inconsistent header and library versions (1.8.15 vs 1.8.16 for example). We
> can call it something else like ABI mismatch, but nevertheless, it seems
> has to use their version of HDF5 or not at all. I wrote to this list partly
> so that others interested might help confirm, deny or qualify the
> difficulties I stated.
>
> Ok. On Windows, I think the only way this could be happening is if
> everyone using hdf5 is linking to a version of the dll without the version
> number in the name. The solution is probably for h5py to make sure it links
> against and ships a library called libhdf5-1.8.16.dll or similar.
>
> On OS X... not sure what could be causing this. Maybe h5py's build system
> is passing some sort of please-give-me-symbol-collisions flag to the linker
> (-flat_namespace and friends)? Maybe it has some buggy paths embedded in
> the binaries? On OS X I can't immediately think of a way to make h5py break
> on MATLAB without making it mostly broken everywhere else too, but I guess
> someone found a way. It's probably not hard to fix though.
>
> On Linux, it's also not too hard to avoid these issues, but if things are
> broken now then it suggests that MATLAB is doing something bad internally
> like loading libhdf5 with the RTLD_GLOBAL flag. Which is the flag that
> means "please monkey patch everything loaded after this with these
> symbols", equivalent to LD_PRELOAD. And if that's what's going on then
> there isn't much that h5py can do about it; really the MATLAB folk would
> need to fix that.
>
> >> (I notice they say they're compatible with py27, py33, py34... I wonder
> what compilers they used for that on Windows.)
> >
> > They are using the C-API to libpython.dll, so probably MSVC..?
>
> Normally py27 and py33/py34 use different and incompatible versions of
> MSVC. It's possible to mix them under certain limited circumstances, but
> this is also another chance to mess things up :-).
>
> This matters because if you want to use wheels from PyPI, the py27 wheels
> all assume one version of msvc, and the py33/py34 wheels all assume a
> different and incompatible version. (And py35 is yet another different and
> incompatible version.)
>
> -n
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20160616/44d350eb/attachment.html>