[SciPy-Dev] runtest issue mac os with master 14de485c053cc275682bc7de0316fcca72b341f8

Ralf Gommers ralf.gommers at gmail.com
Wed Jun 6 12:53:01 EDT 2018 

On Wed, Jun 6, 2018 at 3:55 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >>> Hi,
> >>>
> >>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >>>>
> >>>>
> >>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <
> matthew.brett at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com
> >
> >>>>> wrote:
> >>>>> >
> >>>>> >
> >>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
> >>>>> > <mikofski at berkeley.edu> wrote:
> >>>>> >>
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> I am having issues using
> >>>>> >>
> >>>>> >>     python runtest.py
> >>>>> >>
> >>>>> >> from scipy master branch since commit
> >>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
> >>>>> >>
> >>>>> >>
> >>>>> >> https://github.com/scipy/scipy/tree/
> 14de485c053cc275682bc7de0316fcca72b341f8
> >>>>> >>
> >>>>> >> there are two tracebacks, I think this may have something to do
> with
> >>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not
> sure,
> >>>>> >> and
> >>>>> >> don't know how to fix it. Should I brew install openblas? (I'm
> going to
> >>>>> >> try
> >>>>> >> that)
> >>>>> >
> >>>>> >
> >>>>> > Yes that is the recommendation (if you're not using Anaconda).
> >>>>> >
> >>>>> >
> >>>>> >>
> >>>>> >> Use anaconda and create a conda env? (maybe this next?)
> >>>>> >
> >>>>> >
> >>>>> > If you're already using Anaconda, and the numpy from the defaults
> >>>>> > channel,
> >>>>> > SciPy should automatically link against MKL.
> >>>>> >
> >>>>> >
> >>>>> >>
> >>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
> >>>>> >> wheels
> >>>>> >> from PyPI, after I create the virtualenv using the homebrew
> python-3
> >>>>> >> interpreter, I do pip install requirements.txt (see attached).
> >>>>> >>
> >>>>> >> 1st traceback:
> >>>>> >>
> >>>>> >> ImportError while importing test module
> >>>>> >>
> >>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
> >>>>> >> Hint: make sure your test modules/packages have valid Python
> names.
> >>>>> >> Traceback:
> >>>>> >> scipy/cluster/__init__.py:27: in <module>
> >>>>> >>     from . import vq, hierarchy
> >>>>> >> scipy/cluster/vq.py:75: in <module>
> >>>>> >>     from scipy.spatial.distance import cdist
> >>>>> >> scipy/spatial/__init__.py:97: in <module>
> >>>>> >>     from ._spherical_voronoi import SphericalVoronoi
> >>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
> >>>>> >>     from scipy.spatial.distance import pdist
> >>>>> >> scipy/spatial/distance.py:123: in <module>
> >>>>> >>     from ..linalg import norm
> >>>>> >> scipy/linalg/__init__.py:207: in <module>
> >>>>> >>     from ._decomp_update import *
> >>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
> >>>>> >>     ???
> >>>>> >> E   ImportError:
> >>>>> >>
> >>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/
> lib/python3.6/site-packages/scipy/linalg/cython_lapack.
> cpython-36m-darwin.so,
> >>>>> >> 2): Symbol not found: _cbbcsd_
> >>>>> >> E     Referenced from:
> >>>>> >>
> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
> >>>>> >> E     Expected in: flat namespace
> >>>>> >> E    in
> >>>>> >>
> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
> >>>>>
> >>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> >>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
> >>>>>
> >>>>> * Python 3.4:
> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
> >>>>> * Python 3.5:
> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
> >>>>
> >>>>
> >>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either
> need a
> >>>> site.cfg or install it to one of the regular locations):
> >>>>
> >>>> openblas_info:
> >>>>
> >>>>   libraries  not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>> atlas_blas_threads_info:
> >>>>
> >>>> Setting PTATLAS=ATLAS
> >>>>
> >>>>   libraries ptf77blas,ptcblas,atlas not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>> atlas_blas_info:
> >>>>
> >>>>   libraries f77blas,cblas,atlas not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>>   FOUND:
> >>>>
> >>>>     libraries = ['blas']
> >>>>
> >>>>     library_dirs = ['/usr/lib']
> >>>>
> >>>>     define_macros = [('NO_ATLAS_INFO', 1)]
> >>>>
> >>>>     language = f77
> >>>
> >>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
> >>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
> >>> same machinery, it is detected - see:
> >>>
> >>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
> >>
> >> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
> >> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
> >> 1.9.3 does find it, which is why the build is OK for Python 3.6.
> >> Writing an explicit site.cfg:
> >>
> >>     cat << EOF > $HOME/site.cfg
> >> [openblas]
> >> libraries = openblas
> >> library_dirs = /usr/local/lib
> >> include_dirs = /usr/local/include
> >> runtime_library_dirs = /usr/local/lib
> >> EOF
> >>
> >> fixes that locally.  Testing on Travis-CI now:
> >>
> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
> >
> > Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
> > with the site.cfg, giving
> >
> > ld: unknown option: -rpath=/usr/local/lib
> >
> > https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
> >
> > I can replicate this locally.  Deleting site.cfg fixes the error.
> >
> > The other builds continue to be broken with the same kind of errors;
> > they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
> > I think it needs blacklisting.  I hope that would make it possible to
> > build with numpy 1.9.3 at least, making sure to avoid having a
> > site.cfg.
>
> All macOS builds work correctly if I force numpy 1.11.3.
>
> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456
>
> Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?
>

There were a lot of changes to system_info related to rpath and
default_lib_dirs handling, plus some reordering of the check for MKL,
addition of BLIS, etc. Could be any of those, bisecting is probably easiest
here.


>
> Building wheels against 1.11.3 would be a fairly significant leap in
> numpy requirement, so it would be better if we could build with numpy
> 1.9.3, at least.
>

A jump to 1.11.3 would still support 4 numpy versions (1.11-1.14), and 1.15
will arrive before scipy 1.2.0. So not a big problem to make 1.11.3 the
minimum required version.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180606/e13450c4/attachment.html>

More information about the SciPy-Dev mailing list