[SciPy-Dev] runtest issue mac os with master 14de485c053cc275682bc7de0316fcca72b341f8

[Matthew Brett]

On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>> >
>>> >
>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>> > <mikofski at berkeley.edu> wrote:
>>> >>
>>> >> Hi All,
>>> >>
>>> >> I am having issues using
>>> >>
>>> >>     python runtest.py
>>> >>
>>> >> from scipy master branch since commit
>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>> >>
>>> >>
>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>> >>
>>> >> there are two tracebacks, I think this may have something to do with
>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>> >> and
>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>> >> try
>>> >> that)
>>> >
>>> >
>>> > Yes that is the recommendation (if you're not using Anaconda).
>>> >
>>> >
>>> >>
>>> >> Use anaconda and create a conda env? (maybe this next?)
>>> >
>>> >
>>> > If you're already using Anaconda, and the numpy from the defaults
>>> > channel,
>>> > SciPy should automatically link against MKL.
>>> >
>>> >
>>> >>
>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>> >> wheels
>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>> >> interpreter, I do pip install requirements.txt (see attached).
>>> >>
>>> >> 1st traceback:
>>> >>
>>> >> ImportError while importing test module
>>> >>
>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>> >> Hint: make sure your test modules/packages have valid Python names.
>>> >> Traceback:
>>> >> scipy/cluster/__init__.py:27: in <module>
>>> >>     from . import vq, hierarchy
>>> >> scipy/cluster/vq.py:75: in <module>
>>> >>     from scipy.spatial.distance import cdist
>>> >> scipy/spatial/__init__.py:97: in <module>
>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>> >>     from scipy.spatial.distance import pdist
>>> >> scipy/spatial/distance.py:123: in <module>
>>> >>     from ..linalg import norm
>>> >> scipy/linalg/__init__.py:207: in <module>
>>> >>     from ._decomp_update import *
>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>> >>     ???
>>> >> E   ImportError:
>>> >>
>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>> >> 2): Symbol not found: _cbbcsd_
>>> >> E     Referenced from:
>>> >>
>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>> >> E     Expected in: flat namespace
>>> >> E    in
>>> >>
>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>
>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>
>>> * Python 3.4:
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>> * Python 3.5:
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>
>>
>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>> site.cfg or install it to one of the regular locations):
>>
>> openblas_info:
>>
>>   libraries  not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>> atlas_blas_threads_info:
>>
>> Setting PTATLAS=ATLAS
>>
>>   libraries ptf77blas,ptcblas,atlas not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>> atlas_blas_info:
>>
>>   libraries f77blas,cblas,atlas not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>>   FOUND:
>>
>>     libraries = ['blas']
>>
>>     library_dirs = ['/usr/lib']
>>
>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>
>>     language = f77
>
> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
> /usr/local/lib, where it should be detected.  On Python 3.6, using the
> same machinery, it is detected - see:
>
> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606

It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
1.9.3 does find it, which is why the build is OK for Python 3.6.
Writing an explicit site.cfg:

    cat << EOF > $HOME/site.cfg
[openblas]
libraries = openblas
library_dirs = /usr/local/lib
include_dirs = /usr/local/include
runtime_library_dirs = /usr/local/lib
EOF

fixes that locally.  Testing on Travis-CI now:

https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944

Cheers,

Matthew