[SciPy-Dev] runtest issue mac os with master 14de485c053cc275682bc7de0316fcca72b341f8
Matthew Brett
matthew.brett at gmail.com
Wed Jun 6 17:28:41 EDT 2018
On Wed, Jun 6, 2018 at 5:53 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Wed, Jun 6, 2018 at 3:55 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>> > On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com>
>> >> wrote:
>> >>> Hi,
>> >>>
>> >>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>>
>> >>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett
>> >>>> <matthew.brett at gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers
>> >>>>> <ralf.gommers at gmail.com>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>> >>>>> > <mikofski at berkeley.edu> wrote:
>> >>>>> >>
>> >>>>> >> Hi All,
>> >>>>> >>
>> >>>>> >> I am having issues using
>> >>>>> >>
>> >>>>> >> python runtest.py
>> >>>>> >>
>> >>>>> >> from scipy master branch since commit
>> >>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>> >>>>> >>
>> >>>>> >> there are two tracebacks, I think this may have something to do
>> >>>>> >> with
>> >>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not
>> >>>>> >> sure,
>> >>>>> >> and
>> >>>>> >> don't know how to fix it. Should I brew install openblas? (I'm
>> >>>>> >> going to
>> >>>>> >> try
>> >>>>> >> that)
>> >>>>> >
>> >>>>> >
>> >>>>> > Yes that is the recommendation (if you're not using Anaconda).
>> >>>>> >
>> >>>>> >
>> >>>>> >>
>> >>>>> >> Use anaconda and create a conda env? (maybe this next?)
>> >>>>> >
>> >>>>> >
>> >>>>> > If you're already using Anaconda, and the numpy from the defaults
>> >>>>> > channel,
>> >>>>> > SciPy should automatically link against MKL.
>> >>>>> >
>> >>>>> >
>> >>>>> >>
>> >>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>> >>>>> >> wheels
>> >>>>> >> from PyPI, after I create the virtualenv using the homebrew
>> >>>>> >> python-3
>> >>>>> >> interpreter, I do pip install requirements.txt (see attached).
>> >>>>> >>
>> >>>>> >> 1st traceback:
>> >>>>> >>
>> >>>>> >> ImportError while importing test module
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>> >>>>> >> Hint: make sure your test modules/packages have valid Python
>> >>>>> >> names.
>> >>>>> >> Traceback:
>> >>>>> >> scipy/cluster/__init__.py:27: in <module>
>> >>>>> >> from . import vq, hierarchy
>> >>>>> >> scipy/cluster/vq.py:75: in <module>
>> >>>>> >> from scipy.spatial.distance import cdist
>> >>>>> >> scipy/spatial/__init__.py:97: in <module>
>> >>>>> >> from ._spherical_voronoi import SphericalVoronoi
>> >>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>> >>>>> >> from scipy.spatial.distance import pdist
>> >>>>> >> scipy/spatial/distance.py:123: in <module>
>> >>>>> >> from ..linalg import norm
>> >>>>> >> scipy/linalg/__init__.py:207: in <module>
>> >>>>> >> from ._decomp_update import *
>> >>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>> >>>>> >> ???
>> >>>>> >> E ImportError:
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>> >>>>> >> 2): Symbol not found: _cbbcsd_
>> >>>>> >> E Referenced from:
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >>>>> >> E Expected in: flat namespace
>> >>>>> >> E in
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >>>>>
>> >>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>> >>>>> macOS machines, using our usual wheel-building procedure
>> >>>>> (+OpenBLAS):
>> >>>>>
>> >>>>> * Python 3.4:
>> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>> >>>>> * Python 3.5:
>> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>> >>>>
>> >>>>
>> >>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either
>> >>>> need a
>> >>>> site.cfg or install it to one of the regular locations):
>> >>>>
>> >>>> openblas_info:
>> >>>>
>> >>>> libraries not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>> NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>> atlas_blas_threads_info:
>> >>>>
>> >>>> Setting PTATLAS=ATLAS
>> >>>>
>> >>>> libraries ptf77blas,ptcblas,atlas not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>> NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>> atlas_blas_info:
>> >>>>
>> >>>> libraries f77blas,cblas,atlas not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>> NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>> FOUND:
>> >>>>
>> >>>> libraries = ['blas']
>> >>>>
>> >>>> library_dirs = ['/usr/lib']
>> >>>>
>> >>>> define_macros = [('NO_ATLAS_INFO', 1)]
>> >>>>
>> >>>> language = f77
>> >>>
>> >>> Hmm - you're right - that's very odd. OpenBLAS is being unpacked into
>> >>> /usr/local/lib, where it should be detected. On Python 3.6, using the
>> >>> same machinery, it is detected - see:
>> >>>
>> >>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>> >>
>> >> It looks like it's numpy distutils. numpy 1.8.2 doesn't appear to find
>> >> OpenBLAS in /usr/local/lib, even though it claims it looked. Numpy
>> >> 1.9.3 does find it, which is why the build is OK for Python 3.6.
>> >> Writing an explicit site.cfg:
>> >>
>> >> cat << EOF > $HOME/site.cfg
>> >> [openblas]
>> >> libraries = openblas
>> >> library_dirs = /usr/local/lib
>> >> include_dirs = /usr/local/include
>> >> runtime_library_dirs = /usr/local/lib
>> >> EOF
>> >>
>> >> fixes that locally. Testing on Travis-CI now:
>> >>
>> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
>> >
>> > Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
>> > with the site.cfg, giving
>> >
>> > ld: unknown option: -rpath=/usr/local/lib
>> >
>> > https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
>> >
>> > I can replicate this locally. Deleting site.cfg fixes the error.
>> >
>> > The other builds continue to be broken with the same kind of errors;
>> > they pick up OpenBLAS, but also the /usr/lib stuff. So, you're right,
>> > I think it needs blacklisting. I hope that would make it possible to
>> > build with numpy 1.9.3 at least, making sure to avoid having a
>> > site.cfg.
>>
>> All macOS builds work correctly if I force numpy 1.11.3.
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456
>>
>> Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?
>
>
> There were a lot of changes to system_info related to rpath and
> default_lib_dirs handling, plus some reordering of the check for MKL,
> addition of BLIS, etc. Could be any of those, bisecting is probably easiest
> here.
>
>>
>>
>> Building wheels against 1.11.3 would be a fairly significant leap in
>> numpy requirement, so it would be better if we could build with numpy
>> 1.9.3, at least.
>
>
> A jump to 1.11.3 would still support 4 numpy versions (1.11-1.14), and 1.15
> will arrive before scipy 1.2.0. So not a big problem to make 1.11.3 the
> minimum required version.
1.11.3 is only 1 year 6 months old, it would be a shame to force that
if we could avoid it...
Cheers,
Matthew
More information about the SciPy-Dev
mailing list