SciPy in conda-forge env on Windows
Hi, Sorry if this question is redundant. I checked the "install" page in the SciPy docs, and couldn't find any mention of conda-forge. Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the numpy-feedstock from conda-forge? When I run the tests for scipy, the fail and hang after "scipy\linalg\tests\test_basic.py". The reason I thought this might work is because they both use a variant of OpenBLAS, but perhaps they are not compatible? I can see that NumPy openblas.dll is in miniconda/envs/<envname>/Library/bin (31,053KB) where as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and the name is vendorized with a long alphanumeric string. Unfortunately if I try to use conda to install SciPy, conda uses the default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock has been disabled, and when I run the tests using this mix of conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", which I would expect because the anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need to be compatible for scipy to work, right? The test that fails is "TestDet.test_random()" scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] this is the pytest traceback: --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-3-ff0509141d7f> in <module>() ----> 1 test_det.test_random() ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py in test_random(self) 907 d1 = det(a) 908 d2 = basic_det(a) --> 909 assert_almost_equal(d1, d2) 910 911 def test_random_complex(self): ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py in assert_almost_equal(actual, desired, decimal, err_msg, verbose) 579 pass 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): --> 581 raise AssertionError(_build_err_msg()) 582 583 AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.001303440758814572 DESIRED: -0.09307217461347188 Then the next test hangs for several minute, and I have to kill the process. scipy/linalg/tests/test_basic.py::TestDet::test_random_complex It also hangs when I call it directly from an interpreter. If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy ( scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I can see the vendorized version of openBLAS for numpy in site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, the vendor alpha-numeric string is also the same: " BNVRK7633HSX7YVO2TADGR4A5KEKXJAW" Maybe this is very basic, and I shouldn't be asking. But I don't understand how to use conda-forge then with SciPy and NumPy on Windows. It seems like users should only use all pip or only the default "anaconda" channel. Or if using conda-forge on window, then only use NumPy without SciPy, but that also means other packages like statsmodels are out too. Also maybe this is a question for conda-forge and not scipy-dev. Again, so sorry if my question is out of place. I'm also sorry if this is received poorly. I am so happy and very grateful for all of the volunteer work that has gone into making SciPy and NumPy work on Windows. I remember the old days when I had to refer people to cgohlke's website. If there is anything I can do to help, I can try. I will take a look at reactivating the conda-forge build for win_amd64, I believe that would solve the problem. Also I will bring up the issue of dependency clashes with conda-forge as well. Thanks, Mark -- Mark Mikofski, PhD (2005) *Fiat Lux*
Mixing wheels and conda packages can cause some problems over time because of pip and conda getting confused about what's installed. But if you made a conda env, and then did a pip install, and that's all you've done, then that should work fine AFAIK. And the scipy wheels should be self contained and work correctly in any environment, including a conda env where numpy is using mkl or a different version of openblas. I don't know what's going wrong in your case, but I can at least say that as far as I know, it *should* work and you're hitting a genuine bug. On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski <mikofski@berkeley.edu> wrote:
Hi,
Sorry if this question is redundant. I checked the "install" page in the SciPy docs, and couldn't find any mention of conda-forge.
Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the numpy-feedstock from conda-forge? When I run the tests for scipy, the fail and hang after "scipy\linalg\tests\test_basic.py".
The reason I thought this might work is because they both use a variant of OpenBLAS, but perhaps they are not compatible? I can see that NumPy openblas.dll is in miniconda/envs/<envname>/Library/bin (31,053KB) where as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and the name is vendorized with a long alphanumeric string.
Unfortunately if I try to use conda to install SciPy, conda uses the default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock has been disabled, and when I run the tests using this mix of conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", which I would expect because the anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need to be compatible for scipy to work, right?
The test that fails is "TestDet.test_random()"
scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%]
this is the pytest traceback:
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-3-ff0509141d7f> in <module>() ----> 1 test_det.test_random()
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py in test_random(self) 907 d1 = det(a) 908 d2 = basic_det(a) --> 909 assert_almost_equal(d1, d2) 910 911 def test_random_complex(self):
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py in assert_almost_equal(actual, desired, decimal, err_msg, verbose) 579 pass 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): --> 581 raise AssertionError(_build_err_msg()) 582 583
AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.001303440758814572 DESIRED: -0.09307217461347188
Then the next test hangs for several minute, and I have to kill the process.
scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
It also hangs when I call it directly from an interpreter.
If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy ( scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I can see the vendorized version of openBLAS for numpy in site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, the vendor alpha-numeric string is also the same: " BNVRK7633HSX7YVO2TADGR4A5KEKXJAW"
Maybe this is very basic, and I shouldn't be asking. But I don't understand how to use conda-forge then with SciPy and NumPy on Windows. It seems like users should only use all pip or only the default "anaconda" channel. Or if using conda-forge on window, then only use NumPy without SciPy, but that also means other packages like statsmodels are out too. Also maybe this is a question for conda-forge and not scipy-dev. Again, so sorry if my question is out of place.
I'm also sorry if this is received poorly. I am so happy and very grateful for all of the volunteer work that has gone into making SciPy and NumPy work on Windows. I remember the old days when I had to refer people to cgohlke's website. If there is anything I can do to help, I can try. I will take a look at reactivating the conda-forge build for win_amd64, I believe that would solve the problem. Also I will bring up the issue of dependency clashes with conda-forge as well.
Thanks, Mark
-- Mark Mikofski, PhD (2005) *Fiat Lux* _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Thanks for your response. I tested it again, and I still hit the same issue. Here's my procedure: (base)> conda create -n forge (base)> conda activate forge (forge)> conda config --env --add channels conda-forge (forge)> conda install python=3 pip setuptools wheel (forge)> conda install nose pytest (forge)> conda install numpy (forge)>python
import numpy numpy.test()
Ran 6413 tests in 84.363s OK (KNOWNFAIL=19, SKIP=16) <nose.result.TextTestResult run=6413 errors=0 failures=0>
exit()
(forge)> pip install scipy (forge)> python
import scipy scipy.test(verbose=3, tests=['scipy.linalg.tests.test_basic'])
scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] scipy/linalg/tests/test_basic.py::TestDet::test_random_complex I have to kill the process here --- Note, the same thing happens if I install scipy from conda, which installs scipy from anaconda channel on win_amd64 b/c scipy-feedstock is currently disabled on conda-forge --- But, everything passes fine if I just install both numpy and scipy from pip. And everything passes fine if I install scipy first, which then installs numpy from the default "anaconda" channel --- I posted an issue on the numpy-feedstock, and a comment on the conda-forge google group: * https://github.com/conda-forge/numpy-feedstock/issues/82 * https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/conda-forge/PPLWn4U1ZHA/frHthukWCgAJ --- I'm really sorry if this is a red-herring or wild goose chase. I didn't mean to cause any trouble. I think following Chris Barker's advice, I should have installed scipy first, and then I wouldn't have been in this mess. Thanks! On Thu, Apr 19, 2018 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
Mixing wheels and conda packages can cause some problems over time because of pip and conda getting confused about what's installed. But if you made a conda env, and then did a pip install, and that's all you've done, then that should work fine AFAIK.
And the scipy wheels should be self contained and work correctly in any environment, including a conda env where numpy is using mkl or a different version of openblas.
I don't know what's going wrong in your case, but I can at least say that as far as I know, it *should* work and you're hitting a genuine bug.
On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski <mikofski@berkeley.edu> wrote:
Hi,
Sorry if this question is redundant. I checked the "install" page in the SciPy docs, and couldn't find any mention of conda-forge.
Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the numpy-feedstock from conda-forge? When I run the tests for scipy, the fail and hang after "scipy\linalg\tests\test_basic.py".
The reason I thought this might work is because they both use a variant of OpenBLAS, but perhaps they are not compatible? I can see that NumPy openblas.dll is in miniconda/envs/<envname>/Library/bin (31,053KB) where as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and the name is vendorized with a long alphanumeric string.
Unfortunately if I try to use conda to install SciPy, conda uses the default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock has been disabled, and when I run the tests using this mix of conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", which I would expect because the anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need to be compatible for scipy to work, right?
The test that fails is "TestDet.test_random()"
scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%]
this is the pytest traceback:
------------------------------------------------------------ --------------- AssertionError Traceback (most recent call last) <ipython-input-3-ff0509141d7f> in <module>() ----> 1 test_det.test_random()
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\ site-packages\scipy\linalg\tests\test_basic.py in test_random(self) 907 d1 = det(a) 908 d2 = basic_det(a) --> 909 assert_almost_equal(d1, d2) 910 911 def test_random_complex(self):
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\ site-packages\numpy\testing\nose_tools\utils.py in assert_almost_equal(actual, desired, decimal, err_msg, verbose) 579 pass 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): --> 581 raise AssertionError(_build_err_msg()) 582 583
AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.001303440758814572 DESIRED: -0.09307217461347188
Then the next test hangs for several minute, and I have to kill the process.
scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
It also hangs when I call it directly from an interpreter.
If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy (scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I can see the vendorized version of openBLAS for numpy in site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, the vendor alpha-numeric string is also the same: " BNVRK7633HSX7YVO2TADGR4A5KEKXJAW"
Maybe this is very basic, and I shouldn't be asking. But I don't understand how to use conda-forge then with SciPy and NumPy on Windows. It seems like users should only use all pip or only the default "anaconda" channel. Or if using conda-forge on window, then only use NumPy without SciPy, but that also means other packages like statsmodels are out too. Also maybe this is a question for conda-forge and not scipy-dev. Again, so sorry if my question is out of place.
I'm also sorry if this is received poorly. I am so happy and very grateful for all of the volunteer work that has gone into making SciPy and NumPy work on Windows. I remember the old days when I had to refer people to cgohlke's website. If there is anything I can do to help, I can try. I will take a look at reactivating the conda-forge build for win_amd64, I believe that would solve the problem. Also I will bring up the issue of dependency clashes with conda-forge as well.
Thanks, Mark
-- Mark Mikofski, PhD (2005) *Fiat Lux* _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
-- Mark Mikofski, PhD (2005) *Fiat Lux*
I'm suspicious about the really complex pattern of success/failure, including things that seem like they shouldn't matter, and the fact that the test is called "test_random". It does have a setUp method that attempts to seed the RNG, but maybe that's not working somehow, and it's actually a non-deterministic test failure? (To be clear: if it fails at all then that's probably a bug, but it might be a bug that's happening in all these configurations but only sometimes results in a test failure.) You should file a bug at https://github.com/scipy/scipy/issues -n On Thu, Apr 19, 2018 at 5:14 PM, Mark Alexander Mikofski <mikofski@berkeley.edu> wrote:
Thanks for your response. I tested it again, and I still hit the same issue.
Here's my procedure:
(base)> conda create -n forge (base)> conda activate forge (forge)> conda config --env --add channels conda-forge (forge)> conda install python=3 pip setuptools wheel (forge)> conda install nose pytest (forge)> conda install numpy (forge)>python
import numpy numpy.test()
Ran 6413 tests in 84.363s
OK (KNOWNFAIL=19, SKIP=16) <nose.result.TextTestResult run=6413 errors=0 failures=0>
exit()
(forge)> pip install scipy (forge)> python
import scipy scipy.test(verbose=3, tests=['scipy.linalg.tests.test_basic'])
scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
I have to kill the process here
---
Note, the same thing happens if I install scipy from conda, which installs scipy from anaconda channel on win_amd64 b/c scipy-feedstock is currently disabled on conda-forge
---
But, everything passes fine if I just install both numpy and scipy from pip.
And everything passes fine if I install scipy first, which then installs numpy from the default "anaconda" channel
---
I posted an issue on the numpy-feedstock, and a comment on the conda-forge google group:
* https://github.com/conda-forge/numpy-feedstock/issues/82
---
I'm really sorry if this is a red-herring or wild goose chase. I didn't mean to cause any trouble. I think following Chris Barker's advice, I should have installed scipy first, and then I wouldn't have been in this mess.
Thanks!
On Thu, Apr 19, 2018 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
Mixing wheels and conda packages can cause some problems over time because of pip and conda getting confused about what's installed. But if you made a conda env, and then did a pip install, and that's all you've done, then that should work fine AFAIK.
And the scipy wheels should be self contained and work correctly in any environment, including a conda env where numpy is using mkl or a different version of openblas.
I don't know what's going wrong in your case, but I can at least say that as far as I know, it *should* work and you're hitting a genuine bug.
On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski <mikofski@berkeley.edu> wrote:
Hi,
Sorry if this question is redundant. I checked the "install" page in the SciPy docs, and couldn't find any mention of conda-forge.
Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the numpy-feedstock from conda-forge? When I run the tests for scipy, the fail and hang after "scipy\linalg\tests\test_basic.py".
The reason I thought this might work is because they both use a variant of OpenBLAS, but perhaps they are not compatible? I can see that NumPy openblas.dll is in miniconda/envs/<envname>/Library/bin (31,053KB) where as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and the name is vendorized with a long alphanumeric string.
Unfortunately if I try to use conda to install SciPy, conda uses the default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock has been disabled, and when I run the tests using this mix of conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", which I would expect because the anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need to be compatible for scipy to work, right?
The test that fails is "TestDet.test_random()"
scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%]
this is the pytest traceback:
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-3-ff0509141d7f> in <module>() ----> 1 test_det.test_random()
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py in test_random(self) 907 d1 = det(a) 908 d2 = basic_det(a) --> 909 assert_almost_equal(d1, d2) 910 911 def test_random_complex(self):
~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py in assert_almost_equal(actual, desired, decimal, err_msg, verbose) 579 pass 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): --> 581 raise AssertionError(_build_err_msg()) 582 583
AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.001303440758814572 DESIRED: -0.09307217461347188
Then the next test hangs for several minute, and I have to kill the process.
scipy/linalg/tests/test_basic.py::TestDet::test_random_complex
It also hangs when I call it directly from an interpreter.
If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy (scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I can see the vendorized version of openBLAS for numpy in site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, the vendor alpha-numeric string is also the same: "BNVRK7633HSX7YVO2TADGR4A5KEKXJAW"
Maybe this is very basic, and I shouldn't be asking. But I don't understand how to use conda-forge then with SciPy and NumPy on Windows. It seems like users should only use all pip or only the default "anaconda" channel. Or if using conda-forge on window, then only use NumPy without SciPy, but that also means other packages like statsmodels are out too. Also maybe this is a question for conda-forge and not scipy-dev. Again, so sorry if my question is out of place.
I'm also sorry if this is received poorly. I am so happy and very grateful for all of the volunteer work that has gone into making SciPy and NumPy work on Windows. I remember the old days when I had to refer people to cgohlke's website. If there is anything I can do to help, I can try. I will take a look at reactivating the conda-forge build for win_amd64, I believe that would solve the problem. Also I will bring up the issue of dependency clashes with conda-forge as well.
Thanks, Mark
-- Mark Mikofski, PhD (2005) Fiat Lux _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
-- Mark Mikofski, PhD (2005) Fiat Lux
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
-- Nathaniel J. Smith -- https://vorpus.org
participants (2)
-
Mark Alexander Mikofski
-
Nathaniel Smith