Debugging NumPy development build in conda environment
Dear devs, If this is the wrong place for this kind of question I apologize and feel free to ignore this or point me to the right place. I'm using the conda package manager for my development environment which used to work just fine with these steps:
conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage source activate dev-numpy conda uninstall --force numpy numpy-base pip install -e .
This would yield me a nice isolated playground and I could use
python setup.py build_ext --incplace -j 4
at anytime to rebuild the binaries. Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with
RuntimeError: Broken toolchain: cannot link a simple C program
(full output in [1]) on a clean repository. No recent changes to my tool chain come to mind which could help me pinpoint the problem. Because it used to work just fine I have the annoying feeling that the fix is something small and trivial and have the hope that it's immediately obvious to someone here or that you can suggest a way to debug this. In any case your help would be appreciated. I'm aware that this may be an exotic dev setup. Using a virtual environment instead of the conda one works just fine. The build script seems to use a different linear algebra library in that case: blas from my OS as opposed to mkl in the conda environment. This is not my preferred solution but it would do if nothing comes of this. Some additional facts and research I've done: - Using conda environments with asv has stopped working as well due to not finding any LA library at all. - If I'm understanding [1] correctly the build script finds the mkl library provided by the conda environment. - Trying earlier versions of NumPy shows the same behavior. I tested the tags v1.14.6, v1.15.4. - However creating a development environment for SciPy and scikit-image using this approach still works fine. - Searching the internet or looking at NumPy's build guide didn't yield me a solution. - I'm on linux (Manjaro) if that is of any relevance. - Miniconda was reinstalled in my home directory with the installation script provided through Anaconda's website. - Haven't yet tried this approach in a fresh virtual machine. I hope that I haven't overlooked something obvious. In any case thank you for your time and attention. Cheers, Lars [1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5
On Friday, Jan 04, 2019 at 2:44 PM, Lars Grueter <lagru@mailbox.org (mailto:lagru@mailbox.org)> wrote: Dear devs,
If this is the wrong place for this kind of question I apologize and feel free to ignore this or point me to the right place.
I'm using the conda package manager for my development environment which used to work just fine with these steps:
conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage source activate dev-numpy conda uninstall --force numpy numpy-base pip install -e .
This would yield me a nice isolated playground and I could use
python setup.py build_ext --incplace -j 4
at anytime to rebuild the binaries. Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with
RuntimeError: Broken toolchain: cannot link a simple C program
(full output in [1]) on a clean repository. No recent changes to my tool chain come to mind which could help me pinpoint the problem. Because it used to work just fine I have the annoying feeling that the fix is something small and trivial and have the hope that it's immediately obvious to someone here or that you can suggest a way to debug this. In any case your help would be appreciated.
I'm aware that this may be an exotic dev setup. Using a virtual environment instead of the conda one works just fine. The build script seems to use a different linear algebra library in that case: blas from my OS as opposed to mkl in the conda environment. This is not my preferred solution but it would do if nothing comes of this.
Some additional facts and research I've done:
- Using conda environments with asv has stopped working as well due to not finding any LA library at all. - If I'm understanding [1] correctly the build script finds the mkl library provided by the conda environment. - Trying earlier versions of NumPy shows the same behavior. I tested the tags v1.14.6, v1.15.4. - However creating a development environment for SciPy and scikit-image using this approach still works fine. - Searching the internet or looking at NumPy's build guide didn't yield me a solution. - I'm on linux (Manjaro) if that is of any relevance. - Miniconda was reinstalled in my home directory with the installation script provided through Anaconda's website. - Haven't yet tried this approach in a fresh virtual machine.
I hope that I haven't overlooked something obvious. In any case thank you for your time and attention.
Cheers, Lars
[1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5 _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Hi Lars, What I’m used to doing is simply the following (inside the NumPy root):
conda create -n numpy-dev python=3[.x] pytest conda activate numpy-dev
python setup.py build_ext --inplace -j 4
pip install -e .
Often, when I run into issues with the build, I simply do this:
[backup site.cfg] git clean -xfd
[restore site.cfg]
python setup.py build_ext --inplace -j 4
pip install -e .
In your case, it might be the second that helps. If it’s not finding any LA library, you can try compiling with the site.cfg that points to MKL. Cheers and Best Regards, Hameer Abbasi
Sorry, I just noticed that I didn't send the message below to the mailing list but replied directly to Hameer. I've resent it for the archives below. @Hameer, sorry for the spam. :) Best regards, Lars On 04/01/2019 20:06, Lars Grueter wrote:> Hi Hameer,
thanks for the fast answer! I'm even more confused now. If any of my assumptions below are wrong please point them out.
On 04.01.19 14:53, Hameer Abbasi wrote:
Hi Lars,
What I’m used to doing is simply the following (inside the NumPy root):
conda create -n numpy-dev python=3[.x] pytest conda activate numpy-dev python setup.py build_ext --inplace -j 4
This fails right here telling me that it doesn't find any LA library such as ATLAS, BLIS, MKL, etc. (assuming I read the output correctly). The error message is however different from the workflow I described in my first mail. It's missing the sections à la
lapack_mkl_info: customize UnixCCompiler FOUND: libraries = ['mkl_rt', 'pthread']
I checked my OS's package manager and found blas, cblas and lapack. Aren't these LA libraries? Am I still missing an OS version of either ATLAS, BLIS or MKL?
https://docs.scipy.org/doc/numpy/user/building.html#disabling-atlas-and-othe... tells me it should be possible to build NumPy without these libraries. This has no effect and won't build.
But if that's the case why is the build working inside an environment created with virtualenv? Is it building without those libraries?
Often, when I run into issues with the build, I simply do this:
[backup site.cfg] git clean -xfd [restore site.cfg] python setup.py build_ext --inplace -j 4 pip install -e . > In your case, it might be the second that helps. If it’s not finding any LA library, you can try compiling with the site.cfg that points to MKL.
I didn't yet need my own site.cfg. I tried creating one myself right now but regardless of how I try to point it to the conda envs lib and include directories it has now effect on the output of the setup.py command. The build script already finds these paths by default. E.g.:
[mkl] library_dirs = /home/lg/.miniconda3/envs/dev-numpy/lib include_dirs = /home/lg/.miniconda3/envs/dev-numpy/include mkl_libs = mkl_rt lapack_libs =
I'm not sure what else to provide in this file.
Thanks again and best regards, Lars
On 4/1/19 3:34 pm, Lars Grueter wrote:
Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with
RuntimeError: Broken toolchain: cannot link a simple C program
The compiler error is a few lines above this (copied below). When I have gotten that kind of output, it is because I am using a 32 bit compiler on 64 bit or the opposite. C compiler: gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c' gcc: _configtest.c gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ _configtest.o -o _configtest /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info _configtest.o: file not recognized: file format not recognized Can you compile anything with that gcc? Matti
On 04/01/2019 14:55, Matti Picus wrote:
On 4/1/19 3:34 pm, Lars Grueter wrote:
Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with
RuntimeError: Broken toolchain: cannot link a simple C program
The compiler error is a few lines above this (copied below). When I have gotten that kind of output, it is because I am using a 32 bit compiler on 64 bit or the opposite.
[...]
Can you compile anything with that gcc?
This problem even persists on a new system. I noticed that conda includes libgcc-ng [1] in every environment with Python. Is it possible that this library (or another one) is incompatible with the GCC version installed on my system? That would also explain why changing the available LA libraries has not effect and why the setup works fine using plain virtual environments created with venv. Is someone else successfully maintaining a development setup with conda on an up-to-date Arch-based system or even Manjaro? Best regards, Lars [1] http://docs.continuum.io/anaconda/packages/py3.7_linux-64/
participants (3)
-
Hameer Abbasi -
Lars Grueter -
Matti Picus