Debugging NumPy development build in conda environment
Dear devs, If this is the wrong place for this kind of question I apologize and feel free to ignore this or point me to the right place. I'm using the conda package manager for my development environment which used to work just fine with these steps:
This would yield me a nice isolated playground and I could use
python setup.py build_ext --incplace -j 4
at anytime to rebuild the binaries. Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with
RuntimeError: Broken toolchain: cannot link a simple C program
(full output in [1]) on a clean repository. No recent changes to my tool chain come to mind which could help me pinpoint the problem. Because it used to work just fine I have the annoying feeling that the fix is something small and trivial and have the hope that it's immediately obvious to someone here or that you can suggest a way to debug this. In any case your help would be appreciated. I'm aware that this may be an exotic dev setup. Using a virtual environment instead of the conda one works just fine. The build script seems to use a different linear algebra library in that case: blas from my OS as opposed to mkl in the conda environment. This is not my preferred solution but it would do if nothing comes of this. Some additional facts and research I've done: - Using conda environments with asv has stopped working as well due to not finding any LA library at all. - If I'm understanding [1] correctly the build script finds the mkl library provided by the conda environment. - Trying earlier versions of NumPy shows the same behavior. I tested the tags v1.14.6, v1.15.4. - However creating a development environment for SciPy and scikit-image using this approach still works fine. - Searching the internet or looking at NumPy's build guide didn't yield me a solution. - I'm on linux (Manjaro) if that is of any relevance. - Miniconda was reinstalled in my home directory with the installation script provided through Anaconda's website. - Haven't yet tried this approach in a fresh virtual machine. I hope that I haven't overlooked something obvious. In any case thank you for your time and attention. Cheers, Lars [1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5
Hi Lars, What I’m used to doing is simply the following (inside the NumPy root):
Often, when I run into issues with the build, I simply do this:
In your case, it might be the second that helps. If it’s not finding any LA library, you can try compiling with the site.cfg that points to MKL. Cheers and Best Regards, Hameer Abbasi
On 4/1/19 3:34 pm, Lars Grueter wrote:
The compiler error is a few lines above this (copied below). When I have gotten that kind of output, it is because I am using a 32 bit compiler on 64 bit or the opposite. C compiler: gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c' gcc: _configtest.c gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ _configtest.o -o _configtest /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info _configtest.o: file not recognized: file format not recognized Can you compile anything with that gcc? Matti
On 04/01/2019 14:55, Matti Picus wrote:
This problem even persists on a new system. I noticed that conda includes libgcc-ng [1] in every environment with Python. Is it possible that this library (or another one) is incompatible with the GCC version installed on my system? That would also explain why changing the available LA libraries has not effect and why the setup works fine using plain virtual environments created with venv. Is someone else successfully maintaining a development setup with conda on an up-to-date Arch-based system or even Manjaro? Best regards, Lars [1] http://docs.continuum.io/anaconda/packages/py3.7_linux-64/
Hi Lars, What I’m used to doing is simply the following (inside the NumPy root):
Often, when I run into issues with the build, I simply do this:
In your case, it might be the second that helps. If it’s not finding any LA library, you can try compiling with the site.cfg that points to MKL. Cheers and Best Regards, Hameer Abbasi
On 4/1/19 3:34 pm, Lars Grueter wrote:
The compiler error is a few lines above this (copied below). When I have gotten that kind of output, it is because I am using a 32 bit compiler on 64 bit or the opposite. C compiler: gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c' gcc: _configtest.c gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ _configtest.o -o _configtest /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info _configtest.o: file not recognized: file format not recognized Can you compile anything with that gcc? Matti
On 04/01/2019 14:55, Matti Picus wrote:
This problem even persists on a new system. I noticed that conda includes libgcc-ng [1] in every environment with Python. Is it possible that this library (or another one) is incompatible with the GCC version installed on my system? That would also explain why changing the available LA libraries has not effect and why the setup works fine using plain virtual environments created with venv. Is someone else successfully maintaining a development setup with conda on an up-to-date Arch-based system or even Manjaro? Best regards, Lars [1] http://docs.continuum.io/anaconda/packages/py3.7_linux-64/
participants (3)
-
Hameer Abbasi -
Lars Grueter -
Matti Picus