Numscons issues: numpy.core.umath_tests not built, built-in ld detection, MAIN__ not being set-up
I've got a few issues that I hope won't be overwhelming on one message: (1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128). $ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests What needs to be done to get this module incorporated into the numscons build? (2) I've found that in numscons-0.9.4, the detection of the correct linker assumes that if gcc is in use, the linker is gnu ld. However, on solaris this isn't the recommended toolchain, so it's typical to build gcc with gnu as and the solaris /usr/ccs/bin/ld under the hood. What this means is that when setting a run_path in the binary (which we need to do) the linker flags are set to "-Wl,-rpath=<library>". However, this isn't valid for the solaris ld. It needs -R<libname>, or -Wl,-R<libname>. I'm pretty sure that on Solaris trying to link a library with -Wl,-rpath= and looking for an error should be enough to determine the correct format for the linker. (3) Numscons tries to check for the need for a MAIN__ function when linking with gfortran. However, any libraries built with numscons come out with an unsatisfied dependency on MAIN__. The log looks like this in build/scons/numpy/linalg/config.log looks like this: scons: Configure: Checking if gfortran needs dummy main - scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.c" is up to date. scons: Configure: The original builder output was: |build/scons/numpy/linalg/sconf/conftest_0.c <- | | | |int dummy() { return 0; } | | | scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.o" is up to date. scons: Configure: The original builder output was: |gcc -o build/scons/numpy/linalg/sconf/conftest_0.o -c -O3 -m64 -g -fPIC -DPIC build/scons/numpy/linalg/sconf/conftest_0.c | scons: Configure: Building "build/scons/numpy/linalg/sconf/conftest_0" failed in a previous run and all its sources are up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_0 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_0.o | It then goes on to discover that it needs main: scons: Configure: "build/scons/numpy/linalg/sconf/conftest_1" is up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_1 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_1.o | scons: Configure: (cached) MAIN__. Doesn't this clearly indicate that a dummy main is needed? I'm working around this with a silly library that just has the MAIN__ symbol in it, but I'd love to do without that. Thanks, Peter
On Tue, Dec 9, 2008 at 4:50 PM, Peter Norton < spacey-numpy-discussion@lenin.net> wrote:
I've got a few issues that I hope won't be overwhelming on one message:
(1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128).
$ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests
What needs to be done to get this module incorporated into the numscons build?
It's also commented out of the usual setup.py file also because of blas/lapack linkage problems that need to be fixed; I was working on other things. It's probably time to fix it. Chuck
Peter Norton wrote:
I've got a few issues that I hope won't be overwhelming on one message:
(1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128).
$ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests
What needs to be done to get this module incorporated into the numscons build?
you should not need this module, it is not built using the normal build of numpy either. Did you do a clean build (rm -rf build and removing the install directory first) ? It was enabled before but is commented out ATM.
(2) I've found that in numscons-0.9.4, the detection of the correct linker assumes that if gcc is in use, the linker is gnu ld. However, on solaris this isn't the recommended toolchain, so it's typical to build gcc with gnu as and the solaris /usr/ccs/bin/ld under the hood. What this means is that when setting a run_path in the binary (which we need to do) the linker flags are set to "-Wl,-rpath=<library>". However, this isn't valid for the solaris ld. It needs -R<libname>, or -Wl,-R<libname>. I'm pretty sure that on Solaris trying to link a library with -Wl,-rpath= and looking for an error should be enough to determine the correct format for the linker.
Scons and hence numscons indeed assume that the linker is the same as the compiler by default. It would be possible to avoid this by detecting the linker at runtime, to bypass scons tools choice, like I do for C, C++ and Fortran compilers. The whole scons tools sub-system is unfortunately very limited ATM, so there is a lot of manual work to do (that's actually what most of the code in numscons/core is for).
(3) Numscons tries to check for the need for a MAIN__ function when linking with gfortran. However, any libraries built with numscons come out with an unsatisfied dependency on MAIN__. The log looks like this in build/scons/numpy/linalg/config.log looks like this:
It may be linked to the sun linker problem above. Actually, the dummy main detection is not used at all for the building - it is necessary to detect name mangling used by the fortran compiler, but that's it. I assumed that a dummy main was never needed for shared libraries, but that assumption may well be ill founded. I never had problems related to this on open solaris, with both native and gcc toolchains, so I am willing to investiage first whether it is linked to the sun linker problem or not. Unfortunately, I won't have the time to work on this in the next few months because of my PhD thesis; the sun linker problem can be fixed by following a strategy similar to compilers, in numscons/core/initialization.py. You first need to add a detection scheme for the linker in compiler_detection.py. David
On Wed, Dec 10, 2008 at 12:59 AM, David Cournapeau
Peter Norton wrote:
I've got a few issues that I hope won't be overwhelming on one message:
(1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128).
$ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests
What needs to be done to get this module incorporated into the numscons build?
you should not need this module, it is not built using the normal build of numpy either. Did you do a clean build (rm -rf build and removing the install directory first) ? It was enabled before but is commented out ATM.
Our users would like to have this module for testing purposes I believe. It should be enabled. - Hide quoted text -
(2) I've found that in numscons-0.9.4, the detection of the correct linker assumes that if gcc is in use, the linker is gnu ld. However, on solaris this isn't the recommended toolchain, so it's typical to build gcc with gnu as and the solaris /usr/ccs/bin/ld under the hood. What this means is that when setting a run_path in the binary (which we need to do) the linker flags are set to "-Wl,-rpath=<library>". However, this isn't valid for the solaris ld. It needs -R<libname>, or -Wl,-R<libname>. I'm pretty sure that on Solaris trying to link a library with -Wl,-rpath= and looking for an error should be enough to determine the correct format for the linker.
Scons and hence numscons indeed assume that the linker is the same as the compiler by default. It would be possible to avoid this by detecting the linker at runtime, to bypass scons tools choice, like I do for C, C++ and Fortran compilers. The whole scons tools sub-system is unfortunately very limited ATM, so there is a lot of manual work to do (that's actually what most of the code in numscons/core is for).
(3) Numscons tries to check for the need for a MAIN__ function when linking with gfortran. However, any libraries built with numscons come out with an unsatisfied dependency on MAIN__. The log looks like this in build/scons/numpy/linalg/config.log looks like this:
It may be linked to the sun linker problem above. Actually, the dummy main detection is not used at all for the building - it is necessary to detect name mangling used by the fortran compiler, but that's it. I assumed that a dummy main was never needed for shared libraries, but that assumption may well be ill founded.
I never had problems related to this on open solaris, with both native and gcc toolchains, so I am willing to investiage first whether it is linked to the sun linker problem or not.
Unfortunately, I won't have the time to work on this in the next few months because of my PhD thesis; the sun linker problem can be fixed by following a strategy similar to compilers, in numscons/core/initialization.py. You first need to add a detection scheme for the linker in compiler_detection.py.
Thanks, I'll look into this. It is true that working with opensolaris is a lot easier. Sun should have done it years ago. Thanks again, -Peter
participants (3)
-
Charles R Harris -
David Cournapeau -
Peter Norton