Is numpy ignoring CFLAGS?
On some 64-bit platforms, which include, but is not limited to: * Some version of OS X (I don't know what versions or processors) * Solaris on SPARC processors. * Solaris on x86 processors. * OpenSolaris on SPARC processors. * OpenSolaris on x86 processors. * HP-UX on PA-RISC processors. the default is to build 32-bit objects, but 64-bit objects can be created if needed. This is usually done via adding the -m64 flag when compiling with GCC or SunStudio, though the flag will be different with HP's compiler. Numpy is used as part of Sage, but it would appear that adding -m64 to CFLAGS will not work. A comment in the script used to build numpy shows: # numpy's distutils is buggy and runs a conftest without # taking CFLAGS into account. With 64 bit OSX this results # in *boom* it then goes on to copy a file called gcc_fake, which is basically a script which gets renamed to gcc, but includes the -m64 flag. We are using numpy-1.3.0. Is this a known bug? If not, can I submit it to a bug database? Better still, does anyone have a patch to resolve it - I hate the idea of making I've now changed the build method in Sage so it does not only work on OS X, and does not hard-code the path to gcc. I have: if [ "x$SAGE64" = xyes ]; then echo "Building a 64-bit version of numpy" # HACK ALERT # HACK ALERT # HACK ALERT echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "Creating a sort of fake gcc, which has the -m64 flag" echo "#!/usr/bin/env bash" > $SAGE_LOCAL/bin/gcc echo "`command -v gcc` -m64 \$@" >> $SAGE_LOCAL/bin/gcc chmod 755 $SAGE_LOCAL/bin/gcc fi This this file $SAGE_LOCAL/bin/gcc is used to build just numpy. After that, we can remove it if [ "x$SAGE64" = xyes ]; then echo "deleting fake gcc" rm $SAGE_LOCAL/bin/gcc fi and let the normal gcc program be used with an appropriate setting of CFLAGS to make other packages build 64-bit. I know this issue has been known in Sage for a long time (years) but I don't know if it was ever reported to the Numpy mailing list or added to any bug database Dave
Dr. David Kirkby wrote:
On some 64-bit platforms, which include, but is not limited to:
* Some version of OS X (I don't know what versions or processors) * Solaris on SPARC processors. * Solaris on x86 processors. * OpenSolaris on SPARC processors. * OpenSolaris on x86 processors. * HP-UX on PA-RISC processors.
the default is to build 32-bit objects, but 64-bit objects can be created if needed. This is usually done via adding the -m64 flag when compiling with GCC or SunStudio, though the flag will be different with HP's compiler.
Numpy is used as part of Sage, but it would appear that adding -m64 to CFLAGS will not work. A comment in the script used to build numpy shows:
First: Is Python built using -m64? If not, is there a reason that NumPy in 64 bit and load it into 32 bit Python work? If Python is built with -m64 I'd expect NumPy to pick it up automatically as it queries Python for the build flags to use...
# numpy's distutils is buggy and runs a conftest without # taking CFLAGS into account. With 64 bit OSX this results # in *boom*
it then goes on to copy a file called gcc_fake, which is basically a script which gets renamed to gcc, but includes the -m64 flag.
We are using numpy-1.3.0.
Is this a known bug? If not, can I submit it to a bug database? Better still, does anyone have a patch to resolve it - I hate the idea of making
Until somebody who really knows an answer chimes in; AFAIK this is a "feature" in distutils itself, so it affects most Python software. (Setting CFLAGS overwrites the necesarry CFLAGS settings, like -fPIC and -fno-strict-aliasing, that is queried from Python). Try setting "OPT" instead? Dag Sverre
On 06/28/10 09:38 AM, Dag Sverre Seljebotn wrote:
Dr. David Kirkby wrote:
On some 64-bit platforms, which include, but is not limited to:
* Some version of OS X (I don't know what versions or processors) * Solaris on SPARC processors. * Solaris on x86 processors. * OpenSolaris on SPARC processors. * OpenSolaris on x86 processors. * HP-UX on PA-RISC processors.
the default is to build 32-bit objects, but 64-bit objects can be created if needed. This is usually done via adding the -m64 flag when compiling with GCC or SunStudio, though the flag will be different with HP's compiler.
Numpy is used as part of Sage, but it would appear that adding -m64 to CFLAGS will not work. A comment in the script used to build numpy shows:
First: Is Python built using -m64? If not, is there a reason that NumPy in 64 bit and load it into 32 bit Python work? If Python is built with -m64 I'd expect NumPy to pick it up automatically as it queries Python for the build flags to use...
Yes, Python is built 64-bit, using the -m64 option.
# numpy's distutils is buggy and runs a conftest without # taking CFLAGS into account. With 64 bit OSX this results # in *boom*
it then goes on to copy a file called gcc_fake, which is basically a script which gets renamed to gcc, but includes the -m64 flag.
We are using numpy-1.3.0.
Is this a known bug? If not, can I submit it to a bug database? Better still, does anyone have a patch to resolve it - I hate the idea of making
Until somebody who really knows an answer chimes in;
AFAIK this is a "feature" in distutils itself, so it affects most Python software. (Setting CFLAGS overwrites the necesarry CFLAGS settings, like -fPIC and -fno-strict-aliasing, that is queried from Python). Try setting "OPT" instead?
Dag Sverre
OPT has -m64 in it. This is the bit that shows how Python is built on Solaris (uname=SunOS). SAGE64 will be set to "yes" for a 64-bit build. OPT="-g -O3 -m64 -Wall -Wstrict-prototypes"; export OPT ./configure $EXTRAFLAGS --prefix="$SAGE_LOCAL" \ --enable-unicode=ucs4 --with-gcc="gcc -m64" Many other parts of Sage seem to inherit the flags ok from Python, but not numpy. It is not a Solaris specific issue, as the same issue results on OS X. Dave Dave
On Mon, Jun 28, 2010 at 6:56 PM, Dr. David Kirkby <david.kirkby@onetel.net> wrote:
On 06/28/10 09:38 AM, Dag Sverre Seljebotn wrote:
Dr. David Kirkby wrote:
On some 64-bit platforms, which include, but is not limited to:
* Some version of OS X (I don't know what versions or processors) * Solaris on SPARC processors. * Solaris on x86 processors. * OpenSolaris on SPARC processors. * OpenSolaris on x86 processors. * HP-UX on PA-RISC processors.
the default is to build 32-bit objects, but 64-bit objects can be created if needed. This is usually done via adding the -m64 flag when compiling with GCC or SunStudio, though the flag will be different with HP's compiler.
Numpy is used as part of Sage, but it would appear that adding -m64 to CFLAGS will not work. A comment in the script used to build numpy shows:
First: Is Python built using -m64? If not, is there a reason that NumPy in 64 bit and load it into 32 bit Python work? If Python is built with -m64 I'd expect NumPy to pick it up automatically as it queries Python for the build flags to use...
Yes, Python is built 64-bit, using the -m64 option.
# numpy's distutils is buggy and runs a conftest without # taking CFLAGS into account. With 64 bit OSX this results # in *boom*
it then goes on to copy a file called gcc_fake, which is basically a script which gets renamed to gcc, but includes the -m64 flag.
We are using numpy-1.3.0.
Is this a known bug? If not, can I submit it to a bug database? Better still, does anyone have a patch to resolve it - I hate the idea of making
Until somebody who really knows an answer chimes in;
AFAIK this is a "feature" in distutils itself, so it affects most Python software. (Setting CFLAGS overwrites the necesarry CFLAGS settings, like -fPIC and -fno-strict-aliasing, that is queried from Python). Try setting "OPT" instead?
Dag Sverre
OPT has -m64 in it.
This is the bit that shows how Python is built on Solaris (uname=SunOS). SAGE64 will be set to "yes" for a 64-bit build.
OPT="-g -O3 -m64 -Wall -Wstrict-prototypes"; export OPT ./configure $EXTRAFLAGS --prefix="$SAGE_LOCAL" \ --enable-unicode=ucs4 --with-gcc="gcc -m64"
Many other parts of Sage seem to inherit the flags ok from Python, but not numpy.
Are you saying that OPT is not taken into account ? It seems to work for me, e.g. OPT="-m64" python setup.py build_ext does put -m64 somewhere in CFLAGS. When using numpy.distutils, CFLAGS should never be overriden unless you are ready to set up the whole set of options manually. By default, CFLAGS is the concatenation of BASECFLAGS, OPT and CCSHARED (in that order), and only OPT should be tweaked in general. David
On 06/28/10 11:28 AM, David Cournapeau wrote:
On Mon, Jun 28, 2010 at 6:56 PM, Dr. David Kirkby
Many other parts of Sage seem to inherit the flags ok from Python, but not numpy.
Are you saying that OPT is not taken into account ? It seems to work for me, e.g.
OPT="-m64" python setup.py build_ext
does put -m64 somewhere in CFLAGS. When using numpy.distutils, CFLAGS should never be overriden unless you are ready to set up the whole set of options manually. By default, CFLAGS is the concatenation of BASECFLAGS, OPT and CCSHARED (in that order), and only OPT should be tweaked in general.
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
OPT is not being totally ignored, but some part of numpy is trying to build without taking -m64 into account. Note in the output below that as numpy is compiled, -m64 is shown on some lines, which is what I expect. But note also the messages about "wrong ELF class: ELFCLASS64" from the linker, indicating to me the linker is not expecting to find 64-bit objects. All the libraries in the directory /export/home/drkirkby/sage-4.5.alpha0/local/lib are 64-bit. Someone worked around this on OS X by creating a script called 'gcc_fake', with this content #!/bin/bash /usr/bin/gcc -m64 $@ If a 64-bit build was going to be done, this was remaned to 'gcc' and put in the path first before building numpy. Once numpy was built, the script can be deleted. All the script does is basically invoke gcc with the -m64 option, which is making me think that -m64 is being missed somewhere. I just deleted this gcc_fake, as it had a hard-coded path. I then created it dynamically, so the path of the real gcc does not need to be /usr/bin/gcc. Doing that, I can build numpy 64-bit on OpenSolaris. But of course this is a hack, and I'd rather avoid the hack if possible. BTW, if it would help, I could create you an account on this machine and test it yourself. I'm not trying to get out of doing the work, but the offer is there. Dave numpy-1.3.0.p3/spkg-install numpy-1.3.0.p3/.hgignore Finished extraction **************************************************** Host system uname -a: SunOS hawk 5.11 snv_134 i86pc i386 i86pc **************************************************** **************************************************** CC Version gcc -v Using built-in specs. Target: i386-pc-solaris2.11 Configured with: /export/home/drkirkby/gcc-4.4.4/configure --prefix=/usr/local/gcc-4.4.4-multilib --enable-languages=c,c++,fortran --with-gmp=/usr/local/gcc-4.4.4-multilib --with-mpfr=/usr/local/gcc-4.4.4-multilib --disable-nls --enable-checking=release --enable-werror=no --enable-multilib --with-system-zlib --enable-bootstrap --with-gnu-as --with-as=/usr/local/binutils-2.20/bin/as --without-gnu-ld --with-ld=/usr/ccs/bin/ld Thread model: posix gcc version 4.4.4 (GCC) **************************************************** Running from numpy source directory. F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib NOT AVAILABLE atlas_blas_info: FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/lib'] language = c include_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/include'] /export/home/drkirkby/sage-4.5.alpha0/spkg/build/numpy-1.3.0.p3/src/numpy/distutils/command/config.py:361: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of get_output is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize Sage_FCompiler_1 customize Sage_FCompiler_1 customize Sage_FCompiler_1 using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -O3 -m64 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc _configtest.o -L/export/home/drkirkby/sage-4.5.alpha0/local/lib -lf77blas -lcblas -latlas -o _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to _configtest collect2: ld returned 1 exit status ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to _configtest collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/lib'] language = c define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/include'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib libraries lapack_atlas not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries lapack_atlas not found in /export/home/drkirkby/sage-4.5.alpha0/local/lib numpy.distutils.system_info.atlas_info FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/lib'] language = f77 include_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/include'] customize Sage_FCompiler_1 customize Sage_FCompiler_1 customize Sage_FCompiler_1 using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -O3 -m64 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc _configtest.o -L/export/home/drkirkby/sage-4.5.alpha0/local/lib -llapack -lf77blas -lcblas -latlas -o _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to _configtest collect2: ld returned 1 exit status ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to _configtest collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/export/home/drkirkby/sage-4.5.alpha0/local/include'] running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources creating build creating build/src.solaris-2.11-i86pc-2.6 creating build/src.solaris-2.11-i86pc-2.6/numpy creating build/src.solaris-2.11-i86pc-2.6/numpy/distutils building library "npymath" sources creating build/src.solaris-2.11-i86pc-2.6/numpy/core creating build/src.solaris-2.11-i86pc-2.6/numpy/core/src conv_template:> build/src.solaris-2.11-i86pc-2.6/numpy/core/src/npy_math.c building extension "numpy.core._sort" sources Generating build/src.solaris-2.11-i86pc-2.6/numpy/core/include/numpy/config.h customize Sage_FCompiler_1 customize Sage_FCompiler_1 customize Sage_FCompiler_1 using config C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -O3 -m64 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/export/home/drkirkby/sage-4.5.alpha0/local/include/python2.6 -c' gcc: _configtest.c success! removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -O3 -m64 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/export/home/drkirkby/sage-4.5.alpha0/local/include/python2.6 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -O3 -m64 -Wall -Wstrict-prototypes -fPIC
On Mon, Jun 28, 2010 at 9:28 PM, Dr. David Kirkby <david.kirkby@onetel.net> wrote:
On 06/28/10 11:28 AM, David Cournapeau wrote:
On Mon, Jun 28, 2010 at 6:56 PM, Dr. David Kirkby
Many other parts of Sage seem to inherit the flags ok from Python, but not numpy.
Are you saying that OPT is not taken into account ? It seems to work for me, e.g.
OPT="-m64" python setup.py build_ext
does put -m64 somewhere in CFLAGS. When using numpy.distutils, CFLAGS should never be overriden unless you are ready to set up the whole set of options manually. By default, CFLAGS is the concatenation of BASECFLAGS, OPT and CCSHARED (in that order), and only OPT should be tweaked in general.
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
OPT is not being totally ignored, but some part of numpy is trying to build without taking -m64 into account.
Note in the output below that as numpy is compiled, -m64 is shown on some lines, which is what I expect. But note also the messages about "wrong ELF class: ELFCLASS64" from the linker, indicating to me the linker is not expecting to find 64-bit objects.
Ah, that makes the issue much clearer: the linker is not passed the -m64 option. It works with distutils because CFLAGS is appended to LDFLAGS if CFLAGS is in os.environ,but we use CFLAGS differently. I am not sure how to fix that issue... David
On 06/28/10 02:54 PM, David Cournapeau wrote:
On Mon, Jun 28, 2010 at 9:28 PM, Dr. David Kirkby <david.kirkby@onetel.net> wrote:
On 06/28/10 11:28 AM, David Cournapeau wrote:
On Mon, Jun 28, 2010 at 6:56 PM, Dr. David Kirkby
Many other parts of Sage seem to inherit the flags ok from Python, but not numpy.
Are you saying that OPT is not taken into account ? It seems to work for me, e.g.
OPT="-m64" python setup.py build_ext
does put -m64 somewhere in CFLAGS. When using numpy.distutils, CFLAGS should never be overriden unless you are ready to set up the whole set of options manually. By default, CFLAGS is the concatenation of BASECFLAGS, OPT and CCSHARED (in that order), and only OPT should be tweaked in general.
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
OPT is not being totally ignored, but some part of numpy is trying to build without taking -m64 into account.
Note in the output below that as numpy is compiled, -m64 is shown on some lines, which is what I expect. But note also the messages about "wrong ELF class: ELFCLASS64" from the linker, indicating to me the linker is not expecting to find 64-bit objects.
Ah, that makes the issue much clearer: the linker is not passed the -m64 option. It works with distutils because CFLAGS is appended to LDFLAGS if CFLAGS is in os.environ,but we use CFLAGS differently.
I am not sure how to fix that issue...
David
I've tried adding -m64 to LDFLAGS before exporting that. That does not help. Also, the Sun linker has this option -64 (note, no letter 'm'). drkirkby@hawk:~$ man ld Reformatting page. Please Wait... done User Commands ld(1) NAME ld - link-editor for object files SYNOPSIS ld [-32 | -64] [-a | -r] [-b] [-Bdirect | nodirect] [-B dynamic | static] [-B eliminate] [-B group] [-B local] [-B reduce] [-B symbolic] [-c name] [-C] [-d y | n] [-D token,...] [-e epsym] [-f name | -F name] [-G] [-h name] [-i] [-I name] [-l x] [-L path] [-m] [-M mapfile] [-N string] [-o outfile] [-p auditlib] [-P auditlib] [-Q y | n] [-R path] [-s] [-S supportlib] [-t] [-u symname] [-V] [-Y P,dirlist] [-z absexec] [-z allextract | defaultextract | weakextract ] [-z altexec64] [-z combreloc | nocombreloc ] [-z defs | nodefs] [-z direct | nodirect] [-z endfiltee] [-z finiarray=function] [-z globalaudit] [-z groupperm | nogroupperm] [-z help ] [-z ignore | record] [-z initarray=function] [-z initfirst] [-z interpose] [-z lazyload | nolazyload] [-z ld32=arg1,arg2,...] [-z ld64=arg1,arg2,...] [-z loadfltr] [-z muldefs] [-z nocompstrtab] [-z nodefaultlib] [-z nodelete] [-z nodlopen] [-z nodump] [-z noldynsym] [-z nopartial] [-z noversion] [-z now] [-z origin] [-z preinitarray=function] [-z redlocsym] [-z relaxreloc] [-z rescan-now] [-z recan] [-z rescan-start ... -z rescan-end]] [-z target=sparc|x86] [-z text | textwarn | textoff] [-z verbose] [-z wrap=symbol] filename... DESCRIPTION The link-editor, ld, combines relocatable object files by resolving symbol references to symbol definitions, together <snip> OPTIONS The following options are supported. -32 | -64 Creates a 32-bit, or 64-bit object. By default, the class of the object being generated is determined from the first ELF object processed from the command line. If no objects are specified, the class is determined by the first object encountered within the first archive processed from the command line. If there are no objects or archives, the link-editor creates a 32-bit object. The -64 option is required to create a 64-bit object solely from a mapfile. The -32 or -64 options can also be used in the rare case of linking entirely from an archive that contains a mix- ture of 32 and 64-bit objects. If the first object in the archive is not the class of the object that is required to be created, then the -32 or -64 option can be used to direct the link-editor. SunOS 5.11 Last change: 3 Dec 2009 3 User Commands ld(1) I tried adding that to LDFLAGS. Again it did not help. The *only* solution I can find to date is to create a script which invokes gcc with the -m64 option, but that is a horrible hack. Dave
participants (3)
-
Dag Sverre Seljebotn
-
David Cournapeau
-
Dr. David Kirkby