Switching off optimization for python setup.py config
Dear pythoneers, Recently I ran into a bug in Numerical Python's eigenvalues function, causing it to hang on Cygwin. The bug is caused by Numerical Python being compiled with optimization, while a number of routines (the dlamc* functions in dlapack_lite.c) should be compiled without optimization to function properly (as described in the lapack documentation). In reading the distutils archives, it seems that the issue of changing optimization levels and other compiler flags has been raised before. From what I understand, the current version of distutils does not allow to change the optimization level. The problem seems to be that some compiler arguments are required for an extension module to work with Python, and cannot just be changed (for example, my Makefile has OPT=-DNDEBUG -g -O3 -Wall -Wstrict-prototypes in it, which is more than just optimization flags). Now what the dlamc* functions are really doing is to find out some machine parameters before the actual numerical routines are executed. In today's world of autoconf, automake, and python setup.py config, I would think that it makes more sense to let distutils' config step figure out the machine parameters, and hardwire the results into the source code with some #define's. This can be achieved by letting python setup.py config run the dlamc* routines. However, for this we need to switch off optimization for the trial compilations and runs done by python setup.py config. Is this possible with the current version of distutils? If not, would it be worthwhile to add this to distutils? We have more freedom in changing compiler switches for trial compilations than for the actual extension module, and I would think that this would be useful for cases other than Numerical Python. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Michiel Jan Laurens de Hoon wrote:
Dear pythoneers,
Recently I ran into a bug in Numerical Python's eigenvalues function, causing it to hang on Cygwin. The bug is caused by Numerical Python being compiled with optimization, while a number of routines (the dlamc* functions in dlapack_lite.c) should be compiled without optimization to function properly (as described in the lapack documentation).
In reading the distutils archives, it seems that the issue of changing optimization levels and other compiler flags has been raised before. From what I understand, the current version of distutils does not allow to change the optimization level. The problem seems to be that some compiler arguments are required for an extension module to work with Python, and cannot just be changed (for example, my Makefile has OPT=-DNDEBUG -g -O3 -Wall -Wstrict-prototypes in it, which is more than just optimization flags).
Now what the dlamc* functions are really doing is to find out some machine parameters before the actual numerical routines are executed. In today's world of autoconf, automake, and python setup.py config, I would think that it makes more sense to let distutils' config step figure out the machine parameters, and hardwire the results into the source code with some #define's.
This can be achieved by letting python setup.py config run the dlamc* routines. However, for this we need to switch off optimization for the trial compilations and runs done by python setup.py config. Is this possible with the current version of distutils? If not, would it be worthwhile to add this to distutils? We have more freedom in changing compiler switches for trial compilations than for the actual extension module, and I would think that this would be useful for cases other than Numerical Python.
You can use the distutils.command.config command class to write configuration and test compilation methods. These have complete control over the compiler being used, so switching off optimization for the purpose of configuration should be possible. The config command is still way from being complete and not all that useful by itself, but it provides a nice hook to integrate and add your own configuration tools. For an example on what can be done, see mxSetup.py in the eGenix mx Extensions tar-balls. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 01 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
EuroPython 2003, Charleroi, Belgium: 54 days left
On donderdag, mei 1, 2003, at 15:03 Europe/Amsterdam, M.-A. Lemburg wrote:
You can use the distutils.command.config command class to write configuration and test compilation methods. These have complete control over the compiler being used, so switching off optimization for the purpose of configuration should be possible.
Does this work on non-unix platforms? MacOS9 is on the way out (it definitely won't work there), but I wouldn't be surprised if stuff like this also won't work on Windows or BeOS or whatever. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
Jack Jansen wrote:
On donderdag, mei 1, 2003, at 15:03 Europe/Amsterdam, M.-A. Lemburg wrote:
You can use the distutils.command.config command class to write configuration and test compilation methods. These have complete control over the compiler being used, so switching off optimization for the purpose of configuration should be possible.
Does this work on non-unix platforms? MacOS9 is on the way out (it definitely won't work there), but I wouldn't be surprised if stuff like this also won't work on Windows or BeOS or whatever.
If you mean the config command, yes, this also works on Windows. I use it to check for C APIs and to find locations of C libs and header files. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, May 01 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
EuroPython 2003, Charleroi, Belgium: 54 days left
On Thu, 1 May 2003, M.-A. Lemburg wrote:
Jack Jansen wrote:
On donderdag, mei 1, 2003, at 15:03 Europe/Amsterdam, M.-A. Lemburg wrote:
You can use the distutils.command.config command class to write configuration and test compilation methods. These have complete control over the compiler being used, so switching off optimization for the purpose of configuration should be possible.
Does this work on non-unix platforms? MacOS9 is on the way out (it definitely won't work there), but I wouldn't be surprised if stuff like this also won't work on Windows or BeOS or whatever.
If you mean the config command, yes, this also works on Windows. I use it to check for C APIs and to find locations of C libs and header files.
Hmm. Then would this also work for making a setup.py that was capable of cross-compiling to other targets? I've long been thinking the combination of Distutils, the catalog and a cross-compiler setup on the server would be able to deliver native binary packages from a source repository. Take that, CPAN! mwa -- Mark W. Alexander slash@dotnetslash.net
participants (4)
-
Jack Jansen -
M.-A. Lemburg -
Mark W. Alexander -
Michiel Jan Laurens de Hoon