I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. This causes links to fail since the same symbol shows up twice. David, any ideas? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Feb 11, 2008 2:21 AM, Robert Kern
I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice.
Spoke too soon. It fails with r4772, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Feb 11, 2008 5:38 PM, Robert Kern
On Feb 11, 2008 2:21 AM, Robert Kern
wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice.
Spoke too soon. It fails with r4772, too.
Does it mean that it was already broken before numscons merge or not ? Nobody should be changed when numpy is built using setup.py, so any problem here is something I should fix ASAP. Supporting setuptools with numscons (using setupscons.py) is not a priority for me, though. David
On Feb 11, 2008 6:49 PM, David Cournapeau
On Feb 11, 2008 5:38 PM, Robert Kern
wrote: On Feb 11, 2008 2:21 AM, Robert Kern
wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice.
Spoke too soon. It fails with r4772, too.
Does it mean that it was already broken before numscons merge or not ? Nobody should be changed when numpy is built using setup.py, so any ^^^^^^ I meant something, not nobody, here, obviously.
David
On Feb 11, 2008 3:49 AM, David Cournapeau
On Feb 11, 2008 5:38 PM, Robert Kern
wrote: On Feb 11, 2008 2:21 AM, Robert Kern
wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice.
Spoke too soon. It fails with r4772, too.
Does it mean that it was already broken before numscons merge or not ?
Yes. I think I figured out the problem. "python setup.py develop" works fine. However, I ran into the problem when I added "develop" to the end of a command line that already had "build". So essentially, "build_src" gets run twice. I'm not sure there is anything that can be done about that given distutils' handling of options. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Feb 11, 2008 1:21 AM, Robert Kern
I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. This causes links to fail since the same symbol shows up twice.
While we're talking build, how do I set the compiler flags? Numpy here always compiles with -march=i386, which seems a bit conservative. My environment flags are also ignored, but I assume there is someway of getting the compile to behave. Chuck
On Feb 11, 2008 5:40 PM, Charles R Harris
On Feb 11, 2008 1:21 AM, Robert Kern
wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. This causes links to fail since the same symbol shows up twice.
While we're talking build, how do I set the compiler flags? Numpy here always compiles with -march=i386, which seems a bit conservative. My environment flags are also ignored, but I assume there is someway of getting the compile to behave.
Well, you assumed wrong :) It is one of the reasons why I started working on scons support. It is extremely ackward to control compilation flags with the python distutils from stdlib, and by design (flags are added at several different locations, which depend on the platform, and understanding the exact logic is beyond mere mortal capability). You can use CFLAGS with distutils, but I don't really understand what's happening in this case: it seems that they are appended to the current compilation flags, but the order also seems to depend on the architecture. Sometimes they are inserted toward the end of compilation flags. With numscons, CFLAGS override the optimization/warning flags (but not -fPIC, etc... which are necessary to build the code; internally, optimization/debug/warning/necessary flags are separated). One example: with numpy.distutils, compiling for debug is not trivial: CFLAGS="-DDEBUG" python setup.py build will still compile with -O2, and with -DNDEBUG. You need to add -O0 to override the default O2 used by distutils (at least for gcc, when several options of the same kind are given, the last one wins), and you have problems for options which cannot be overriden by later options. With numscons: CFLAGS="-DDEBUG" will compile with only this option (and -fPIC, etc... on linux). I intend to add more options on the UI, but anway, they will be all overridable (for example, only overriding warning flags, etc...). cheers, David
On Feb 11, 2008 7:10 PM, David Cournapeau
On Feb 11, 2008 5:40 PM, Charles R Harris
wrote: On Feb 11, 2008 1:21 AM, Robert Kern
wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools "develop" command. In order to make sure that setuptools' "develop" works with numpy.distutils' "build_src", we override the "develop" command to reinitialize the "build_src" command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. This causes links to fail since the same symbol shows up twice.
While we're talking build, how do I set the compiler flags? Numpy here always compiles with -march=i386, which seems a bit conservative. My environment flags are also ignored, but I assume there is someway of
getting
the compile to behave. Well, you assumed wrong :)
For some reason I was hoping you would pipe up :0) Yeah, that in itself is a good reason for trying something like scons. I note that with -O2 -finline-functions, or -O3, I can knock almost 30% off the string sort times. That's a lot better than I can do fooling around with the code. It is one of the reasons why I started
working on scons support. It is extremely ackward to control compilation flags with the python distutils from stdlib, and by design (flags are added at several different locations, which depend on the platform, and understanding the exact logic is beyond mere mortal capability).
Thanks Chuck
On Mon, 2008-02-11 at 20:42 -0700, Charles R Harris wrote:
On Feb 11, 2008 7:10 PM, David Cournapeau
wrote: On Feb 11, 2008 5:40 PM, Charles R Harris
wrote: > > > > On Feb 11, 2008 1:21 AM, Robert Kern wrote: > > > I've just updated the SVN trunk to get the latest numscons merge. > > Something broke the support I put in for the setuptools "develop" > > command. In order to make sure that setuptools' "develop" works with > > numpy.distutils' "build_src", we override the "develop" command to > > reinitialize the "build_src" command to add the --inplace option. This > > used to work as of r4772, but now any Fortran Extensions have the > > generated sources added twice. This causes links to fail since the > > same symbol shows up twice. > > > > While we're talking build, how do I set the compiler flags? Numpy here > always compiles with -march=i386, which seems a bit conservative. My > environment flags are also ignored, but I assume there is someway of getting > the compile to behave. Well, you assumed wrong :)
For some reason I was hoping you would pipe up :0) Yeah, that in itself is a good reason for trying something like scons.
To be more exact: you can simply add flags in distutils. It is just that there is not much logic to handle different cases (e.g. having different set of warnings for pyrex vs swig vs normal C extensions is difficult).
I note that with -O2 -finline-functions, or -O3, I can knock almost 30% off the string sort times. That's a lot better than I can do fooling around with the code.
You should be able to do it just with numpy.distutils: CFLAGS="-O2 -finline-functions" setup.py build What should be easier with numscons at some point is a fine grained control. For example, the basic configuration is set in files similar to site.cfg right now, and I intend to add the possibility to set your own file in a near future (you can look for the file numscons/core/compiler.cfg to see how I do things now, in numscons sources). If that's the kind of things you are interested in, do not hesitate to try thing with numscons and tell me what's missing/what could be improved. I am mostly focused on getting things working by default on all platforms right now, but I am interested by inputs from other people with different needs. cheers, David
On Feb 11, 2008 9:24 PM, David Cournapeau
On Mon, 2008-02-11 at 20:42 -0700, Charles R Harris wrote:
On Feb 11, 2008 7:10 PM, David Cournapeau
wrote: On Feb 11, 2008 5:40 PM, Charles R Harris
wrote: > > > > On Feb 11, 2008 1:21 AM, Robert Kern wrote: > > > I've just updated the SVN trunk to get the latest numscons merge. > > Something broke the support I put in for the setuptools "develop" > > command. In order to make sure that setuptools' "develop" works with > > numpy.distutils' "build_src", we override the "develop" command to > > reinitialize the "build_src" command to add the --inplace option. This > > used to work as of r4772, but now any Fortran Extensions have the > > generated sources added twice. This causes links to fail since the > > same symbol shows up twice. > > > > While we're talking build, how do I set the compiler flags? Numpy here > always compiles with -march=i386, which seems a bit conservative. My > environment flags are also ignored, but I assume there is someway of getting > the compile to behave. Well, you assumed wrong :)
For some reason I was hoping you would pipe up :0) Yeah, that in itself is a good reason for trying something like scons.
To be more exact: you can simply add flags in distutils. It is just that there is not much logic to handle different cases (e.g. having different set of warnings for pyrex vs swig vs normal C extensions is difficult).
I note that with -O2 -finline-functions, or -O3, I can knock almost 30% off the string sort times. That's a lot better than I can do fooling around with the code.
You should be able to do it just with numpy.distutils:
CFLAGS="-O2 -finline-functions" setup.py build
Curiously, CFLAGS="-O3 -finline-functions" causes the -fno-strict-aliasing flag to disappear when the random module is compiled, resulting in a lot of warnings and, in my experience, probably buggy code generation. So a safer bet is CFLAGS="-O3 -finline-functions -fno-strict-aliasing" . Chuck
On Feb 13, 2008 11:27 PM, Charles R Harris
Curiously, CFLAGS="-O3 -finline-functions" causes the -fno-strict-aliasing flag to disappear when the random module is compiled, resulting in a lot of warnings and, in my experience, probably buggy code generation. So a safer bet is CFLAGS="-O3 -finline-functions -fno-strict-aliasing" .
That's the kind of weirdness which does not happen (or at least is not support to happen) with numscons :) cheers, David
participants (4)
-
Charles R Harris
-
David Cournapeau
-
David Cournapeau
-
Robert Kern