problems with numpy.setuptools("single_version_externally_managed")
hi all, If I use in setup.py: import setuptools from numpy.distutils.system_info import system_info, NotFoundError, dict_append, so_ext I successfully install (OpenOpt) but Nils and I get the warning (with latest numpy) ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! distutils has been imported before numpy.distutils and now numpy.distutils cannot apply all of its customizations to distutils effectively. To avoid this warning, make sure that numpy.distutils is imported *before* distutils. ******************************************************** (BTW note that distutils is not setuptools) If I use vise versa, as it is recommended: from numpy.distutils.system_info import system_info, NotFoundError, dict_append, so_ext import setuptools Then I get: $ sudo python setup.py install running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_py running install_lib Traceback (most recent call last): File "setup.py", line 93, in <module> 'Topic :: Scientific/Engineering'] File "/usr/lib/python2.5/site-packages/numpy/distutils/core.py", line 176, in setup return old_setup(**new_attr) File "distutils/core.py", line 151, in setup File "distutils/dist.py", line 974, in run_commands File "distutils/dist.py", line 994, in run_command File "/usr/lib/python2.5/site-packages/numpy/distutils/command/install.py", line 16, in run r = old_install.run(self) File "distutils/command/install.py", line 510, in run File "/usr/lib/python2.5/cmd.py", line 333, in run_command del help[cmd] File "distutils/dist.py", line 994, in run_command File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 21, in run outfiles = self.install() File "distutils/command/install_lib.py", line 116, in install File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 47, in copy_tree exclude = self.get_exclusions() File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 30, in get_exclusions if (nsp and self.get_finalized_command('install') File "/usr/lib/python2.5/cmd.py", line 112, in __getattr__ import readline AttributeError: single_version_externally_managed Do you know what does it means and how it could be fixed? Best regards, D.
dmitrey wrote:
hi all,
If I use in setup.py:
import setuptools from numpy.distutils.system_info import system_info, NotFoundError, dict_append, so_ext
I successfully install (OpenOpt) but Nils and I get the warning (with latest numpy) ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!
distutils has been imported before numpy.distutils and now numpy.distutils cannot apply all of its customizations to distutils effectively.
To avoid this warning, make sure that numpy.distutils is imported *before* distutils. ******************************************************** (BTW note that distutils is not setuptools)
setuptools probably then imports distutils as the above warning is triggered using the following condition: 'distutils' in sys.modules
If I use vise versa, as it is recommended: from numpy.distutils.system_info import system_info, NotFoundError, dict_append, so_ext import setuptools Then I get:
$ sudo python setup.py install
running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_py running install_lib Traceback (most recent call last): File "setup.py", line 93, in <module> 'Topic :: Scientific/Engineering'] File "/usr/lib/python2.5/site-packages/numpy/distutils/core.py", line 176, in setup return old_setup(**new_attr) File "distutils/core.py", line 151, in setup File "distutils/dist.py", line 974, in run_commands File "distutils/dist.py", line 994, in run_command File "/usr/lib/python2.5/site-packages/numpy/distutils/command/install.py", line 16, in run r = old_install.run(self) File "distutils/command/install.py", line 510, in run File "/usr/lib/python2.5/cmd.py", line 333, in run_command del help[cmd] File "distutils/dist.py", line 994, in run_command File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 21, in run outfiles = self.install() File "distutils/command/install_lib.py", line 116, in install File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 47, in copy_tree exclude = self.get_exclusions() File "/usr/lib/python2.5/site-packages/setuptools/command/install_lib.py", line 30, in get_exclusions if (nsp and self.get_finalized_command('install') File "/usr/lib/python2.5/cmd.py", line 112, in __getattr__ import readline AttributeError: single_version_externally_managed
Do you know what does it means and how it could be fixed?
I don't know much of setuptools but may guess is that setuptools overwrites some distutils functions/methods like numpy.distutils does. Does setuptools say anything about the order of importing distutils and setuptools? If not then I would call this setuptools bug. On systems that do not use spaces in path names the order of importing distutils or numpy.distutils is not that important (but in some cases importing numpy.distutils first is important). However, for windows support the import order is crusial (numpy.distutils fixes the paths with spaces properly). We could make the above warning windows specific but only if there are fundamental reasons for not fixing setuptools. Pearu
dmitrey <dmitrey.kroshko@scipy.org> writes:
hi all,
If I use in setup.py:
import setuptools from numpy.distutils.system_info import system_info, NotFoundError, dict_append, so_ext
I successfully install (OpenOpt) but Nils and I get the warning (with latest numpy) ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!
distutils has been imported before numpy.distutils and now numpy.distutils cannot apply all of its customizations to distutils effectively.
To avoid this warning, make sure that numpy.distutils is imported *before* distutils. ******************************************************** (BTW note that distutils is not setuptools)
This is because of a recent change by Pearu. This is the right way to do it with setuptools. Pearu: I'm going to back your change out. For one thing, this is now broken: $ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.] I *think* (and I'll fix any bugs that don't correspond to this ;-) that importing numpy.distutils after distutils is ok, as long as you're not doing anything 'odd'. Basically, distutils isn't very robust, so it's quite hard to tell what's a bad way to mix things. But, we *do* explicitly support setuptools if it has been imported first. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
I successfully install (OpenOpt) but Nils and I get the warning (with latest numpy) ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!
distutils has been imported before numpy.distutils and now numpy.distutils cannot apply all of its customizations to distutils effectively.
To avoid this warning, make sure that numpy.distutils is imported *before* distutils. ******************************************************** (BTW note that distutils is not setuptools)
This is because of a recent change by Pearu. This is the right way to do it with setuptools.
Pearu: I'm going to back your change out. For one thing, this is now broken:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
I *think* (and I'll fix any bugs that don't correspond to this ;-) that importing numpy.distutils after distutils is ok, as long as you're not doing anything 'odd'.
I am just trying to ensure that numpy/scipy build on all platforms we can support. The questions is which one is more important: to support setuptools or to make the build work on more platforms? In the given case Windows with MS Visual compiler.
Basically, distutils isn't very robust, so it's quite hard to tell what's a bad way to mix things. But, we *do* explicitly support setuptools if it has been imported first.
Ok, then we have a problem. Details follow below. The problem with importing distutils before numpy.distutils is in redefining distutils.ccompiler.gen_lib_options function. This function needs to apply our quote_args to library_dirs and runtime_library_dirs because distutils _nt_quote_args has a bug of not checking if directory lists have already been quoted. So, if distutils is imported before numpy.distutils then distutils.ccompiler.gen_lib_options = <our gen_lib_options> is not effective as distutils/msvccompiler.py does from distutils.ccompiler import gen_lib_options so that the code in msvccompiler.py would still use the old distutils gen_lib_options function. Pearu
"Pearu Peterson" <pearu@cens.ioc.ee> writes:
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
When a supported method of building throws up a huge warning, I consider that broken ;-)
I *think* (and I'll fix any bugs that don't correspond to this ;-) that importing numpy.distutils after distutils is ok, as long as you're not doing anything 'odd'.
I am just trying to ensure that numpy/scipy build on all platforms we can support. The questions is which one is more important: to support setuptools or to make the build work on more platforms? In the given case Windows with MS Visual compiler.
Ok, then we have a problem. Details follow below.
So, if distutils is imported before numpy.distutils then
distutils.ccompiler.gen_lib_options = <our gen_lib_options>
is not effective as distutils/msvccompiler.py does
from distutils.ccompiler import gen_lib_options
so that the code in msvccompiler.py would still use the old distutils gen_lib_options function.
Fixed that for you. See r4134. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
On Tue, Sep 25, 2007 at 06:51:15PM +0300, Pearu Peterson wrote:
On Tue, September 25, 2007 6:43 pm, David M. Cooke wrote:
Fixed that for you. See r4134.
Right. It does not apply to msvccompiler.py though;)
Oops. Missed that :-) Even thought I tested it, although now I see that my testing was wrong. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
On Tue, September 25, 2007 6:57 pm, David M. Cooke wrote:
On Tue, Sep 25, 2007 at 06:51:15PM +0300, Pearu Peterson wrote:
On Tue, September 25, 2007 6:43 pm, David M. Cooke wrote:
Fixed that for you. See r4134.
Though, I am not sure if this is a proper fix. Importing, say, distutils.msvccompiler may trigger messages that may be completely irrelevant. E.g. in my linux box:
from distutils import log log.set_verbosity(4) import distutils.msvccompiler Warning: Can't read registry to find the necessary compiler setting Make sure that Python modules _winreg, win32api or win32con are installed.
I think distutils.*compiler modules are not designed to be imported unless specifically requested via --compiler flag. Though only importing distutils.msvccompiler has side effect. A seemingly easy fix would be to use the fix by checking os.name and sys.platform but if one is using mingw32 compiler under windows, the given warnings would be still wrong. But we'll see if this will be an issue in future.. Pearu
O
I think distutils.*compiler modules are not designed to be imported unless specifically requested via --compiler flag. Though only importing distutils.msvccompiler has side effect. My impression is that distutils.msvccompiler is quite different from all
Pearu Peterson wrote: the other ones, right ? It does not have the same attributes, etc... This fact is explicitely mentioned in distutils (msvccompiler.py): # Just set this so CCompiler's constructor doesn't barf. We currently # don't use the 'set_executables()' bureaucracy provided by CCompiler, # as it really isn't necessary for this sort of single-compiler class. # Would be nice to have a consistent interface with UnixCCompiler, # though, so it's worth thinking about.
A seemingly easy fix would be to use the fix by checking os.name and sys.platform but if one is using mingw32 compiler under windows, the given warnings would be still wrong.
Would it be possible to implement our own msvccompiler instead ? cheers, David
David Cournapeau wrote:
O
I think distutils.*compiler modules are not designed to be imported unless specifically requested via --compiler flag. Though only importing distutils.msvccompiler has side effect. My impression is that distutils.msvccompiler is quite different from all
Pearu Peterson wrote: the other ones, right ? It does not have the same attributes, etc... This fact is explicitely mentioned in distutils (msvccompiler.py):
# Just set this so CCompiler's constructor doesn't barf. We currently # don't use the 'set_executables()' bureaucracy provided by CCompiler, # as it really isn't necessary for this sort of single-compiler class. # Would be nice to have a consistent interface with UnixCCompiler, # though, so it's worth thinking about.
A seemingly easy fix would be to use the fix by checking os.name and sys.platform but if one is using mingw32 compiler under windows, the given warnings would be still wrong.
Would it be possible to implement our own msvccompiler instead ?
Actually, there exist a simple and proper fix (as usual;): numpy ccompiler.py should check if distutils.msvccompiler is imported (via sys.modules). If it is, then set distutils.msvccompiler.gen_lib_options = <our gen_lib_options> And if it is not, then do nothing. Note that when distutils.msvccompiler is imported then it will pick up the correct gen_lib_options from distutils.ccompiler which has been already enhanced by numpy.distutils. Pearu
Pearu Peterson wrote:
Actually, there exist a simple and proper fix (as usual;):
numpy ccompiler.py should check if distutils.msvccompiler is imported (via sys.modules). If it is, then set
distutils.msvccompiler.gen_lib_options = <our gen_lib_options>
But is this the only case where setuptools and numpy.distutils rewrite some classes ? I am just a bit afraid that modifying runtime objects both in numpy.distutils and setuptools will cause other problems in the near future (my impression is that it is safe to assume setuptools will become part of the standard library in python at some point, and we will have to cope with it). cheers, David
David Cournapeau wrote:
Pearu Peterson wrote:
Actually, there exist a simple and proper fix (as usual;):
numpy ccompiler.py should check if distutils.msvccompiler is imported (via sys.modules). If it is, then set
distutils.msvccompiler.gen_lib_options = <our gen_lib_options>
But is this the only case where setuptools and numpy.distutils rewrite some classes ?
No. gen_lib_options is a function. And numpy.distutils rewrites several methods in distutils classes. I don't know what setuptools does.
I am just a bit afraid that modifying runtime objects both in numpy.distutils and setuptools will cause other problems in the near future
There will be problems for sure and we are fixing them gradually as they appear - that is a normal process. Since we as stuck to distutils then we don't have alternatives to these kind of fixes.
(my impression is that it is safe to assume setuptools will become part of the standard library in python at some point, and we will have to cope with it).
Note that we are currently supporting python versions starting from 2.3. I don't know if setuptools is a part of python 2.6 already but even then we need to keep distutils support until we drop support for python 2.5. I bet this will not happen in the coming 5 years at least. Pearu
Pearu Peterson wrote:
I am just a bit afraid that modifying runtime objects both in numpy.distutils and setuptools will cause other problems in the near future
There will be problems for sure and we are fixing them gradually as they appear - that is a normal process. Since we as stuck to distutils then we don't have alternatives to these kind of fixes.
Isn't it possible to use a subclass instead of modifying the existing one ?
(my impression is that it is safe to assume setuptools will
become part of the standard library in python at some point, and we will have to cope with it).
Note that we are currently supporting python versions starting from 2.3. I don't know if setuptools is a part of python 2.6 already but even then we need to keep distutils support until we drop support for python 2.5. I bet this will not happen in the coming 5 years at least.
Sure, I've never implied to use setuptools only or to stop depending on distutils, just that it may be better to avoid modifying distutils on the fly when possible to avoid friction with setuptools as much as possible. cheers, David
David Cournapeau wrote:
Pearu Peterson wrote:
I am just a bit afraid that modifying runtime objects both in numpy.distutils and setuptools will cause other problems in the near future
There will be problems for sure and we are fixing them gradually as they appear - that is a normal process. Since we as stuck to distutils then we don't have alternatives to these kind of fixes.
Isn't it possible to use a subclass instead of modifying the existing one ?
It is possible. But in the given case it's not reasonable. Either we have few lines of code or we have lots of code replication that needs to be maintained.
(my impression is that it is safe to assume setuptools will
become part of the standard library in python at some point, and we will have to cope with it).
Note that we are currently supporting python versions starting from 2.3. I don't know if setuptools is a part of python 2.6 already but even then we need to keep distutils support until we drop support for python 2.5. I bet this will not happen in the coming 5 years at least.
Sure, I've never implied to use setuptools only or to stop depending on distutils, just that it may be better to avoid modifying distutils on the fly when possible to avoid friction with setuptools as much as possible.
I agree on saying that modifying packages on the fly is always hackish and should be discoraged. I disagree on prefering setuptools over distutils for practical reasons. If both setuptools and numpy.distutils overwrite the same functionality in distutils then these overwrites should be syncronized. i guess we need to support building scipy/numpy with both sets of tools: setuptools and numpy.distutils. Pearu
On Sep 25, 2007, at 11:43 AM, David M. Cooke wrote:
"Pearu Peterson" <pearu@cens.ioc.ee> writes:
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
When a supported method of building throws up a huge warning, I consider that broken ;-)
Let me echo this. I'm not sure what all the recent discussion on the fixes has left this particular issue but I think there needs to be some way to deal with this (either a way to suppress the message or handle things differently). This kind of warning is not going to be helpful to those that download software that uses numpy. They won't know what it means and won't know what to do about it. Perry Greenfield
Perry Greenfield wrote:
On Sep 25, 2007, at 11:43 AM, David M. Cooke wrote:
"Pearu Peterson" <pearu@cens.ioc.ee> writes:
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
When a supported method of building throws up a huge warning, I consider that broken ;-)
Let me echo this. I'm not sure what all the recent discussion on the fixes has left this particular issue but I think there needs to be some way to deal with this (either a way to suppress the message or handle things differently). This kind of warning is not going to be helpful to those that download software that uses numpy. They won't know what it means and won't know what to do about it.
While I agree with you, AFAIK, this appears only in svn. If you download from svn, you are accepting the higher possibility of something being broken, in my opinion. David
David Cournapeau wrote:
Perry Greenfield wrote:
On Sep 25, 2007, at 11:43 AM, David M. Cooke wrote:
"Pearu Peterson" <pearu@cens.ioc.ee> writes:
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
When a supported method of building throws up a huge warning, I consider that broken ;-)
Let me echo this. I'm not sure what all the recent discussion on the fixes has left this particular issue but I think there needs to be some way to deal with this (either a way to suppress the message or handle things differently). This kind of warning is not going to be helpful to those that download software that uses numpy. They won't know what it means and won't know what to do about it.
While I agree with you, AFAIK, this appears only in svn. If you download from svn, you are accepting the higher possibility of something being broken, in my opinion.
Well yes, but eventually we will cut a release. If we don't think the warning should be in the release, we should remove it from SVN. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
David Cournapeau wrote:
Perry Greenfield wrote:
On Sep 25, 2007, at 11:43 AM, David M. Cooke wrote:
"Pearu Peterson" <pearu@cens.ioc.ee> writes:
On Tue, September 25, 2007 5:57 pm, David M. Cooke wrote:
$ python setupegg.py bdist_egg Running from numpy source directory. ******************************************************** WARNING!WARNING!WARNING!WARNING!WARNING!WARNING!WARNING! [etc.]
This is a warning message. It does not break code.
When a supported method of building throws up a huge warning, I consider that broken ;-)
Let me echo this. I'm not sure what all the recent discussion on the fixes has left this particular issue but I think there needs to be some way to deal with this (either a way to suppress the message or handle things differently). This kind of warning is not going to be helpful to those that download software that uses numpy. They won't know what it means and won't know what to do about it.
While I agree with you, AFAIK, this appears only in svn. If you download from svn, you are accepting the higher possibility of something being broken, in my opinion.
Well yes, but eventually we will cut a release. If we don't think the warning should be in the release, we should remove it from SVN.
I think issuing such warnings is useful for developers as quite often one writes a code into a big system without being able to test the code with all practical corner cases. numpy.distutils is an example. So, this kind of a warning is like a mild assertion failure that should draw other developers attention that the code might do (or not do) something that is different what they have assumed. But the code will still run. If nobody will see the warning, then the new code is (may be) safe. If other developers see the warning then that means that they should work together to remove the cause of possible issues from the start. Given issue is a prefect example of this kind of situation that captures possible problems in the development state rather than from users feedback that may have very long time span. I think we should practice this kind of warnings more often. When making a release, such warnings should be disabled (read: removed when the issue is resolved). Pearu
On Fri, Sep 28, 2007 at 09:26:53AM +0200, Pearu Peterson wrote:
should be in the release, we should remove it from SVN.
I think issuing such warnings is useful for developers as quite often one writes a code into a big system without being able to test the code with all practical corner cases. numpy.distutils is an example.
So, this kind of a warning is like a mild assertion failure that should draw other developers attention that the code might do (or not do) something that is different what they have assumed. But the code will still run. If nobody will see the warning, then the new code is (may be) safe. If other developers see the warning then that means that they should work together to remove the cause of possible issues from the start.
Given issue is a prefect example of this kind of situation that captures possible problems in the development state rather than from users feedback that may have very long time span.
I think we should practice this kind of warnings more often.
When making a release, such warnings should be disabled (read: removed when the issue is resolved).
I would prefer if numpy were *always* in a release-ready state. Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place? David's idea of using scons to build everything sounds more appealing by the day. Distutils can still do the packaging, but at least we would have a modularised, testable, no-hassles build environment. Stéfan
On Fri, September 28, 2007 11:40 pm, Stefan van der Walt wrote:
I would prefer if numpy were *always* in a release-ready state.
Most of the time it actually is.
Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place?
The code is not so confusing - it has well-defined (though undocumented) structure that can be extended after one gets an idea how things work in distutils. The variety of different compilers and platforms makes developing distutils difficult. I guess none of the developers who dare to touch distutils have access to all platforms and compilers that we are trying to support. This fact will not change after switching to some other tool such as scons. We actually have tests for distutils - see buildbot.scipy.org.
David's idea of using scons to build everything sounds more appealing by the day. Distutils can still do the packaging, but at least we would have a modularised, testable, no-hassles build environment.
You obviously have little idea what does it mean to replace numpy.distutils with some other building tool;) Let this to be a challenge to work this out for anyone. Pearu
Pearu Peterson wrote:
On Fri, September 28, 2007 11:40 pm, Stefan van der Walt wrote:
I would prefer if numpy were *always* in a release-ready state.
Most of the time it actually is.
Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place?
The code is not so confusing - it has well-defined (though undocumented) structure that can be extended after one gets an idea how things work in distutils.
The variety of different compilers and platforms makes developing distutils difficult. I guess none of the developers who dare to touch distutils have access to all platforms and compilers that we are trying to support. This fact will not change after switching to some other tool such as scons. That's where I think we fundamentally disagree. I for one think distutils (here I mean the official, python.distutils, and only to build C/Fortran extensions; I will never talk about any other capabilities of distutils) is fundamentally flawed wrt to several key points, and I am not even talking about its implementation. Since you are much more familiar with distutils than me, I would really appreciate being told wrong: 1 - the way it tries to detect the platform capability. When you
Before I start to make my points, and because I feel this discussion is heating a bit, let me say that I never intend to criticize any numpy's developers work/code. I understand that distutils is complicated to extend, that numpy's needs go far beyond the usual needs of python package, and I really think that numpy.distutils has already achieved quite a lot. All other impressions are surely du to the fact that English is not my native language :) Now, although I certainly do not have as much experience as you with distutils, I have used autotools for non trivial projects, I have used scons as well, and I have some experience with complicated gnu make based projects (eg my numpy.gar script which can builds numpy and all its dependencies in different configuration/compilers on linux/unix). This is not to say I am knowledgeable, but at least I think I am aware of the difficulties of a cross platform build system. try to adapt to a platform in a build tool, you have at least two possibilities: either you hardcode every new platform, or you try to get the information from the platform. Distutils does the former, autoconf does the latter. For me, there is absolutely no question which one is better in this respect. This alone explains a lot of distutils fragility IMHO. For example, numpy.distutils define the "f2c" library for each new tool, but autoconf (more explicitely, the autoconf macro AC_F77_LIBRARY_LDFLAGS) finds it automatically from compiler output. This is a fundamental point, maybe the only most important one. 2 - numpy.distutils is difficult to extend. This, I think we all agree. This is also significant, because it means that when numpy.distutils fails, only a few people can change the situation. This should not be the case. If the problem is one compiler flag, it should not require anyone's involvement but the ones who experience the problem. 3 - Customizing compilation options: this is extremely difficult right now, and this again by design. 4 - Detecting libraries is extremely difficult: the first time I used system_info, it was totally wrong, and I had to modify it for each new platform. This is not good: I needed more time to make it right than I ever did for autotools based projects of mine, which really says a lot in my book. 5 - By design, scons does not depend on the environment. That is, it does NOT use PATH information, LD_LIBRARY_FLAGS, etc... which again makes it much more reliable. Of course, you can add command line arguments for configuration, but by default, nothing will stab you in the back. Now, you would rightly argue that doing all this in scons/whatever would take a lot of time just to replicate distutils, and you would be right. But I think it would require *less* time than implementing the things stated eg in http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp. But I don't ask you to take my word on it: I am more than willing to code a prototype of what I have in mind. But then, I need to know what is required for a prototype, to make a plan of what needs to be implemented: - Is a prototype required to support all platforms supported now ? - Is a prototype which can build numpy on Mac OS X/Linux/Win32 enough (keeping in mind that the key point is that adding new platforms would be much easier once the prototype is done). In which configuration ? - If required, I could try to make it possible to have two top setup scripts, one which uses the current distutils, one which use the scons-based one. As said earlier, I have started hacking on a numpy branch which, right now: - implements a scons command - a support library, which enables to build/run ctypes extensions working on many platforms (all the one I managed to use on the buildbot) - a support library to find libraries and headers in standard location on all supported platforms by scons (of which numpy.distutils is a subset AFAIK), and can be customized using site.cfg. One problem I did not foresee is how difficult it is to be sure that scons uses exactly the same tools than distutils (because distutils has no common api to get things such as compiler, compiler path, etc...); that's why contrary to what I thought first, using scons to build all the compiled extension may actually be easier than making them coexist, because scons and distutils have fundamentally different ways of looking for tools. To start building numpy itself with it would require more work (mostly tests for fortran / C abi), but with the help of people willing to help, I don't see major problems on this side, since scons provide a framework for autoconf-like testing. Also, this will certainly use some facilities of numpy.distutils (basically, for a first prototype, scons would mostly replace system_info.py, command/build_clib.py, command/build_ext.py). Does it sound like an acceptable plan to you ? David
Hi,
But I don't ask you to take my word on it: I am more than willing to code a prototype of what I have in mind. But then, I need to know what is required for a prototype, to make a plan of what needs to be implemented:
Thanks - this is an excellent mail, and makes the points very clearly. I think you are absolutely correct in saying a) we might be able to make things much easier with scons and b) the only way we can decide this is trying it. So, the prototype is key here, and thank you for persisting with this, it seems to me a very important effort which will at very least clarify where we want to go in the future, Matthew
David Cournapeau <david@ar.media.kyoto-u.ac.jp> writes:
Pearu Peterson wrote:
On Fri, September 28, 2007 11:40 pm, Stefan van der Walt wrote:
I would prefer if numpy were *always* in a release-ready state.
Most of the time it actually is.
Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place?
The code is not so confusing - it has well-defined (though undocumented) structure that can be extended after one gets an idea how things work in distutils.
Before I start to make my points, and because I feel this discussion is heating a bit, let me say that I never intend to criticize any numpy's developers work/code. I understand that distutils is complicated to extend, that numpy's needs go far beyond the usual needs of python package, and I really think that numpy.distutils has already achieved quite a lot. All other impressions are surely du to the fact that English is not my native language :)
Now, although I certainly do not have as much experience as you with distutils, I have used autotools for non trivial projects, I have used scons as well, and I have some experience with complicated gnu make based projects (eg my numpy.gar script which can builds numpy and all its dependencies in different configuration/compilers on linux/unix). This is not to say I am knowledgeable, but at least I think I am aware of the difficulties of a cross platform build system.
The variety of different compilers and platforms makes developing distutils difficult. I guess none of the developers who dare to touch distutils have access to all platforms and compilers that we are trying to support. This fact will not change after switching to some other tool such as scons.
That's where I think we fundamentally disagree. I for one think distutils (here I mean the official, python.distutils, and only to build C/Fortran extensions; I will never talk about any other capabilities of distutils) is fundamentally flawed wrt to several key points, and I am not even talking about its implementation. Since you are much more familiar with distutils than me, I would really appreciate being told wrong:
1 - the way it tries to detect the platform capability. When you try to adapt to a platform in a build tool, you have at least two possibilities: either you hardcode every new platform, or you try to get the information from the platform. Distutils does the former, autoconf does the latter. For me, there is absolutely no question which one is better in this respect. This alone explains a lot of distutils fragility IMHO. For example, numpy.distutils define the "f2c" library for each new tool, but autoconf (more explicitely, the autoconf macro AC_F77_LIBRARY_LDFLAGS) finds it automatically from compiler output. This is a fundamental point, maybe the only most important one.
When you mention the 'f2c' library, you mean the Fortran intrinsic and runtime libraries? We try to use the Fortran compiler for linking, so knowing those tends not to be needed. For comparision, we've got 2348 lines of source in numpy/distutils/fcompiler/, and fortran.m4 included with autoconf is 1234 lines. However, fortran.m4 mostly only deals with finding a Fortran compiler, finding flags for the libraries mentioned above, and name mangling (and nothing about optimisation flags). SCons also does its own handling of individual tools in much the same way as numpy.distutils (with, again, no optimisations). Now, one design point that I think distutils gets wrong is to use lists for describing compiler options: the description of a compiler (or any of the command line tools used) should be an object, that 'knows' how to turn, say, a .c file into a .o file (something like scons' Builders, I think). We have some tools where the pattern EXECUTABLE_NAME + SOME_OPTIONS + file.c + MORE_OPTIONS + file.o + EVEN_MORE_OPTIONS doesn't work (most notably, linking shared libraries on AIX must be handled specially). The CCompiler and FCompiler classes are too high-level for this, as they contain knowledge about multiple tools.
2 - numpy.distutils is difficult to extend. This, I think we all agree. This is also significant, because it means that when numpy.distutils fails, only a few people can change the situation. This should not be the case. If the problem is one compiler flag, it should not require anyone's involvement but the ones who experience the problem.
Fiddling compiler options shouldn't be too hard, espicially for the Fortran compilers; it's usually pretty obvious where to fiddle for those. But other things (C compilers, linking), I agree, could be easier.
3 - Customizing compilation options: this is extremely difficult right now, and this again by design.
My comment about using objects to represent tools applies here; creating a custom tool object should be easy.
4 - Detecting libraries is extremely difficult: the first time I used system_info, it was totally wrong, and I had to modify it for each new platform. This is not good: I needed more time to make it right than I ever did for autotools based projects of mine, which really says a lot in my book.
Could you detail your problems? system_info.py needs to be cleaned up (removal of repeative code, for instance) and finding libraries standardised (for instance, there should always be an environment variable and/or distutils key for things system_info wants to find, that will override system_info).
5 - By design, scons does not depend on the environment. That is, it does NOT use PATH information, LD_LIBRARY_FLAGS, etc... which again makes it much more reliable. Of course, you can add command line arguments for configuration, but by default, nothing will stab you in the back.
Blech, I hate the fact that scons doesn't respect my PATH. That means the compiler it may find is not necessarily the one that I would find at my command line. Makes bugs more obscure. Other variables (from my environment) that *should* be used as-is and not second-guessed by a build tool are things like TMPDIR, TMP, LANG, and PKG_CONFIG_PATH. Any variables like PATH or PKG_CONFIG_PATH are probably set for a reason.
Now, you would rightly argue that doing all this in scons/whatever would take a lot of time just to replicate distutils, and you would be right. But I think it would require *less* time than implementing the things stated eg in http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp.
Not if I steal stuff from scons ;-) (Or, for that matter, if I actually *had* any time to hack on it, which I don't right now.) Speaking of which, if the scons developers would make their code an actual Python package (for one thing, install it by default in site-packages/, and try to keep interfaces constantish), I would support using scons modules in numpy.distutils. From the looks of it, it's nicely laid out, with a good set of abstractions. Instead, they've set it up so that scons must be in control, through SConstruct files. Those have always had a faux-python feel to them for me: there are names injected into them (i.e. Environment) that don't come from an import statement, and the sharing of variables between SConscript files using the Export function is quite unpythonic.
But I don't ask you to take my word on it: I am more than willing to code a prototype of what I have in mind. But then, I need to know what is required for a prototype, to make a plan of what needs to be implemented: - Is a prototype required to support all platforms supported now ? - Is a prototype which can build numpy on Mac OS X/Linux/Win32 enough (keeping in mind that the key point is that adding new platforms would be much easier once the prototype is done). In which configuration ?
Those are the big three. If it builds on those, we're probably 80% of the way there. Of course, it's the other 20% that'll take 80% of the work, as usual.
- If required, I could try to make it possible to have two top setup scripts, one which uses the current distutils, one which use the scons-based one.
As said earlier, I have started hacking on a numpy branch which, right now: - implements a scons command - a support library, which enables to build/run ctypes extensions working on many platforms (all the one I managed to use on the buildbot) - a support library to find libraries and headers in standard location on all supported platforms by scons (of which numpy.distutils is a subset AFAIK), and can be customized using site.cfg.
One problem I did not foresee is how difficult it is to be sure that scons uses exactly the same tools than distutils (because distutils has no common api to get things such as compiler, compiler path, etc...); that's why contrary to what I thought first, using scons to build all the compiled extension may actually be easier than making them coexist, because scons and distutils have fundamentally different ways of looking for tools.
We can see what we can do to make it easier to get at those properties. For instance, it would be possible to separate the Fortran compiler detection/option-setting more from the distutils parts.
To start building numpy itself with it would require more work (mostly tests for fortran / C abi), but with the help of people willing to help, I don't see major problems on this side, since scons provide a framework for autoconf-like testing. Also, this will certainly use some facilities of numpy.distutils (basically, for a first prototype, scons would mostly replace system_info.py, command/build_clib.py, command/build_ext.py).
If you have the time, I think it's worth a try. Unfortunately, I don't have much time to contribute to it. Basically, my opinion is not to replace numpy.distutils with scons (for the simple reason of interacting nicely with distutils). However, I'm not opposed to chucking parts of numpy.distutils and replacing them with scons-like concepts. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
David M. Cooke wrote:
When you mention the 'f2c' library, you mean the Fortran intrinsic and runtime libraries? We try to use the Fortran compiler for linking, so knowing those tends not to be needed.
But what if we want to use C++ code in the future ? Then you have no choice but using the C++ compiler. But more to the point, this was an example. All AC_F77_* macro (or AC_C_*, etc...) are much, much more solid than distutils. And this is by design.
For comparision, we've got 2348 lines of source in numpy/distutils/fcompiler/, and fortran.m4 included with autoconf is 1234 lines.
To make it clear: I was not talking about using autoconf (which for once does not work on windows), but about the concept that almost everybody agrees autoconf is good at, that is working on a new platform not foreseen by the developer. And this is due to a fundamentally different way to look for platform capabilities. We have a huge advantage compared to autoconf: python vs m4 + shell. Most of the lines in fortran.m4 are difficult because autoconf has to deal with weird shells, antic platform bugs, and because of m4 and its antic quoting rules.
However, fortran.m4 mostly only deals with finding a Fortran compiler, finding flags for the libraries mentioned above, and name mangling (and nothing about optimisation flags). SCons also does its own handling of individual tools in much the same way as numpy.distutils (with, again, no optimisations). What do you mean by no optimizations ? Dealing with facilities provided by scons and dealing with capabilities are two different things, IMHO, because I trust scons enough to work on what is supported. But for example, when distutils is looking for something, it is always hardcoded: looking for a library ? Distutils looks for a file, autoconf is looking for something usable by the linker. This makes a huge difference concerning the robustness (and the maintenance).
Now, one design point that I think distutils gets wrong is to use lists for describing compiler options: the description of a compiler (or any of the command line tools used) should be an object, that 'knows' how to turn, say, a .c file into a .o file (something like scons' Builders, I think). We have some tools where the pattern EXECUTABLE_NAME + SOME_OPTIONS + file.c + MORE_OPTIONS + file.o + EVEN_MORE_OPTIONS doesn't work (most notably, linking shared libraries on AIX must be handled specially). The CCompiler and FCompiler classes are too high-level for this, as they contain knowledge about multiple tools. This is something that scons seems to make right.
Fiddling compiler options shouldn't be too hard, espicially for the Fortran compilers; it's usually pretty obvious where to fiddle for those. But other things (C compilers, linking), I agree, could be easier. The point is, did it ever happen that somebody who had a problem with a compiler flag solved it by himself ?
Could you detail your problems? system_info.py needs to be cleaned up (removal of repeative code, for instance) and finding libraries standardised (for instance, there should always be an environment variable and/or distutils key for things system_info wants to find, that will override system_info).
system_info is looking for files, which is not the right way in my opinion. For example, in audiolab, my system_info subclass is 70 lines (http://projects.scipy.org/scipy/scikits/browser/trunk/audiolab/setup.py), all this to find one library ! In scons (or autoconf, or any other build tool I know), this is one line. Maybe I am doing things wrong, the code started when I knew nothing about python and just copied what I found in system_info.py, dunno. But the code in system_info does not seem to imply this is simple or robust. This is really striking for me: I have no problem with complicated code, if it matches the complexity of the task it is trying to solve. But to me, distutils makes anything trivial complicated (again, just talking about the things related to compiled code).
Blech, I hate the fact that scons doesn't respect my PATH. That means the compiler it may find is not necessarily the one that I would find at my command line. Makes bugs more obscure.
I fundamentally disagree on this one. But anyway, we don't even need to agree, since you can easily import the variables you want. So we can make this an option (which has always been my intention anyway), but by default, not importing anything is more robust for deployment, no ? People who have non standard tools or tools in non standard locations are the ones who should make the effort, not the contrary IMHO.
Not if I steal stuff from scons ;-)
I doubt you can "steal" things from scons, for the very reason you described.
(Or, for that matter, if I actually *had* any time to hack on it, which I don't right now.)
Speaking of which, if the scons developers would make their code an actual Python package (for one thing, install it by default in site-packages/, and try to keep interfaces constantish), I would support using scons modules in numpy.distutils. From the looks of it, it's nicely laid out, with a good set of abstractions. Instead, they've set it up so that scons must be in control, through SConstruct files.
Those have always had a faux-python feel to them for me: there are names injected into them (i.e. Environment) that don't come from an import statement, and the sharing of variables between SConscript files using the Export function is quite unpythonic. I would agree in theory, but compared to python distutils, which one do you think people can understand more easily ? For once, distutils (the one from python) adds many members attributes at runtime, in many functions (not just __init__); this combined to the fact that it is mostly not commented + not documented + not maintained (nobody to ask
I think this is partly due to the fact that scons support any python starting from 1.5.2, and partly to the fact that scons is not imperative (in fact, this is one key difference with waf AFAIK). But scons has a package scons-local, which can be used as a drop in (in fact, that's what I did in numpy.scons branch: in theory, we could make our modification there), which somewhat alleviates the problem. this, nothing on the internet), makes it harder for me to understand than I ever did with the autotools, which are not known for their user friendliness.
Those are the big three. If it builds on those, we're probably 80% of the way there.
Of course, it's the other 20% that'll take 80% of the work, as usual.
I agree that making the 20 % will take the most time (anybody with at least some experience with cross platform building would, I guess). But I think this will be much easier than with distutils.
We can see what we can do to make it easier to get at those properties. For instance, it would be possible to separate the Fortran compiler detection/option-setting more from the distutils parts.
For example, one thing which caused me problems is to get the name of the compiler: for unix compiler, just using the member compiler[0] gives me the executable, from which I have a dict to convert to scons names. But MVSCCompiler does not have such a member; and to find the path of the executable, *sometimes*, you need to call initialize (which of course is not documented). This is not just unpythonic, this is just really bad code; what's the point to have a base class for compiler if this does not even have a common way to get the compiler name ?
To start building numpy itself with it would require more work (mostly tests for fortran / C abi), but with the help of people willing to help, I don't see major problems on this side, since scons provide a framework for autoconf-like testing. Also, this will certainly use some facilities of numpy.distutils (basically, for a first prototype, scons would mostly replace system_info.py, command/build_clib.py, command/build_ext.py).
If you have the time, I think it's worth a try. Unfortunately, I don't have much time to contribute to it.
Basically, my opinion is not to replace numpy.distutils with scons (for the simple reason of interacting nicely with distutils).
Again, never had the intention to totally replace distutils: distutils would still drive the whole build. Ideally, if it works out, I intend to replace the build_ext and build_clib commands; it would still works by calling setup.py build, it would still put everything in build, and install using --prefix, etc... This would be totally transparent to the user (except maybe for setting compiler at command line, and even, this may be doable to stay 100% backward compatible at this level). You can take a look at openalea, who did the exact same thing: http://openalea.gforge.inria.fr/wiki/doku.php Everything else seems to work quite well in distutils, no ? cheers, David
Hi Pearu On Sat, Sep 29, 2007 at 12:15:23AM +0300, Pearu Peterson wrote:
Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place?
The code is not so confusing - it has well-defined (though undocumented) structure that can be extended after one gets an idea how things work in distutils.
The variety of different compilers and platforms makes developing distutils difficult. I guess none of the developers who dare to touch distutils have access to all platforms and compilers that we are trying to support. This fact will not change after switching to some other tool such as scons.
We actually have tests for distutils - see buildbot.scipy.org.
I should have distinguished between numpy.distutils and distutils in my previous code. I think numpy.distutils does as well as can be expected based on (python) distutils. If we were to design a build system from the ground up, it would probably look very different.
David's idea of using scons to build everything sounds more appealing by the day. Distutils can still do the packaging, but at least we would have a modularised, testable, no-hassles build environment.
You obviously have little idea what does it mean to replace numpy.distutils with some other building tool;) Let this to be a challenge to work this out for anyone.
Would the effort be worth it in the end, if it allows more developers to extend the build system (without worrying about the distutils police at night)? Maybe you're right: they say a person should be careful what you ask for, lest it come true :) Cheers Stéfan
On Fri, Sep 28, 2007 at 09:26:53AM +0200, Pearu Peterson wrote:
should be in the release, we should remove it from SVN. I think issuing such warnings is useful for developers as quite often one writes a code into a big system without being able to test the code with all practical corner cases. numpy.distutils is an example.
So, this kind of a warning is like a mild assertion failure that should draw other developers attention that the code might do (or not do) something that is different what they have assumed. But the code will still run. If nobody will see the warning, then the new code is (may be) safe. If other developers see the warning then that means that they should work together to remove the cause of possible issues from the start.
Given issue is a prefect example of this kind of situation that captures possible problems in the development state rather than from users feedback that may have very long time span.
I think we should practice this kind of warnings more often.
When making a release, such warnings should be disabled (read: removed when the issue is resolved).
I would prefer if numpy were *always* in a release-ready state. Why can't we instrument tests for distutils? If the code is so confusing that we can't test it (or "practical corner cases"), should we be using it as a core ingredient in the first place? I think part of this issue is related to the source control system we are using. My impression is that with the current system, having always something at least buildable in the trunk and usable is preferable; with a better source constrol system, we would not have such a problem, or at least would make the problem less pervasive (we would not need to work
Stefan van der Walt wrote: directly in the trunk so often). Now, concerning the build tool itself: testing build tools is inherently difficult; This is similar to why cross compilation is inherently difficult: the platform to build the build tool and the platform to test it are different. Replacing "everything" with scons (or any other tool) is non trivial, and will take some time. I think it worths it, because IMHO, distutils has fundamental design flaw wrt our needs in numpy/scipy (see my answer to Pearu in the same thread). David
participants (9)
-
cookedm＠physics.mcmaster.ca -
David Cournapeau -
David M. Cooke
-
dmitrey -
Matthew Brett -
Pearu Peterson -
Perry Greenfield -
Robert Kern -
Stefan van der Walt