David Cournapeau wrote:

Pearu Peterson wrote:

...

2) It seems that scons does not interfare with numpy.distutils much. If this is true and numpy/scipy builds will not break when scons is not installed then I think you could continue the scons support development in trunk. It won't break if scons is not installed because scons sources are copied into the branch. Scons developers explicitely support this:

http://www.scons.org/faq.php#SS_3_3

(AFAIK, it does not pose any problem license-wise, since scons is new BSD license; it adds ~350 kb of compressed source code to numpy).

I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils.

Now, concerning migrating all compiled extensions to scons, I would prefer avoiding doing it in the trunk; but I heard horror stories about subversion and merging, so maybe staying outside the trunk is too risky ?

I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge.

Also, when I said that I don't see big problems to replace distutils for compiled extensions, I lied by over-simplification. If scons is used for compiled extension, with the current design, distutils will call scons for each package. Calling scons is expensive (importing many modules, etc...: this easily takes around one second for each non trivial Sconscript), and also, because each of them is independent, we may check several times for the same thing (fortran compiler, etc...), which would really add up to the build time.

I see two approaches here: - do not care about it because numpy is unlikely to become really bigger - considering that numpy is really one big package (contrary to scipy which, at least in my understanding, is gearing toward less inter-package dependencies ?), we should only have one big sconscript for configuration (checking blas/lapack, compilers, etc...), and use scons recursively. In this case, it should not be much slower than the current system.

Note that building a subpackage in subpackage directory must be supported. So a big sconscript may not be an option. The third approach would be to cache those checks that are called frequently to, say, $HOME/.numpy/scons. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult.

...

4) though, we cannot remove numpy.distutils for backward compatibility with software using numpy.distutils. However, it should be possible to enhance Configuration class to generate the corresponding SConstruct scripts. It will save some work when converting configuration() functions to SConstruct scripts. Could you elaborate on this point ? I am not sure what you mean by generating Configuration class ?

I meant that Configuration class could have a method, say toscons(<filename>), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. Pearu

David Cournapeau wrote:

Pearu Peterson wrote:

...

I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ?

Not always. For example, in debian one has to install python-dev separately from python.

Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils).

Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3.

...

I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. I don't have strong experience in subversion, so I was afraid of that. Do I understand correctly that you suggest opening a new dev branch, and then do all subsequent dev (including non distutils/scons related ones) there ?

No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that.

...

numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. Scons supports recursive calls. To be more clear about the possibilities of scons wrt to this, let's take a simplified example:

root/Sconstruct root/numpy/Sconscript root/numpy/core/Sconscript root/numpy/linalg/Sconscript

If you are in root and call scons (> is a shell prompt):

root > scons

Then it will call recursively all the sconscript (as long as you request it in the sconscript files). The other supported method is

root/numpy/linalg > scons -u

this will look every parent directory to find the "root" Sconstruct.

My point was that root/numpy/linalg > scons should work (without the -u option). A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage.

...

I meant that Configuration class could have a method, say toscons(<filename>), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. I don't think it would worth the effort for numpy (the main work really is to implement and test all the checkers: blas/lapack, fortran). Now, as a general migration tool, this may be useful. But since we would still use distutils, it would only be useful if it is easy to develop such as tool.

Yes, that's a low priority feature.

P.S: would it be easy for you to make a list of requirements for fortran ? By requirement, I mean things like name mangling and so on ? Something like the autoconf macros: http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf...

To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils. f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly. Note that using hardcoded name mangeling flags may be still required for certian Fortran 90 compilers (which ones exactly, I don't member now) that by default produce symbol names with special characters like $ or . for F90 modules and making these names unaccessible to C programs. Pearu

Pearu Peterson wrote:

...

Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils).

Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3.

Sure, I just meant that we don't have to worry that scons may depend on 2.4 or 2.5 features (I am not sure waf have such a requirement, for example).

No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that.

Ah, ok. I am not sure I can guarantee it will never affect the build process. As this is a pretty fundamental change, I though about doing that in a 1.1 branch of numpy. But I do not have strong opinion on that, nor much experience in that area.

My point was that

root/numpy/linalg > scons

should work (without the -u option).

Since scons is called by distutils, and never by the user, I don't see how this can be a problem ? distutils would have to figure out when to use -u, not the user.

A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage.

Note that scons caches the configuration, so I don't think speed will be an issue here (except maybe if you keep changing the configuration before building). What I could do is for each subpackage, to declare the tests to use in each subpackage, but then, what to do if two packages have some common tests ? I cannot just remove double tests, because the order is significant (because of link options, for example). The way I see it, either we keep the current behaviour (each package is totally independant, scons is called for each subpackage, and scons has no way to know about other subpackages; this has the disadvantage of being slower: this will be significant of many subpackages use costly checks like fortran mangling and so on), or we have a main sconscript with the configuration, which does not prevent building subpackages, but which requires a global configuration. If none of those approach seems right to you, I will see if I can come up with something better, but this will certainly add some complexity (or I am just stupid to see an obvious solution :) ).

To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils.

f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly.

Thank you for those information. Do I understand correctly (sorry for being slow, but I don't know anything about fortran) that what you need is macro like: - AC_F77_WRAPPERS (which defines C macros for converting C functions to fortran compiler mangling) - AC_F77_FUNC (which retrieves the name mangled by fortran linked from the 'canonical' name: this is already implemented, as it is necessary for checking blas/lapack) And that's it ? cheers, David

Pearu Peterson wrote:

Yes. I was considering the situations where distutils in not used at all. May be it will be relevant in future..

If distutils is not used at all, there will be more difficult problems to solve than this one, then. I think it is safe to assume that distutils will stay there for quite some time, no ? Note that I try hard in numpy/scons to separate the scons specific from the parts which are not specific to scons. If in the future, we decide to go on the waf route, or whatever, I hope that most of the logic would be easily reused: for example, parsing the fortran link output to guess the link options for C/Fortran is totally independant from scons (this has the nice advantage of making regression tests possible on all platforms).

I would prefer the former solution where each package is totally independent. Let's first see what is the acctual performance and then maybe consider more complex solutions if the build is really slow.

I measured the initialization time of scons if called for every subpackage in numpy/: this adds less than two percents of total build time, so this is not as bad as I feared (specially since only a few package would need scons: pure python ones do not need them). And scons can also give some speed advantage, because it explicitely support in a totally safe way parallel builds. So basically, I went to the same conclusion than you. I have started porting numpy.core to scons (by far the most difficult subpackage, at least a priori) following this route.

These macros are fine. May be we need more such macros for F90 compilers but we can discuss about these when the corresponding issues will emerge.

Oh, I forgot to say it, but the macro would be implemented for both F77 and F90 (and F2003 or whatever should we need it). As most of the logic is independent of the version used, this is not a problem.

However, if you could also provide the macros that f2py uses then everything would work with no change in f2py part. f2py uses the following defintion of F_FUNC, for example:

I can of course change the name of the macro (this could be an option, even). I guess it is better to follow f2py convention than autotools here, in our case.

system_info has a feature of providing architecture specific and optimization flags for fortran compilers. Could scons support such a feature?

scons uses the concept of Environment objects: they have sensible defaults (compiler/platform combination), but you can override anything you want, and they are used for compilation. env = Environment() env.AppendUnique(F77FLAGS = "-O3") env.Program('hello', source = ['hello.f']) Of course, in our case, we should provide some support such as instead of adding -O3, we have something like a FULL_OPTIMIZED variable which contains the optimization flags tweaked for a particular compiler/platform. Better, we can check that a given flag is actually supported. This is something I have not done yet, but that's the kind of things scons does really well, so I don't see any problems here. The main problem right now is to convert all the checks in numpy/core/setup.py, which is the only non trivial setup.py file as far as I can tell in numpy. This will take a few hours to get it right on major platforms :) cheers, David P.S If you have access to some fortran compiler you would like to check, you can check my bzr branch on launchpad (this is done outside numpy because some things may be merged to scons sources at some point): https://code.launchpad.net/~david-ar/numpy.scons.support/dev And test it by: scons CC=ccompiler F77=f77compiler F90=f90compiler. For example: scons CC=icc F77=g77 F90=ifort. (If you change compiler, you may need to add the option --config=force). This is a "fake" sconscript which tests the following: - does the fortran compiler works (e.g. can compile a dummy program) - find the dummy main - find the verbose flag (useful to find informations at runtime) - find the name mangling - find the necessary options for C/Fortran mix (only implemented for F77 for now). There is a pretty good chance that you will be able to break it (in particular finding the options for C/Fortran: I have not yet implemented all the necessary parsing), but in this case, please provides me with the config.log (which logs all the configuration dones by scons).