Michiel Jan Laurens de Hoon wrote:
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together.
Michiel, you are free to work on setup.py all you want :-) Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy. I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries. There should be a way to replace functionality that is clean and does not require editing setup.py files. Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not. Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas. Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here. Feedback and criticisms encouraged and welcomed..... -Travis
On Thu, 24 Mar 2005, Travis Oliphant wrote:
Michiel Jan Laurens de Hoon wrote:
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together.
Michiel, you are free to work on setup.py all you want :-)
Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy.
I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries.
There should be a way to replace functionality that is clean and does not require editing setup.py files.
Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not.
Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas.
Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here.
Hmm, the idea of introducing scipylite, scipyatlas, scipyfortran packages does not sound like a good idea. The usage of atlas or fortran blas/lapack or vendor based blas/lapack libraries is an implementation detail and should not be reflected in scipy_base package structure. This is because such an approach is not suitable for writing portable Numeric3 based applications or packages. For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of potential users for this package. I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here. Ideally, scipy_base should provide a complete interface to LAPACK routines, but not immidiately, of course. Now, depending on the availability of compilers and resources in a particular computer, the following would happen: 1) No Fortran compiler, no lapack libraries in the system, only C compiler is available --- f2c generated lite-lapack C sources are used to build lapack extension module; wrappers to lapack routines, for which there are no f2c generated sources, are disabled by f2py `only:` feature. lite-lapack C sources come with scipy_base sources. 2) No Fortran compiler, system has lapack libraries (atlas or Accelerate or vecLib), C compiler is available --- system lapack library will be used and a complete lapack extension module can be built. 3) Fortran and C compiler are available, no lapack libraries in the system --- Fortran lite-lapack sources are used to build lapack extension module; lite-lapack Fortran sources come with scipy_base sources. Similar to the case (1), some wrappers are disabled. 4-..) other combinations are possible and users can choose their favorite approach. The availability of system resources can be checked using scipy_distutils.system_info.get_info. Checking the availability of Fortran compiler should be done in a configuration step and only when an user specifically asks for it, by default we should assume that Fortran compiler is not available. The same should apply also to atlas/lapack/blas libraries, by default f2c generated lite-lapack C sources will be used. In this way users that only need Numeric3 array capabilities will avoid all possible troubles that may show up when using all possible resources for speed on an arbitrary computer. Btw, I would suggest using `scipy <dot> <package>` instead of `scipy <package>` or `scipy <underscore> <package>` for naming packages. Pearu
For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of >potential users for this package.
While I'm not in love with my suggestion and would prefer to see better ones put forward, wouldn't any system that uses routines not available unless you have a fortran-compiled package installed be a problem? I was just proposing not "hiding" this from the developer but making it explicit. What do you propose to do for those situations? I was just proposing putting them in a separate hierarchy so the developer is aware he is using something that requires fortran. I actually think that it's somewhat of a non-issue myself, and feel that people who don't have fortran compilers will look for binaries anyway.
I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here.
Ideally, scipy_base should provide a complete interface to LAPACK routines, but not immidiately, of course. Now, depending on the availability of compilers and resources in a particular computer, the following would happen: 1) No Fortran compiler, no lapack libraries in the system, only C compiler is available --- f2c generated lite-lapack C sources are used to build lapack extension module; wrappers to lapack routines, for which there are no f2c generated sources, are disabled by f2py `only:` feature. lite-lapack C sources come with scipy_base sources. 2) No Fortran compiler, system has lapack libraries (atlas or Accelerate or vecLib), C compiler is available --- system lapack library will be used and a complete lapack extension module can be built. 3) Fortran and C compiler are available, no lapack libraries in the system --- Fortran lite-lapack sources are used to build lapack extension module; lite-lapack Fortran sources come with scipy_base sources. Similar to the case (1), some wrappers are disabled. 4-..) other combinations are possible and users can choose their favorite approach.
Great, Sounds like Pearu has some good ideas here. I nominate Pearu to take the lead here. Michiel sounds like he? wants to keep the Numeric, RandomArray, LinearAlgebra naming conventions forever. I want them to be more coordinated like scipy is doing with scipy.linalg scipy.stats and scipy_base ( I agree scipy.base is better). What are the opinions of others on this point. Of course the names Numeric, RandomArray, and LinearAlgebra will still work, but I think they should be deprecated in favor of a better overall design for numerical packages. What do others think?
On Fri, 25 Mar 2005, Travis Oliphant wrote:
For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of >potential users for this package.
While I'm not in love with my suggestion and would prefer to see better ones put forward, wouldn't any system that uses routines not available unless you have a fortran-compiled package installed be a problem? I was just proposing not "hiding" this from the developer but making it explicit.
What do you propose to do for those situations? I was just proposing putting them in a separate hierarchy so the developer is aware he is using something that requires fortran. I actually think that it's somewhat of a non-issue myself, and feel that people who don't have fortran compilers will look for binaries anyway.
Such an situation can be avoided if a package is extended with new wrappers parallel for all backend cases. For example, when adding a new interface to a lapack routine then to the scipy_base sources must be added both Fortran and f2c versions of the corresponding routine. Pearu
participants (2)
-
Pearu Peterson -
Travis Oliphant