First of all, thanks to the Numerical Python developers for releasing version 23.8 of Numerical Python. It compiles out of the box and avoids the blas/lapack compilation problems in earlier versions, which makes my life as a developer a lot easier. Thanks! Travis Oliphant wrote:
I wanted to let people who may be waiting, that now is a good time to help with numeric3. The CVS version builds (although I"m sure there are still bugs), but more eyes could help me track them down.
Currently, all that remains for the arrayobject is to implement the newly defined methods (really it"s just a re-organization and re-inspection of the code in multiarraymodule.c to call it using methods).
I downloaded Numeric3 today and installed it. The compilation and installation run fine. There are still some warnings from the compiler here and there, but I guess they will be fixed some other time. During compilation, I noticed that some test program is run, presumably for configuration. The test program is compiled by a different compiler as the one used in the build process. Note that "python setup.py config" is available in the standard distutils, so it may be better to use that instead of a self-defined configuration tool. For one thing, it'll make sure that the compiler used for configuration is the same as the one used for compilation. To use Numeric3, I did "from ndarray import *". I guess for the final version, this will be "from Numeric import *"? When using ndarray, I got a core dump using "zeros": $ python Python 2.5a0 (#1, Mar 2 2005, 12:15:06) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information.
from ndarray import * zeros(5) creating data 0xa0c03d0 associated with 0xa0d52c0 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd') Segmentation fault (core dumped)
With Python 2.4, the segmentation fault occurs slightly later: $ python2.4 Python 2.4 (#1, Dec 5 2004, 20:47:03) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information.
from ndarray import * zeros(5) creating data 0xa0a07f8 associated with 0xa0d6230 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd')
^D freeing 0xa0a07f8 associated with array 0xa0d6230 freeing 0xa123b88 associated with array 0xa0d6230 Segmentation fault (core dumped)
Finally, I tried to compile a C extension module that uses Numerical Python (by replacing #include <Numeric/arrayobject.h> by #include <ndarray/arrayobject.h>): $ python setup.py build running build running build_py running build_ext building 'Pycluster.cluster' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -Isrc -Iranli b/src -I/usr/local/include/python2.5 -c python/clustermodule.c -o build/temp.cyg win-1.5.12-i686-2.5/python/clustermodule.o In file included from python/clustermodule.c:2: /usr/local/include/python2.5/ndarray/arrayobject.h:76: warning: redefinition of `ushort' /usr/include/sys/types.h:85: warning: `ushort' previously declared here /usr/local/include/python2.5/ndarray/arrayobject.h:77: warning: redefinition of `uint' /usr/include/sys/types.h:86: warning: `uint' previously declared here These two warnings are probably not so serious, but it would be better to get rid of them anyway. python/clustermodule.c: In function `parse_data': python/clustermodule.c:38: warning: passing arg 1 of pointer to function from in compatible pointer type The offending line 38 is: { PyArrayObject* av = (PyArrayObject*) PyArray_Cast(*array, PyArray_DOUBLE); where array is a PyArrayObject**. Another warning was that PyArrayObject's "dimensions" doesn't seem to be an int array any more. Finally, when linking I get an undefined reference to _PyArray_API. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Hi Travis, I just had a quick look at Numeric3, checked out with cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/numpy co -D 2005-03-18 -P Numeric3 (as you already warned, the current CVS does not compile for me). After that I saw Michiels mail, so my results below just adds another "data-point"... On Fri, 18 Mar 2005, Michiel Jan Laurens de Hoon wrote:
Travis Oliphant wrote:
I wanted to let people who may be waiting, that now is a good time to help with numeric3. The CVS version builds (although I"m sure there are still bugs), but more eyes could help me track them down.
Currently, all that remains for the arrayobject is to implement the newly defined methods (really it"s just a re-organization and re-inspection of the code in multiarraymodule.c to call it using methods).
[...]
When using ndarray, I got a core dump using "zeros":
$ python Python 2.5a0 (#1, Mar 2 2005, 12:15:06) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information.
from ndarray import * zeros(5) creating data 0xa0c03d0 associated with 0xa0d52c0 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd') Segmentation fault (core dumped)
With Python 2.4, the segmentation fault occurs slightly later: $ python2.4 Python 2.4 (#1, Dec 5 2004, 20:47:03) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information.
from ndarray import * zeros(5) creating data 0xa0a07f8 associated with 0xa0d6230 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd')
^D freeing 0xa0a07f8 associated with array 0xa0d6230 freeing 0xa123b88 associated with array 0xa0d6230 Segmentation fault (core dumped)
Python 2.3.5 (#1, Mar 22 2005, 11:11:34) Type "copyright", "credits" or "license" for more information. IPython 0.6.13_cvs -- An enhanced Interactive Python. ? -> Introduction to IPython's features. %magic -> Information about IPython's 'magic' % functions. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]:from ndarray import * In [2]:arange(10) Out[2]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 'l') In [3]:arange(10.0) Out[3]:array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], 'd') In [4]: In [4]:arange(10.0) zsh: 7191 segmentation fault ipython Without ipython the segfault is even earlier: Python 2.3.5 (#1, Mar 22 2005, 11:11:34) [GCC 3.3.5 (Debian 1:3.3.5-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from ndarray import * arange(10) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 'l') arange(10.0) zsh: 7192 segmentation fault python
Have you already found the origin of this? If so, which version should I download for further testing? If not, if you need help in debugging this one, just let me know (+some hints how to tackle this). Best, Arnd
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together. --Michiel. Arnd Baecker wrote:
Hi Travis,
I just had a quick look at Numeric3, checked out with cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/numpy co -D 2005-03-18 -P Numeric3 (as you already warned, the current CVS does not compile for me).
-- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Michiel Jan Laurens de Hoon wrote:
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together.
Well, CVS, makes that somewhat easy if we just commit changes regularly, and update regularly. But, I understand that people may want to know what kinds of things they could work on right now. I'm working on finishing adding methods. I'd like to create the new core distribution on an SVN server. Enthought is willing to host the SVN server as far as I know. SVN is easy to use and is supposed to be easier to manage than CVS. Current needs: - the PEP for the __index__ method added to Python needs to be written and the code implemented --- this is not that hard for the budding Python contributor - the PEP for a good "buffer" object (this has been called by others a "byte" array which might be a good name. Essentially, it needs to be a light-weight object around a chunk of memory -- i.e. a way to allocate memory through Python. We would like to standardize on a set of meta information that could be used to "understand" this memory as a numeric array. Then, other objects which used this buffer as a memory block would just have to expose the meta information in order to make seamless the transfer of data from one application to another. We need to be vocal about the value of the buffer object. This PEP is one way to do that. There are some people who think buffer objects were a "bad idea." This is primarily because of a fatal flaw in some objects that both expose a memory pointer through the buffer protocol AND allow the object's memory to be reallocated (using realloc) --- Numeric does not do this. This problem could actually be easily fixed by a good Python memory allocator that returns a simple memory object. If people who wanted memory went through it's C-API (instead of using malloc and realloc), much of the problems would be alleviated. This is what the new "byte" object should be. I think it also wise to expect the "byte" object to have an attribute called "meta" that would just be a dictionary of "other information" you might want to pass to something using the buffer protocol. - a record array class. This should be adapted from the numarray record array class and probably inherit from the ndarray type. - ufunc modifications. This is where I'm headed after the array methods task is done. If people have ideas about how ufuncs should be handled, now is the time to voice them. If somebody could help me here, it would be great. But, in a couple of days, I will be spending the next chunck of my (spare) time on ufunc modifications. -Travis
Michiel Jan Laurens de Hoon wrote:
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together.
Michiel, you are free to work on setup.py all you want :-) Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy. I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries. There should be a way to replace functionality that is clean and does not require editing setup.py files. Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not. Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas. Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here. Feedback and criticisms encouraged and welcomed..... -Travis
While I basically agree with your setup, I think that there is no need to call it scipylite. Sticking to the Numeric structure and names is to the advantage of both current SciPy and current Numerical Python users. The advantage to current Numerical Python users is obvious -- and there are many more of them than of SciPy users. For SciPy users, it is in their best interest that as many people as possible go over to Numeric3, in order to avoid another split in the Numerics community. Now, if I talk with the other pygist or biopython developers and tell them there is a new Numerical Python package which solves some of the issues with the older versions, I have a good chance to convince them to update pygist/biopython to the Numeric3 API. If I tell them that there is a scipylite package that intends to replace Numerical Python: Forget it. It will be ignored. You may not care about pygist or biopython in particular, but developers of other packages will make the same consideration, so you may end up with some numerical / graphics packages working with scipylite and others with Numerical Python 23.8. It's better to get everybody on board. Secondly, we have confused users more than enough with the Numerical Python / numarray / Numeric3 split. We should not add one more new name to the equation. Third, there is lots of code out there that imports LinearAlgebra or RandomArray etcetera. Why force our users to go through the trouble of changing those imports? I don't see the benefit to the users. Finally, the word scipylite has no meaning. As SciPy evolves into a website where scientific software for Python can be downloaded, there will not be a scipy-full nor a scipy-lite. --Michiel. Travis Oliphant wrote:
Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy.
I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries.
There should be a way to replace functionality that is clean and does not require editing setup.py files.
Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not.
Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas.
Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here. Feedback and criticisms encouraged and welcomed.....
-- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Michiel Jan Laurens de Hoon wrote:
While I basically agree with your setup, I think that there is no need to call it scipylite. Sticking to the Numeric structure and names is to the advantage of both current SciPy and current Numerical Python users. The advantage to current Numerical Python users is obvious -- and there are many more of them than of SciPy users. For SciPy users, it is in their best interest that as many people as possible go over to Numeric3, in order to avoid another split in the Numerics community. Now, if I talk with the other pygist or biopython developers and tell them there is a new Numerical Python package which solves some of the issues with the older versions, I have a good chance to convince them to update pygist/biopython to the Numeric3 API. If I tell them that there is a scipylite package that intends to replace Numerical Python: Forget it. It will be ignored. You may not care about pygist or biopython in particular, but developers of other packages will make the same consideration, so you may end up with some numerical / graphics packages working with scipylite and others with Numerical Python 23.8. It's better to get everybody on board.
Secondly, we have confused users more than enough with the Numerical Python / numarray / Numeric3 split. We should not add one more new name to the equation.
Third, there is lots of code out there that imports LinearAlgebra or RandomArray etcetera. Why force our users to go through the trouble of changing those imports? I don't see the benefit to the users.
Finally, the word scipylite has no meaning. As SciPy evolves into a website where scientific software for Python can be downloaded, there will not be a scipy-full nor a scipy-lite.
--Michiel.
It looks to me as though getting numarray/Numeric sorted out, and getting it right, will be sufficient work for now. It's far better to concentrate the limited resources on that and to leave the complexities of SciPy for another day. I wonder about introducing another distribution system (SVN?) when some of us have barely mastered CVS. Colin W.
Travis Oliphant wrote:
Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy.
I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries.
There should be a way to replace functionality that is clean and does not require editing setup.py files.
Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not.
Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas.
Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here. Feedback and criticisms encouraged and welcomed.....
Travis Oliphant wrote:
I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries.
There should be a way to replace functionality that is clean and does not require editing setup.py files.
Anybody with good ideas about how to do this well is welcome to speak up.
Doing this automatically without editing setup.py may be too complicated. Quoting from the Numerical Python manual: 'A frequent request is that somehow the maintainers of Numerical Python invent a procedure which will automatically find and use the "best" available versions of these libraries. This is not going to happen.' "these libraries" being BLAS and LAPACK. However, what we can do is to put some frequently encountered options in setup.py commented out, and say "uncomment this line if you have BLAS and LAPACK preinstalled on your Mac" etcetera. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
On Thu, 24 Mar 2005, Travis Oliphant wrote:
Michiel Jan Laurens de Hoon wrote:
Arnd's comment raises the question of how to try out or contribute to Numeric3 if the code base is changing from day to day. It may be a good idea to set up some division of labor, so we can contribute to Numeric3 without getting in each other's way. For example, I'd be interested in working on setup.py and putting different parts of Numeric3/scipy_base together.
Michiel, you are free to work on setup.py all you want :-)
Putting the parts of scipy_base together is a good idea. Exactly how to structure this is going to require some thought and need to be coordinated with current scipy.
I want a package that is as easy to install as current Numeric (so the default will have something like lapack_lite). But, this should not handicap nor ignore a speed-conscious user who wants to install ATLAS or take advantage of vendor-supplied libraries.
There should be a way to replace functionality that is clean and does not require editing setup.py files.
Anybody with good ideas about how to do this well is welcome to speak up. Perhaps, the easiest thing to do is to keep the basic Numeric structure (with C-based easy-to-install additions) and call it scipylite (with backwards compatibility provided for Numeric, LinearAlgebra, RandomArray, and MLab names). This also installs the namespace scipy which has a little intelligence in it to determine if you have altas and fortran capabilities installed or not.
Then, provide a scipyatlas package that can be installed to take advantage of atlas and vendor-supplied lapack/blas.
Then, a scipyfortran package that can be installed if you have a fortran compiler which provides the functionality provided by fortran libraries. So, there are three divisions here.
Hmm, the idea of introducing scipylite, scipyatlas, scipyfortran packages does not sound like a good idea. The usage of atlas or fortran blas/lapack or vendor based blas/lapack libraries is an implementation detail and should not be reflected in scipy_base package structure. This is because such an approach is not suitable for writing portable Numeric3 based applications or packages. For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of potential users for this package. I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here. Ideally, scipy_base should provide a complete interface to LAPACK routines, but not immidiately, of course. Now, depending on the availability of compilers and resources in a particular computer, the following would happen: 1) No Fortran compiler, no lapack libraries in the system, only C compiler is available --- f2c generated lite-lapack C sources are used to build lapack extension module; wrappers to lapack routines, for which there are no f2c generated sources, are disabled by f2py `only:` feature. lite-lapack C sources come with scipy_base sources. 2) No Fortran compiler, system has lapack libraries (atlas or Accelerate or vecLib), C compiler is available --- system lapack library will be used and a complete lapack extension module can be built. 3) Fortran and C compiler are available, no lapack libraries in the system --- Fortran lite-lapack sources are used to build lapack extension module; lite-lapack Fortran sources come with scipy_base sources. Similar to the case (1), some wrappers are disabled. 4-..) other combinations are possible and users can choose their favorite approach. The availability of system resources can be checked using scipy_distutils.system_info.get_info. Checking the availability of Fortran compiler should be done in a configuration step and only when an user specifically asks for it, by default we should assume that Fortran compiler is not available. The same should apply also to atlas/lapack/blas libraries, by default f2c generated lite-lapack C sources will be used. In this way users that only need Numeric3 array capabilities will avoid all possible troubles that may show up when using all possible resources for speed on an arbitrary computer. Btw, I would suggest using `scipy <dot> <package>` instead of `scipy <package>` or `scipy <underscore> <package>` for naming packages. Pearu
For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of >potential users for this package.
While I'm not in love with my suggestion and would prefer to see better ones put forward, wouldn't any system that uses routines not available unless you have a fortran-compiled package installed be a problem? I was just proposing not "hiding" this from the developer but making it explicit. What do you propose to do for those situations? I was just proposing putting them in a separate hierarchy so the developer is aware he is using something that requires fortran. I actually think that it's somewhat of a non-issue myself, and feel that people who don't have fortran compilers will look for binaries anyway.
I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here.
Ideally, scipy_base should provide a complete interface to LAPACK routines, but not immidiately, of course. Now, depending on the availability of compilers and resources in a particular computer, the following would happen: 1) No Fortran compiler, no lapack libraries in the system, only C compiler is available --- f2c generated lite-lapack C sources are used to build lapack extension module; wrappers to lapack routines, for which there are no f2c generated sources, are disabled by f2py `only:` feature. lite-lapack C sources come with scipy_base sources. 2) No Fortran compiler, system has lapack libraries (atlas or Accelerate or vecLib), C compiler is available --- system lapack library will be used and a complete lapack extension module can be built. 3) Fortran and C compiler are available, no lapack libraries in the system --- Fortran lite-lapack sources are used to build lapack extension module; lite-lapack Fortran sources come with scipy_base sources. Similar to the case (1), some wrappers are disabled. 4-..) other combinations are possible and users can choose their favorite approach.
Great, Sounds like Pearu has some good ideas here. I nominate Pearu to take the lead here. Michiel sounds like he? wants to keep the Numeric, RandomArray, LinearAlgebra naming conventions forever. I want them to be more coordinated like scipy is doing with scipy.linalg scipy.stats and scipy_base ( I agree scipy.base is better). What are the opinions of others on this point. Of course the names Numeric, RandomArray, and LinearAlgebra will still work, but I think they should be deprecated in favor of a better overall design for numerical packages. What do others think?
On Fri, 25 Mar 2005, Travis Oliphant wrote:
For example, if a developer uses scipyfortran package in a package, it immidiately reduces the number of >potential users for this package.
While I'm not in love with my suggestion and would prefer to see better ones put forward, wouldn't any system that uses routines not available unless you have a fortran-compiled package installed be a problem? I was just proposing not "hiding" this from the developer but making it explicit.
What do you propose to do for those situations? I was just proposing putting them in a separate hierarchy so the developer is aware he is using something that requires fortran. I actually think that it's somewhat of a non-issue myself, and feel that people who don't have fortran compilers will look for binaries anyway.
Such an situation can be avoided if a package is extended with new wrappers parallel for all backend cases. For example, when adding a new interface to a lapack routine then to the scipy_base sources must be added both Fortran and f2c versions of the corresponding routine. Pearu
Pearu Peterson wrote:
I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here.
Have you tried integrating scipy_distutils with Python's distutils? My guess is that Python's distutils can benefit from what is in scipy_distutils, particularly the parts dealing with C compilers. A clean integration will also prevent duplicated code, avoids Pearu having to keep scipy_distutils up to date with Python's distutils, and will enlarge the number of potential users. Having two distutils packages seems to be too much of a good thing. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
On Fri, 25 Mar 2005, Michiel Jan Laurens de Hoon wrote:
Pearu Peterson wrote:
I got an impression from earlier threads that scipy_distutils will be included to scipy_base. So, I am proposing to use scipy_distutils tools and our scipy experience for dealing with this issue, scipy.lib.lapack would be a good working prototype here.
Have you tried integrating scipy_distutils with Python's distutils? My guess is that Python's distutils can benefit from what is in scipy_distutils, particularly the parts dealing with C compilers. A clean integration will also prevent duplicated code, avoids Pearu having to keep scipy_distutils up to date with Python's distutils, and will enlarge the number of potential users. Having two distutils packages seems to be too much of a good thing.
No, I have not. Though a year or so ago there was a discussion about this in distutils list, mainly for adding Fortran compiler support to distutils. At the time I didn't have resources to push scipy_distutils features to distutils and even less so for now. So, one can think that scipy_distutils is an extension to distutils, though it also includes few bug fixes for older distutils. On the other hand, since Scipy supports Python starting at 2.2 then it cannot relay much on new features added to distutils of later Python versions. Instead, if these features happen to be useful for Scipy then they are backported for Python 2.2 through implementing them in scipy_distutils. "Luckily", there are not much such features as scipy_distutils has evolved with new very useful features much quicker than distutils. But, for Numeric3, scipy.distutils would be a perfect place to clean up scipy_distutils a bit, e.g. removing some obsolete features and assuming that Numeric3 will support Python 2.3 and up. Based on that, integrating scipy_distutils features to standard distutils can be made less pain if someone decides to do that. Pearu
Pearu Peterson wrote:
Michiel Jan Laurens de Hoon wrote:
Have you tried integrating scipy_distutils with Python's distutils? My guess is that Python's distutils can benefit from what is in scipy_distutils, particularly the parts dealing with C compilers. A clean integration will also prevent duplicated code, avoids Pearu having to keep scipy_distutils up to date with Python's distutils, and will enlarge the number of potential users. Having two distutils packages seems to be too much of a good thing.
No, I have not. Though a year or so ago there was a discussion about this in distutils list, mainly for adding Fortran compiler support to distutils. At the time I didn't have resources to push scipy_distutils features to distutils and even less so for now. So, one can think that scipy_distutils is an extension to distutils, though it also includes few bug fixes for older distutils.
Having a separate scipy_distutils that fixes some bugs in Python's distutils is a design mistake in SciPy that we should not repeat in Numeric3. Not that I don't think the code in scipy_distutils is not useful -- I think it would be very useful. But the fact that it is not integrated with the existing Python distutils makes me wonder if this package really has been thought out that well. As far as I can tell, scipy_distutils now fulfills four functions: 1) Bug fixes for Python's distutils for older Python versions. As Numeric3 will require Python 2.3 or up, these are no longer relevant. 2) Bug fixes for current Python's distutils. These should be integrated with Python's distutils. Writing your own package instead of contributing to Python gives you bad karma. 3) Fortran support. Very useful, and I'd like to see them in Python's distutils. Another option would be to put this in SciPy.fortran or something similar. But since Python's distutils already has a language= option for C++ and Objective-C, the cleanest way would be to add this to Python's distutils and enable language="fortran". 4) Stuff particular to SciPy, for example finding Atlas/Lapack/Blas libraries. These we can decide on a case-by-case basis if it's useful for Numeric3. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Michiel Jan Laurens de Hoon wrote:
Pearu Peterson wrote:
Michiel Jan Laurens de Hoon wrote: ..... Not that I don't think the code in scipy_distutils is not useful -- I think it would be very useful.
One negation too many in this sentence -- sorry. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
On Sun, 27 Mar 2005, Michiel Jan Laurens de Hoon wrote:
Having a separate scipy_distutils that fixes some bugs in Python's distutils is a design mistake in SciPy that we should not repeat in Numeric3. Not that I don't think the code in scipy_distutils is not useful -- I think it would be very useful. But the fact that it is not integrated with the existing Python distutils makes me wonder if this package really has been thought out that well.
I don't think that part of scipy_distutils design was to fix Python's distutils bugs. As we found a bug, its fix was added to scipy_distutils as well as reported to distutils bug tracker. The main reason for adding bug fixes to scipy_distutils was to continue the work with scipy instead of waiting for the next distutils release (i.e. Python release), nor we could expect that SciPy users would use CVS version of Python's distutils. Also, SciPy was meant to support Python 2.1 and up, so the bug fixes remained relevant even when the bugs were fixed in Python 2.2 or 2.3 distutils. So much of history..
As far as I can tell, scipy_distutils now fulfills four functions: 1) Bug fixes for Python's distutils for older Python versions. As Numeric3 will require Python 2.3 or up, these are no longer relevant. 2) Bug fixes for current Python's distutils. These should be integrated with Python's distutils. Writing your own package instead of contributing to Python gives you bad karma. 3) Fortran support. Very useful, and I'd like to see them in Python's distutils. Another option would be to put this in SciPy.fortran or something similar. But since Python's distutils already has a language= option for C++ and Objective-C, the cleanest way would be to add this to Python's distutils and enable language="fortran". 4) Stuff particular to SciPy, for example finding Atlas/Lapack/Blas libraries. These we can decide on a case-by-case basis if it's useful for Numeric3.
Plus I would add the scipy_distutils ability to build sources on-fly feature (build_src command). That's a very fundamental feature useful whenever swig or f2py is used, or when building sources from templates or dynamically during a build process. Btw, I have started scipy_core clean up. The plan is to create the following package tree under Numeric3 source tree: scipy.distutils - contains cpuinfo, exec_command, system_info, etc scipy.distutils.fcompiler - contains Fortran compiler support scipy.distutils.command - contains build_src and config_compiler commands plus few enhancements to build_ext, build_clib, etc commands scipy.base - useful modules from scipy_base scipy.testing - enhancements to unittest module, actually current scipy_test contains one useful module (testing.py) that could also go under scipy.base and so getting rid of scipy.testing scipy.weave - scipy.f2py - not sure yet how to incorporate f2py2e or weave sources here. As a first instance people are assumed to download them to Numeric3/scipy/ directory but in future their sources could be added to Numeric3 repository. For Numeric3 f2py and weave are optional. scipy.lib.lapack - wrappers to Atlas/Lapack libraries, by default f2c generated wrappers are used as in current Numeric. For backwards compatibility, there will be Packages/{FFT,MA,RNG,dotblas+packages from numarray}/ and Lib/{LinearAlgebra,..}.py under Numeric3 that will use modules from scipy. Pearu
Plus I would add the scipy_distutils ability to build sources on-fly feature (build_src command). That's a very fundamental feature useful whenever swig or f2py is used, or when building sources from templates or dynamically during a build process.
I'd like to use this feature in Numeric3 (which has code-generation).
Btw, I have started scipy_core clean up. The plan is to create the following package tree under Numeric3 source tree:
This is great news. I'm thrilled to have Pearu's help in doing this. He understands a lot of these issues very well. I'm sure he will be open to suggestions.
scipy.distutils - contains cpuinfo, exec_command, system_info, etc scipy.distutils.fcompiler - contains Fortran compiler support scipy.distutils.command - contains build_src and config_compiler commands plus few enhancements to build_ext, build_clib, etc commands
scipy.base - useful modules from scipy_base scipy.testing - enhancements to unittest module, actually current scipy_test contains one useful module (testing.py) that could also go under scipy.base and so getting rid of scipy.testing scipy.weave - scipy.f2py - not sure yet how to incorporate f2py2e or weave sources here. As a first instance people are assumed to download them to Numeric3/scipy/ directory but in future their sources could be added to Numeric3 repository. For Numeric3 f2py and weave are optional. scipy.lib.lapack - wrappers to Atlas/Lapack libraries, by default f2c generated wrappers are used as in current Numeric.
For backwards compatibility, there will be Packages/{FFT,MA,RNG,dotblas+packages from numarray}/ and Lib/{LinearAlgebra,..}.py under Numeric3 that will use modules from scipy.
This looks like a good break down. Where will the ndarray object and the ufunc code go in this breakdown? In scipy.base? -Travis
Pearu Peterson wrote:
Btw, I have started scipy_core clean up. The plan is to create the following package tree under Numeric3 source tree: ...
For backwards compatibility, there will be Packages/{FFT,MA,RNG,dotblas+packages from numarray}/ and Lib/{LinearAlgebra,..}.py under Numeric3 that will use modules from scipy.
Just for clarification: Is this scipy_core or Numeric3 that you're working on? Or are they the same? --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
On Mon, 28 Mar 2005, Michiel Jan Laurens de Hoon wrote:
Pearu Peterson wrote:
Btw, I have started scipy_core clean up. The plan is to create the following package tree under Numeric3 source tree: ...
For backwards compatibility, there will be Packages/{FFT,MA,RNG,dotblas+packages from numarray}/ and Lib/{LinearAlgebra,..}.py under Numeric3 that will use modules from scipy.
Just for clarification: Is this scipy_core or Numeric3 that you're working on? Or are they the same?
The idea was merge tools from scipy_core (that basically contains scipy_distutils and scipy_base) to Numeric3. The features of scipy_distutils have been stated in previous messages, some of these features will be used to build Numeric3. scipy_base contains enhancements to Numeric (now to be natural part of Numeric3) plus few useful python modules. Which scipy_core modules exactly should be included to Numeric3 or left out of it, depends on how crusial are they for building/maintaining Numeric3 and whether they are useful in general for Numeric3 users. This is completely open for discussion. No part of scipy_core should be blindly copied to Numeric3 project. Pearu
Pearu Peterson wrote:
The idea was merge tools from scipy_core (that basically contains scipy_distutils and scipy_base) to Numeric3. The features of scipy_distutils have been stated in previous messages, some of these features will be used to build Numeric3. scipy_base contains enhancements to Numeric (now to be natural part of Numeric3) plus few useful python modules. Which scipy_core modules exactly should be included to Numeric3 or left out of it, depends on how crusial are they for building/maintaining Numeric3 and whether they are useful in general for Numeric3 users. This is completely open for discussion. No part of scipy_core should be blindly copied to Numeric3 project.
Sounds good to me. Thanks, Pearu. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
Just for clarification: Is this scipy_core or Numeric3 that you're working on? Or are they the same?
The idea was merge tools from scipy_core (that basically contains scipy_distutils and scipy_base) to Numeric3. The features of scipy_distutils have been stated in previous messages, some of these features will be used to build Numeric3. scipy_base contains enhancements to Numeric (now to be natural part of Numeric3) plus few useful python modules. Which scipy_core modules exactly should be included to Numeric3 or left out of it, depends on how crusial are they for building/maintaining Numeric3 and whether they are useful in general for Numeric3 users. This is completely open for discussion. No part of scipy_core should be blindly copied to Numeric3 project.
My understanding is that scipy_core and Numeric3 are the same thing. I'm using the terminology Numeric3 in emails to avoid confusion, but I would rather see one package emerge from this like scipy_core. I would prefer not to have a "Numeric3" package and a separate "scipy_core" package, unless there is a good reason to have two packages. -Travis
On Mon, 28 Mar 2005, Travis Oliphant wrote:
Just for clarification: Is this scipy_core or Numeric3 that you're working on? Or are they the same?
The idea was merge tools from scipy_core (that basically contains scipy_distutils and scipy_base) to Numeric3. The features of scipy_distutils have been stated in previous messages, some of these features will be used to build Numeric3. scipy_base contains enhancements to Numeric (now to be natural part of Numeric3) plus few useful python modules. Which scipy_core modules exactly should be included to Numeric3 or left out of it, depends on how crusial are they for building/maintaining Numeric3 and whether they are useful in general for Numeric3 users. This is completely open for discussion. No part of scipy_core should be blindly copied to Numeric3 project.
My understanding is that scipy_core and Numeric3 are the same thing. I'm using the terminology Numeric3 in emails to avoid confusion, but I would rather see one package emerge from this like scipy_core. I would prefer not to have a "Numeric3" package and a separate "scipy_core" package, unless there is a good reason to have two packages.
In that case ndarray object and ufunc codes should go under scipy.base. We can postpone this move until scipy.distutils is ready. And if I understand you correctly then from scipy.base import * will replace from Numeric import * or from numarray import * roughly speaking. Pearu
Pearu Peterson wrote:
from scipy.base import *
will replace
from Numeric import *
or
from numarray import *
roughly speaking.
Pearu
This is exactly what I would like to see. We will need, however, to provide that import Numeric and friends still works for backward compatibility, but it should be deprecated. Best, -Travis
Travis Oliphant wrote:
My understanding is that scipy_core and Numeric3 are the same thing. I'm using the terminology Numeric3 in emails to avoid confusion, but I would rather see one package emerge from this like scipy_core. I would prefer not to have a "Numeric3" package and a separate "scipy_core" package, unless there is a good reason to have two packages.
Right now, I think it's probably better to call it it scipy_core instead of Numeric3, since we'll be doing
from scipy.base import *
instead of
from Numeric import *
--Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon
On 27.03.2005, at 07:27, Michiel Jan Laurens de Hoon wrote:
3) Fortran support. Very useful, and I'd like to see them in Python's distutils. Another option would be to put this in SciPy.fortran or something similar. But since Python's distutils already has a language= option for C++ and Objective-C, the cleanest way would be to add this to Python's distutils and enable language="fortran".
I agree in principle, but I wonder how stable the Fortran support in SciPy distutils is. If it contains compiler-specific data, then it might not be a good idea to restrict modifications and additions to new Python releases. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------
konrad.hinsen@laposte.net wrote:
On 27.03.2005, at 07:27, Michiel Jan Laurens de Hoon wrote:
3) Fortran support. Very useful, and I'd like to see them in Python's distutils. Another option would be to put this in SciPy.fortran or something similar. But since Python's distutils already has a language= option for C++ and Objective-C, the cleanest way would be to add this to Python's distutils and enable language="fortran".
I agree in principle, but I wonder how stable the Fortran support in SciPy distutils is. If it contains compiler-specific data, then it might not be a good idea to restrict modifications and additions to new Python releases.
Case in point: Pearu just added g95 support last week. -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
participants (7)
-
Arnd Baecker -
Colin J. Williams -
konrad.hinsen＠laposte.net -
Michiel Jan Laurens de Hoon -
Pearu Peterson -
Robert Kern -
Travis Oliphant