On Mar 9, 2005, at 9:32, Michiel Jan Laurens de Hoon wrote:

...

It would seem that while the scipy conference demonstrates a continuing and even increasing use of Python for scientific computing, not as many of these users are scipy devotees. Why? I think the answers come down to a few issues which I will attempt to answer with proposals. 1) Plotting While plotting is important, I don't think that SciPy needs to offer

Travis Oliphant wrote: plotting capabilities in order to become successful. Numerical Python doesn't include plotting, and it's

... Thanks for your three comments, they reflect exactly my views as well, so I'll just add a "+1" to them. There is only one aspect I would like to add: predictibility of development. Python has become my #1 tool in my everyday research over the last years. I haven't done any scientific computation for at least five years that did not involve some Python code. Which means that I am very much dependent on Python and some Python packages. Moreover, I publish computational methods that I develop in the form of Python code that is used by a community large enough to make support an important consideration. There are only two kinds of computational tools on which I can accept being dependent: those that are supported by a sufficiently big and stable community that I don't need to worry about their disappearence or sudden mutation into something different, and those small enough that I can maintain them in usable state myself if necessary. Python is in the first category, Numeric in the second. SciPy is not in either one. The proposed division of SciPy into separately installable maintainable subpackages could make a big difference there. The core could actually be both easily maintainable and supported by a big enough community. So I am all for it, and I expect to contribute to such a loser package collection as well. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire Léon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ---------------------------------------------------------------------

Hi All

I think the proposal is: development effort is a function of community size,

Undeniably true!

and community size is a function of convenience as well as functionality.

This is only partly true. I think the main barriers to more people using scipy are... 1. Not that many people actually know about it 2. People aren't easily convinced to change from what they were taught to use as an under-graduate (e.g. Matlab, IDL, Mathematica) As it stands, I don't think scipy is particularly inconvenient to install or use. On the two suggested improvements: I think incorporating matplotlib is an excellent idea. But I think the second suggestion of separating Scipy into independent packages will prove to be counter-productive. It might put people off even before they start, because instead of installing one package, they have a bewildering choice of many. And it could prove to be annoying to people using scipy who want to share or distribute code, with the requirement that both parties have scipy becoming a requirement that both parties have a specific combination of scipy packages. Also, another reason why there might be a lack of developers is that there a people like me who find that scipy and matplotlib already do everything that they need. Which is good right? Cheers, Cory.

This seems right to me.

Cheers, Alan Isaac

_______________________________________________ SciPy-user mailing list SciPy-user@scipy.net http://www.scipy.net/mailman/listinfo/scipy-user -- )))))))))))))))))))))))))))))))))))))))))))) Cory Davis Meteorology School of GeoSciences University of Edinburgh King's Buildings EDINBURGH EH9 3JZ

ph: +44(0)131 6505092 fax +44(0)131 6505780 cdavis@staffmail.ed.ac.uk cory@met.ed.ac.uk http://www.geos.ed.ac.uk/contacts/homes/cdavis ))))))))))))))))))))))))))))))))))))))))))))

On 09.03.2005, at 17:52, Prabhu Ramachandran wrote:

To put it bluntly, I don't believe that someone who can't install scipy today is really capable of contributing code to scipy. I

True, but not quite to the point. I can install SciPy, but given that most of my code is written with the ultimate goal of being published and used by people with less technical experience, I need to take those people into account when choosing packages to build on.

seriously doubt claims that scipy is scary or hard to install today.

I get support questions from people who are not aware that they need root permissions to do "python setup.py install" on a standard Linux system. On that scale of expertise, scipy *is* scary. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------

On Mar 9, 2005, at 12:33 PM, Stephen Walton wrote:

...

2) Installation problems -- I'm not completely clear on what the "installation problems" really are.

scipy and matplotlib are both very easy to install. Using ATLAS is the biggest pain, as Travis says, and one can do without it. Now that a simple 'scipy setup.py bdist_rpm' seems to work reliably, I for one am happy.

On Mac OS X, using ATLAS should be pretty trivial because the OS already ships with an optimized implementation! The patch I created for Numeric was very short, and I'm pretty sure it's on the trunk (though last I packaged it, I had to make a trivial fix or two, which I reported on sourceforge). I haven't delved into SciPy's source in a really long time, so I'm not sure where changes would need to be made, but I think someone else should be fine to look at Numeric's setup.py and do what needs to be done to SciPy. FYI, matplotlib, the optimized Numeric, and several other Mac OS X packages are available in binary form here: http://pythonmac.org/packages/

I think splitting scipy up into multiple subpackages isn't such a good idea. Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment. Any large package is going to include a lot of stuff most people don't need, but like a NY Times ad used to say, "You might not read it all, but isn't it nice to know it's all there?"

I also think that a monolithic package is a pretty good idea until it begins to cause problems with the release cycle. Twisted had this problem at 1.3, and went through a major refactoring between then and 2.0 (which is almost out the door). Though Twisted 2.0 is technically many different packages, they still plan on maintaining a "sumo" package that includes all of the Twisted components, plus zope.interface (the only required dependency). There are still several optional dependencies not included, though (such as PyCrypto). SciPy could go this route, and simply market the "sumo" package to anyone who doesn't already know what they're doing. An experienced SciPy user may want to upgrade one particular component of SciPy as early as possible, but leave the rest be, for example. -bob

Hey Travis, It sounds like the Berkeley meeting went well. I am glad that the Numeric3 project is going well and looks like it has a good chance to unify the Numeric/Numarray communities. I really appreciate you putting in so much effort intto its implementation. I also appreciate all the work by Perry, Todd, and the others at StSci have done building Numarray. We've all learned a ton from it. Most of the plans sound right to me (several questions/comments below). Much of SciPy has been structured in this way already, but we really have never worked to make the core useful as a stand alone package. Supporting lite and full versions of fft, linalg, and stats sounds potentially painful, but also worthwhile given the circumstances. Now: 1. How much of stats do we loose from removing fortran dependencies? 2. I do question whether weave really be in this core? I think it was in scipy_core before because it was needed to build some of scipy. 3. Now that I think about it, I also wonder if f2py should really be there -- especially since we are explicitly removing any fortran dependencies from the core. 4. I think keeping scipy a algorithms library and leaving plotting to other libraries is a good plan. At one point, the setup_xplt.py file was more than 1000 lines. It is much cleaner now, but dealing with X11, etc. does take maintenance work. Removing these libraries from scipy would decrease the maintenance effort and leave the plotting to matplotlib, chaco, and others. 5. I think having all the generic algorithm packages (signal, ga, stats, etc. -- basically all the packages that are there now) under the scipy namespace is a good idea. It prevents worry about colliding with other peoples packages. However, I think domain specific libraries (such as astropy) should be in their own namespace and shouldn't be in scipy. thanks, eric Travis Oliphant wrote:

I had a lengthy discussion with Eric today and clarified some things in my mind about the future directions of scipy. The following is basically what we have decided. We are still interested in input so don't think the issues are closed, but I'm just giving people an idea of my (and Eric's as far as I understand it) thinking on scipy.

1) There will be a scipy_core package which will be essentially what Numeric has always been (plus a few easy to install extras already in current scipy_core). It will likely contain the functionality of (the names and placements will be similar to current scipy_core). Numeric3 (actually called ndarray or narray or numstar or numerix or something....) fft (based on c-only code -- no fortran dependency) linalg (a lite version -- no fortran or ATLAS dependency) stats (a lite version --- no fortran dependency) special (only c-code --- no fortran dependency) weave f2py? (still need to ask Pearu about this) scipy_distutils and testing matrix and polynomial classes

...others...?

We will push to make this an easy-to-install effective replacement for Numeric and hopefully for numarray users as well. Therefore community input and assistance will be particularly important.

2) The rest of scipy will be a package (or a series of packages) of algorithms. We will not try to do plotting as part of scipy. The current plotting in scipy will be supported for a time, but users will be weaned off to other packages: matplotlib, pygist (for xplt -- and I will work to get any improvements for xplt into pygist itself), gnuplot, etc.

3) Having everything under a scipy namespace is not necessary, nor worth worrying about at this point.

My scipy-related focus over the next 5-6 months will be to get scipy_core to the point that most can agree it effectively replaces the basic tools of Numeric and numarray.

-Travis

_______________________________________________ SciPy-user mailing list SciPy-user@scipy.net http://www.scipy.net/mailman/listinfo/scipy-user

On 10.03.2005, at 09:49, Pearu Peterson wrote:

f2py is not a fortran-only tool. In scipy it has been used to wrap also C codes (fft, atlas) and imho f2py should be used more so whenever possible.

Good to know. I never looked at f2py because I don't use Fortran any more.

Hmm, what would be the default underlying fft library here? Currently in scipy it is Fortran fftpack. And when fftw is available, it is used instead.

How about an f2c version of FFTPACK? Plus keeping the option of using fftw if installed, of course.

Again, what would be the underlying linear algebra library here? Numeric uses f2c version of lite lapack library. Shall we do the same but wrapping the c codes with f2py rather than by hand? f2c might be useful

I like the idea of the f2c versions because they can easily be replaced by the original Fortran code for more speed. It might even be a good to have scipy_core include the Fortran version as well and use it optionally during installation. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------

On Sun, 13 Mar 2005, Michiel Jan Laurens de Hoon wrote:

Pearu Peterson wrote:

...

...

Travis Oliphant wrote:

...

1) There will be a scipy_core package which will be essentially what Numeric has always been (plus a few easy to install extras already in current scipy_core). ... linalg (a lite version -- no fortran or ATLAS dependency)

Again, what would be the underlying linear algebra library here? Numeric uses f2c version of lite lapack library. Shall we do the same but wrapping the c codes with f2py rather than by hand? f2c might be useful also in other cases to reduce fortran dependency, but only when it is critical to ease the scipy_core installation.

If I understand Travis correctly, the idea is to use Numeric as the basis for scipy_core, allowing current Numerical Python users to switch to scipy_core with a minimum of trouble. So why not use Numeric's lite lapack library directly? What is the advantage of repeating the c code wrapping (by f2py or by hand)?

First, I wouldn't repeat wrapping c codes by hand. But using f2py wrappers has the following advantages: (i) maintaining the wrappers is easier (as the wrappers are generated) (ii) one can easily link linalg_lite against optimized lapack. This is certainly possible with current Numeric but for a smaller set of Fortran compilers than when using f2py generated wrappers (for example, if a compiler produces uppercased symbol names then Numeric wrappers won't work) (iii) scipy provides wrappers to a larger set of lapack subroutines than Numeric, and with f2py it is easier and less errorprone to add new wrappers to lapack functions than wrapping them by hand, i.e. extending f2py generated linalg_lite is much easier than extending the current Numeric lapack_lite. (iv) and finally, f2py generated wrappers tend to be more efficient than Numeric hand coded wrappers. Here are some benchmark results comparing scipy and Numeric linalg functions: Finding matrix determinant ================================== | contiguous | non-contiguous ---------------------------------------------- size | scipy | Numeric | scipy | Numeric 20 | 0.16 | 0.22 | 0.17 | 0.26 (secs for 2000 calls) 100 | 0.29 | 0.41 | 0.28 | 0.56 (secs for 300 calls) 500 | 0.31 | 0.36 | 0.33 | 0.45 (secs for 4 calls) Finding matrix inverse ================================== | contiguous | non-contiguous ---------------------------------------------- size | scipy | Numeric | scipy | Numeric 20 | 0.28 | 0.33 | 0.27 | 0.37 (secs for 2000 calls) 100 | 0.64 | 1.06 | 0.64 | 1.24 (secs for 300 calls) 500 | 0.83 | 1.10 | 0.84 | 1.18 (secs for 4 calls) Solving system of linear equations ================================== | contiguous | non-contiguous ---------------------------------------------- size | scipy | Numeric | scipy | Numeric 20 | 0.26 | 0.18 | 0.26 | 0.21 (secs for 2000 calls) 100 | 0.31 | 0.35 | 0.31 | 0.52 (secs for 300 calls) 500 | 0.33 | 0.34 | 0.35 | 0.41 (secs for 4 calls) Remark: both scipy and Numeric are linked agaist the same ATLAS/Lapack library. Pearu

On Mar 9, 2005, at 11:41 PM, eric jones wrote:

2. I do question whether weave really be in this core? I think it was in scipy_core before because it was needed to build some of scipy. 3. Now that I think about it, I also wonder if f2py should really be there -- especially since we are explicitly removing any fortran dependencies from the core.

It would seem to me that so long as: 1) both these tools have very general usefulness (and I think they do), and 2) are not installation problems (I don't believe they are since they themselves don't require any compilation of Fortran, C++ or whatever--am I wrong on that?) That they are perfectly fine to go into the core. In fact, if they are used by any of the extra packages, they should be in the core to eliminate the extra step in the installation of those packages. Perry

Michiel Jan Laurens de Hoon wrote:

Perry Greenfield wrote:

...

On Mar 9, 2005, at 11:41 PM, eric jones wrote:

...

2. I do question whether weave really be in this core? I think it was in scipy_core before because it was needed to build some of scipy. 3. Now that I think about it, I also wonder if f2py should really be there -- especially since we are explicitly removing any fortran dependencies from the core.

It would seem to me that so long as:

1) both these tools have very general usefulness (and I think they do), and 2) are not installation problems (I don't believe they are since they themselves don't require any compilation of Fortran, C++ or whatever--am I wrong on that?)

That they are perfectly fine to go into the core. In fact, if they are used by any of the extra packages, they should be in the core to eliminate the extra step in the installation of those packages.

-0. 1) In der Beschraenkung zeigt sich der Meister. In other words, avoid software bloat. 2) f2py is a Fortran-Python interface generator, once the interface is created there is no need for the generator. 3) I'm sure f2py is useful, but I doubt that it has very general usefulness. There are lots of other useful Python packages, but we're not including them in scipy-core either. 4) f2py and weave don't fit in well with the rest of scipy-core, which is mainly standard numerical algorithms.

I'm of the opinion that f2py and weave should go into the core. 1) Neither one requires Fortran and both install very, very easily. 2) These packages are fairly small but provide huge utility --- inlining fortran or C code is an easy way to speed up Python. People who don't "need it" will never realize it's there 3) Building the rest of scipy will need at least f2py already installed and it would simplify the process. 4) Enthought packages (to be released in the future and of interest to scientists) rely on weave. Why not make that process easier with a single initial install. 5) It would encourage improvements of weave and f2py from the entire community. 6) The developers of f2py and weave are both scipy developers and so it would make sense for their code that forms a foundation for other work to go into scipy_core. -Travis

Travis Oliphant wrote:

I'm of the opinion that f2py and weave should go into the core. <(6 good points)>

The act of putting something into the core will encourage people to use it. My understanding of the idea of the core is that it is minimal set of packages that various developers can use as a basis for their domain specific stuff. One barrier to entry for people currently using the whole of SciPy is the ease of installation issue, and f2py and weave are easy to install, so that's not a problem. However, if I understand it correctly, neither weave nor f2py is the least bit useful without a compiler. If they are in the core, you are encouraging people to use them in their larger packages, which will then impose a dependency on compilers. This seems to me not to fit in with the purpose of the core, which is to be a SINGLE, robust, easy-to-install dependency that others can build on. I suggest that weave and f2py go into a "devel" or "high-performance" package instead. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Thu, 2005-03-10 at 19:54, Travis Oliphant wrote:

I'm of the opinion that f2py and weave should go into the core.

+1 ralf

On 16.03.2005, at 02:46, Fernando Perez wrote:

The reality is that, in scientific work, you are likely to run into this problem at a very early stage, much more so than for other kinds of python usage. For this reason, it is important that the basic toolset provides a clean solution from the start.

One can in fact argue that f2py, weave, and other tools (Pyrex comes to mind) are the logical extensions of Distutils, which is part of the Python core. As long as they can be installed without additional requirements (in particular requiring the compilers that they need to work), I don't mind having them in the core distribution, though I would still have them as logically separate packages (i.e. not scipy.core.f2py but scipy.f2py) . Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------

Can I put in a good word for Fortran? Not the language itself, but the available packages for it. I've always thought that one of the really good things about Scipy was the effort put into getting all those powerful, well tested, robust Fortran routines from Netlib inside Scipy. Without them, it seems to me that folks who just install the new scipy_base are going to re-invent a lot of wheels. Is it really that hard to install g77 on non-Linux platforms? Steve Walton

konrad.hinsen@laposte.net writes:

On Mar 10, 2005, at 18:33, Stephen Walton wrote:

...

Can I put in a good word for Fortran? Not the language itself, but the available packages for it. I've always thought that one of the really good things about Scipy was the effort put into getting all those powerful, well tested, robust Fortran routines from Netlib inside Scipy. Without them, it seems to me that folks who just install the new scipy_base are going to re-invent a lot of wheels.

Is it really that hard to install g77 on non-Linux platforms?

It takes some careful reading of the instructions, which in turn requires a good command of the English language, including some peculiar technical terms, and either some experience in software installation or a high intimidation threshold.

It also takes a significant amount of time and disk space.

Konrad.

I don't know about Windows, but on OS X it involves going to http://hpc.sourceforge.net/ and following the one paragraph of instructions. That could be even be simplified if an .pkg were made... In fact, it's so easy to make a .pkg with PackageMaker that I've done it :-) I've put a .pkg of g77 3.4 for OS X (using the above binaries) at http://arbutus.mcmaster.ca/dmc/osx/ [Warning: unsupported and lightly-tested. I'll email Gaurav Khanna about making packages of his other binaries.] It'll run, install into /usr/local/g77v3.4, and make a symlink at /usr/local/bin/g77 to the right binary. (To compile SciPy with this, I have to add -lcc_dynamic to the libraries to link with. I've got a patch which I'll submit to the SciPy bug tracker for that, soonish.) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca

Stephen Walton wrote:

Can I put in a good word for Fortran? Not the language itself, but the available packages for it. I've always thought that one of the really good things about Scipy was the effort put into getting all those powerful, well tested, robust Fortran routines from Netlib inside Scipy. Without them, it seems to me that folks who just install the new scipy_base are going to re-invent a lot of wheels.

Is it really that hard to install g77 on non-Linux platforms?

I agree that Netlib should be in SciPy. But why should Netlib be in scipy_base? If SciPy evolves into a website of scientific packages for python, I presume Netcdf will be in one of those packages, maybe even a package by itself. Such a package, together with a couple of binary installers for common platforms, will be appreciated by users and developers who need Netcdf. But if Netcdf is in scipy_base, you're effectively forcing most users to waste time on Fortran only to install something they don't need. In turn, those users will ask their developers for help if something goes wrong (or give up altogether). And those developers, also not willing to waste time on something they don't need, will tell their users to use Numerical Python instead of SciPy. --Michiel.

Paul Moore wrote:

Travis Oliphant <oliphant@ee.byu.edu> writes

...

2) Installation problems -- I'm not completely clear on what the "installation problems" really are.

While I am not a scientific user, I occasionally have a need for something like stats, linear algebra, or other such functions. I'm happy to install something (I'm using Python on Windows, so when I say "install", I mean "download and run a binary installer") but I'm a casual user, so I am not going to go to too much trouble. ... There's no way on Windows that I'd even consider building scipy from source - my need for it simply isn't sufficient to justify the cost.

As I say, this is from someone who is clearly not in the target audience of scipy, but maybe it is of use...

I think you perfectly described the experience of a typical Biopython user. So as far as I'm concerned, you're squarely in the target audience of SciPy, if it intends to replace Numeric. --michiel.