Future directions for SciPy in light of meeting at Berkeley
I wanted to send an update to this list regarding the meeting at Berkeley that I attended. A lot of good disscussions took place at the meeting that should stimulate larger feedback. Personally, I had far more to discuss before I had to leave, and so I hope that the discussions can continue. I was looking to try and understand why with an increasing number of Scientific users of Python, relatively few people actually seem to want to contribute to scipy, regularly, even becoming active developers. There are lots of people who seem to identify problems (though very often vague ones), but not many who seem able (either through time or interest contraints) to actually contribute to code, documentation, or infrastructure. Scipy is an Open source project and relies on the self-selection process of open source contributors. It would seem that while the scipy conference demonstrates a continuing and even increasing use of Python for scientific computing, not as many of these users are scipy devotees. Why? I think the answers come down to a few issues which I will attempt to answer with proposals. 1) Plotting -- scipy's plotting wasn't good enough (we knew that) and the promised solution (chaco) took too long to emerge as a simple replacement. While the elements were all there for chaco to work, very few people knew that and nobody stepped up to take chaco to the level that matplotlib, for example, has reached in terms of cross-gui applicability and user-interface usability. Proposal: Incorporate matplotlib as part of the scipy framework (replacing plt). Chaco is not there anymore and the other two plotting solutions could stay as backward compatible but not progressing solutions. I have not talked to John about this, though I would like to. I think if some other packaging issues are addressed we might be able to get John to agree. 2) Installation problems -- I'm not completely clear on what the "installation problems" really are. I hear people talk about them, but Pearu has made significant strides to improve installation, so I'm not sure what precise issues remain. Yes, installing ATLAS can be a pain, but scipy doesn't require it. Yes, fortran support can be a pain, but if you use g77 then it isn't a big deal. The reality, though, is that there is this perception of installation trouble and it must be based on something. Let's find out what it is. Please speak up users of the world!!!! Proposal (just an idea to start discussion): Subdivide scipy into several super packages that install cleanly but can also be installed separately. Implement a CPAN-or-yum-like repository and query system for installing scientific packages. Base package: scipy_core -- this super package should be easy to install (no Fortran) and should essentially be old Numeric. It was discussed at Berkeley, that very likely Numeric3 should just be included here. I think this package should also include plotting, weave, scipy_distutils, and even f2py. Some of these could live in dual namespaces (i.e. both weave and scipy.weave are available on install). scipy.traits scipy.weave (weave) scipy.plt (matplotlib) scipy.numeric (Numeric3 -- uses atlas when installed later) scipy.f2py scipy.distutils scipy.fft scipy.linalg? (include something like lapack-lite for basic but slow functionality, installation of improved package replaces this with atlas usage) scipy.stats scipy.util (everything else currently in scipy_core) scipy.testing (testing facilities) Each of these should be a separate package installable and distributable separately (though there may be co-dependencies so that scipy.plt would have to be distributed with scipy. Libraries (each separately installable). scipy.lib -- there should be several sub-packages that could live under hear. This is simply raw code with basic wrappers (kind of like a /usr/lib) scipy.lib.lapack -- installation also updates narray and linalg (hooks to do that) scipy.lib.blas -- installation updates narray and linalg scipy.lib.quadpack etc... Extra sub-packages: named in a hierarchy to be determined and probably each dependent on a variety of scipy-sub-packages. I haven't fleshed this thing out yet as you can tell. I'm mainly talking publicly to spur discussion. The basic idea is that we should force ourselves to distribute scipy in separate packages. This would force us to implement a yum-or-CPAN-like package repository, so that we define the interface as to how an additional module could be developed by someone, even maintained separately (with a different license), and simply inserted into an intelligent point under the scipy infrastructure. It also allow installation/compilation issues to be handled on a more per-module basis so that difficult ones could be noted. I think this would also help interested people get some of the enthought stuff put in to the scipy hierarchy as well. Thoughts and comments (and even half-working code) welcomed and encouraged... -Travis O.
Hi Travis,
"TO" == Travis Oliphant <oliphant@ee.byu.edu> writes:
TO> I was looking to try and understand why with an increasing TO> number of Scientific users of Python, relatively few people TO> actually seem to want to contribute to scipy, regularly, even TO> becoming active developers. There are lots of people who seem TO> to identify problems (though very often vague ones), but not TO> many who seem able (either through time or interest TO> contraints) to actually contribute to code, documentation, or TO> infrastructure. I think there are two issues here. 1. Finding developers. Unfortunately, I'm as clueless as anyone else. It looks to me that most folks who are capable of contributing are already occupied with other projects. The rest use scipy and are quite happy with it (except for the occasional problem). Others are either heavily invested in other solutions, or don't have the skill or time to contribute. I also think that there are a fair number of users who use scipy at some level or another but are quiet about it and don't have a chance to contribute. From what I can tell, the intersection of the set of people who possess good computing skills and also persue numerical work from Python is still a very small number compared to other fields. 2. Packaging issues. More on this later. [...] TO> I think the answers come down to a few issues which I will TO> attempt to answer with proposals. TO> 1) Plotting -- scipy's plotting wasn't good enough (we knew I am not sure what this has to do with scipy's utility? Do you mean to say that you'd like to have people starting to use scipy to plot things and then hope that they contribute back to scipy's numeric algorithms? If all they did was to use scipy for plotting, the only contributions would be towards plotting. If you only mean this as a convenience, then this seems like a packaging issue and not related to scipy. Plotting is one part of the puzzle. You don't seem to mention any deficiencies with respect to numerical algorithms. This seems to suggest that apart from things like packaging and docs, the numeric side is pretty solid. Let me take this to an extreme, if plotting be deemed a part of scipy's core then how about f2py? It is definitely core functionality. So why not make f2py part of scipy? How about g77, g95, and gcc. The only direction this looks to be headed is to make a SciPy OS (== Enthon?). I think we are mixing packaging along with other issues here. To make it clear, I am not against incorporating matplotlib in scipy. I just think that the argument for its inclusion does not seem clear to me. [...] TO> 2) Installation problems -- I'm not completely clear on what TO> the TO> "installation problems" really are. I hear people talk about [...] TO> Proposal (just an idea to start discussion): TO> Subdivide scipy into several super packages that install TO> cleanly but can also be installed separately. Implement a TO> CPAN-or-yum-like repository and query system for installing TO> scientific packages. What does this have to do with scipy per se? This is more like a user convenience issue. [scipy-sub-packages] TO> I haven't fleshed this thing out yet as you can tell. I'm TO> mainly talking publicly to spur discussion. The basic idea is TO> that we should force ourselves to distribute scipy in separate TO> packages. This would force us to implement a yum-or-CPAN-like TO> package repository, so that we define the interface as to how TO> an additional module could be developed by someone, even TO> maintained separately (with a different license), and simply TO> inserted into an intelligent point under the scipy TO> infrastructure. This is in general a good idea but one that goes far beyond scipy itself. Joe Cooper mentioned that he had ideas on how to really do this in a cross-platform way. Many of us eagerly await his solution. :) regards, prabhu
On Wed, 9 Mar 2005, Prabhu Ramachandran apparently wrote:
What does this have to do with scipy per se? This is more like a user convenience issue.
I think the proposal is: development effort is a function of community size, and community size is a function of convenience as well as functionality. This seems right to me. Cheers, Alan Isaac
Hi All
I think the proposal is: development effort is a function of community size,
Undeniably true!
and community size is a function of convenience as well as functionality.
This is only partly true. I think the main barriers to more people using scipy are... 1. Not that many people actually know about it 2. People aren't easily convinced to change from what they were taught to use as an under-graduate (e.g. Matlab, IDL, Mathematica) As it stands, I don't think scipy is particularly inconvenient to install or use. On the two suggested improvements: I think incorporating matplotlib is an excellent idea. But I think the second suggestion of separating Scipy into independent packages will prove to be counter-productive. It might put people off even before they start, because instead of installing one package, they have a bewildering choice of many. And it could prove to be annoying to people using scipy who want to share or distribute code, with the requirement that both parties have scipy becoming a requirement that both parties have a specific combination of scipy packages. Also, another reason why there might be a lack of developers is that there a people like me who find that scipy and matplotlib already do everything that they need. Which is good right? Cheers, Cory.
This seems right to me.
Cheers, Alan Isaac
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.net http://www.scipy.net/mailman/listinfo/scipy-user -- )))))))))))))))))))))))))))))))))))))))))))) Cory Davis Meteorology School of GeoSciences University of Edinburgh King's Buildings EDINBURGH EH9 3JZ
ph: +44(0)131 6505092 fax +44(0)131 6505780 cdavis@staffmail.ed.ac.uk cory@met.ed.ac.uk http://www.geos.ed.ac.uk/contacts/homes/cdavis ))))))))))))))))))))))))))))))))))))))))))))
"AI" == Alan G Isaac writes:
AI> On Wed, 9 Mar 2005, Prabhu Ramachandran apparently wrote: >> What does this have to do with scipy per se? This is more like >> a user convenience issue. AI> I think the proposal is: development effort is a function of AI> community size, and community size is a function of AI> convenience as well as functionality. To put it bluntly, I don't believe that someone who can't install scipy today is really capable of contributing code to scipy. I seriously doubt claims that scipy is scary or hard to install today. Therefore, the real problem does not appear to be convenience and IMHO neither is functionality the problem. My only point is this. I think Travis and Pearu have been doing a great job! I'd rather see them working on things like Numeric3 and core scipy functionality rather than spend time worrying about packaging, including other new packages and making things more comfortable for the user (especially when these things are already taken care of). Anyway, Joe's post about ASP's role is spot on! Thanks Joe. More on that thread. cheers, prabhu
I only have a little to contribute at this point:
Proposal: Incorporate matplotlib as part of the scipy framework (replacing plt).
While this is an admirable goal, I personally find scipy and matplotlib easy to install separately. The only difficulty (of course!) is the numarray/numeric split, so I have to be sure that I select numerix as Numeric in my .matplotlibrc file before typing 'ipython -pylab -p scipy', which actually works really well.
2) Installation problems -- I'm not completely clear on what the "installation problems" really are.
scipy and matplotlib are both very easy to install. Using ATLAS is the biggest pain, as Travis says, and one can do without it. Now that a simple 'scipy setup.py bdist_rpm' seems to work reliably, I for one am happy. I think splitting scipy up into multiple subpackages isn't such a good idea. Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment. Any large package is going to include a lot of stuff most people don't need, but like a NY Times ad used to say, "You might not read it all, but isn't it nice to know it's all there?" I can tell you why I'm not contributing much code to the effort at least in one recent instance. Since I'm still getting core dumps when I try to use optimize.leastsq with a defined Jacobian function, I dove into _minpackmodule.c and its associated routines last night. I'm at sea. I know enough Python to be dangerous, used LMDER from Fortran extensively while doing my Ph.D., and am pretty good at C, but am completely unfamiliar with the Python-C API. So I don't even know how to begin tracking the problem down. Finally, as I mentioned at SciPy04, our particular physics department is at an undergraduate institution (no Ph.D. program), so we mainly produce majors who stop at the B.S. or M.S. degree. Their job market seems to want MATLAB skills, not Python, at the moment, so that's what the faculty are learning and teaching to their students. Many of them/us simply don't have the time to learn Python on top of that. Though, when I showed some colleagues how trivial it was to trim some unwanted bits out of data files they had using Python, I think I converted them.
Stephen Walton wrote:
I think splitting scipy up into multiple subpackages isn't such a good idea. Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment. Any large package is going to include a lot of stuff most people don't need, but like a NY Times ad used to say, "You might not read it all, but isn't it nice to know it's all there?"
Debian has a nice package management system with a GUI (aptitude) to go on top of the basic apt. This keeps track of and enforces pakage dependencies. Colin W.
Colin J. Williams wrote:
Stephen Walton wrote:
Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment.
Debian has a nice package management system with a GUI (aptitude) to go on top of the basic apt. This keeps track of and enforces pakage dependencies.
So does yum, and CPAN itself for that matter, but that isn't really what I was raising. If I use cpan to install Perl::MyPackage::MyModule it doesn't show up when I do 'rpm -qa'. I can almost always say 'yum install perl-MyPackage-MyModule' though with my repository collection. Perhaps my main complaint is that the CPAN divisions are just too fine grained. Disk space is way cheaper than my time, and I'd rather get everything I need in one go.
On Mar 9, 2005, at 4:50 PM, Stephen Walton wrote:
Colin J. Williams wrote:
Stephen Walton wrote:
Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment.
Debian has a nice package management system with a GUI (aptitude) to go on top of the basic apt. This keeps track of and enforces pakage dependencies.
So does yum, and CPAN itself for that matter, but that isn't really what I was raising. If I use cpan to install Perl::MyPackage::MyModule it doesn't show up when I do 'rpm -qa'. I can almost always say 'yum install perl-MyPackage-MyModule' though with my repository collection. Perhaps my main complaint is that the CPAN divisions are just too fine grained. Disk space is way cheaper than my time, and I'd rather get everything I need in one go.
I also find that disk space is way cheaper than my time, however, building source packages (CPAN, DarwinPorts, etc.) can take *WAY* too long. Especially for CPAN, where it asks you questions all the goddamn time, so you have to babysit. That annoys me to no end. Personally, I think the focus should be on repositories for win32 and Mac OS X binary packages. Linux/BSD/etc. don't need yet another repository for packages, because they have systems like yum, apt, pkgsrc, etc. with an inherent distribution model. If any distribution effort is put into the *nix OS'es, it should probably be to develop bdist_* targets so that it can integrate with the OS, not to develop One Package Manager To Rule Them All. Not yet, anyway. The best policy I have found is to build everything statically, taking advantage of whatever is already on the OS (for example, the ATLAS capabilities in Apple's Accelerate framework), so you don't have to worry about non-Python dependencies. There are a couple exceptions, like SDL and wxWidgets, where it is better to go with the dynamically linkable version because you're going to have a bunch of things touching it. Python dependencies can be managed rather simply just by maintaining a list such as X *needs* Y, Z.. however, it becomes more complicated when you have X *wants* Y. Normally I just punt on this problem, and say that all wants are needs, so if you install a package you'll have everything it can do at the expense of downloading more stuff. -bob
On Mar 9, 2005, at 12:33 PM, Stephen Walton wrote:
2) Installation problems -- I'm not completely clear on what the "installation problems" really are.
scipy and matplotlib are both very easy to install. Using ATLAS is the biggest pain, as Travis says, and one can do without it. Now that a simple 'scipy setup.py bdist_rpm' seems to work reliably, I for one am happy.
On Mac OS X, using ATLAS should be pretty trivial because the OS already ships with an optimized implementation! The patch I created for Numeric was very short, and I'm pretty sure it's on the trunk (though last I packaged it, I had to make a trivial fix or two, which I reported on sourceforge). I haven't delved into SciPy's source in a really long time, so I'm not sure where changes would need to be made, but I think someone else should be fine to look at Numeric's setup.py and do what needs to be done to SciPy. FYI, matplotlib, the optimized Numeric, and several other Mac OS X packages are available in binary form here: http://pythonmac.org/packages/
I think splitting scipy up into multiple subpackages isn't such a good idea. Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment. Any large package is going to include a lot of stuff most people don't need, but like a NY Times ad used to say, "You might not read it all, but isn't it nice to know it's all there?"
I also think that a monolithic package is a pretty good idea until it begins to cause problems with the release cycle. Twisted had this problem at 1.3, and went through a major refactoring between then and 2.0 (which is almost out the door). Though Twisted 2.0 is technically many different packages, they still plan on maintaining a "sumo" package that includes all of the Twisted components, plus zope.interface (the only required dependency). There are still several optional dependencies not included, though (such as PyCrypto). SciPy could go this route, and simply market the "sumo" package to anyone who doesn't already know what they're doing. An experienced SciPy user may want to upgrade one particular component of SciPy as early as possible, but leave the rest be, for example. -bob
Bob Ippolito wrote:
On Mar 9, 2005, at 12:33 PM, Stephen Walton wrote:
2) Installation problems -- I'm not completely clear on what the "installation problems" really are.
scipy and matplotlib are both very easy to install. Using ATLAS is the biggest pain, as Travis says, and one can do without it. Now that a simple 'scipy setup.py bdist_rpm' seems to work reliably, I for one am happy.
On Mac OS X, using ATLAS should be pretty trivial because the OS already ships with an optimized implementation! The patch I created for Numeric was very short, and I'm pretty sure it's on the trunk (though last I packaged it, I had to make a trivial fix or two, which I reported on sourceforge). I haven't delved into SciPy's source in a really long time, so I'm not sure where changes would need to be made, but I think someone else should be fine to look at Numeric's setup.py and do what needs to be done to SciPy.
Scipy already works just fine with Accelerate. No patch is necessary. However, the ATLAS provided in the Accelerate framework is incomplete. Namely, it's missing the row-major versions of BLAS and the optimized LAPACK routines. Scipy does make good use of these *if* they are available, and the availability of these routines can be quite important (if you are doing heavy linear algebra, of course, if you Just Need It To Build, then Accelerate is the way to go).
FYI, matplotlib, the optimized Numeric, and several other Mac OS X packages are available in binary form here: http://pythonmac.org/packages/
I think splitting scipy up into multiple subpackages isn't such a good idea. Perhaps I'm in the minority, but I find CPAN counter-intuitive, hard to use, and hard to keep track of in an RPM-based environment. Any large package is going to include a lot of stuff most people don't need, but like a NY Times ad used to say, "You might not read it all, but isn't it nice to know it's all there?"
I also think that a monolithic package is a pretty good idea until it begins to cause problems with the release cycle. Twisted had this problem at 1.3, and went through a major refactoring between then and 2.0 (which is almost out the door). Though Twisted 2.0 is technically many different packages, they still plan on maintaining a "sumo" package that includes all of the Twisted components, plus zope.interface (the only required dependency). There are still several optional dependencies not included, though (such as PyCrypto).
SciPy could go this route, and simply market the "sumo" package to anyone who doesn't already know what they're doing. An experienced SciPy user may want to upgrade one particular component of SciPy as early as possible, but leave the rest be, for example.
The Twisted re-org is something I think we should look at. There seem to be reference to using zpkgtools to handle the split, but I can't find any of the data files that are referenced in the build scripts. Does anyone more familiar with the Twisted re-org care to chime in? -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
On Mar 9, 2005, at 8:30 PM, Robert Kern wrote:
Bob Ippolito wrote:
On Mar 9, 2005, at 12:33 PM, Stephen Walton wrote:
2) Installation problems -- I'm not completely clear on what the "installation problems" really are.
scipy and matplotlib are both very easy to install. Using ATLAS is the biggest pain, as Travis says, and one can do without it. Now that a simple 'scipy setup.py bdist_rpm' seems to work reliably, I for one am happy. On Mac OS X, using ATLAS should be pretty trivial because the OS already ships with an optimized implementation! The patch I created for Numeric was very short, and I'm pretty sure it's on the trunk (though last I packaged it, I had to make a trivial fix or two, which I reported on sourceforge). I haven't delved into SciPy's source in a really long time, so I'm not sure where changes would need to be made, but I think someone else should be fine to look at Numeric's setup.py and do what needs to be done to SciPy.
Scipy already works just fine with Accelerate. No patch is necessary. However, the ATLAS provided in the Accelerate framework is incomplete. Namely, it's missing the row-major versions of BLAS and the optimized LAPACK routines. Scipy does make good use of these *if* they are available, and the availability of these routines can be quite important (if you are doing heavy linear algebra, of course, if you Just Need It To Build, then Accelerate is the way to go).
You should file a bug at http://bugreport.apple.com/ if you have not already. There's still a chance for 10.4! -bob
I had a lengthy discussion with Eric today and clarified some things in my mind about the future directions of scipy. The following is basically what we have decided. We are still interested in input so don't think the issues are closed, but I'm just giving people an idea of my (and Eric's as far as I understand it) thinking on scipy. 1) There will be a scipy_core package which will be essentially what Numeric has always been (plus a few easy to install extras already in current scipy_core). It will likely contain the functionality of (the names and placements will be similar to current scipy_core). Numeric3 (actually called ndarray or narray or numstar or numerix or something....) fft (based on c-only code -- no fortran dependency) linalg (a lite version -- no fortran or ATLAS dependency) stats (a lite version --- no fortran dependency) special (only c-code --- no fortran dependency) weave f2py? (still need to ask Pearu about this) scipy_distutils and testing matrix and polynomial classes ...others...? We will push to make this an easy-to-install effective replacement for Numeric and hopefully for numarray users as well. Therefore community input and assistance will be particularly important. 2) The rest of scipy will be a package (or a series of packages) of algorithms. We will not try to do plotting as part of scipy. The current plotting in scipy will be supported for a time, but users will be weaned off to other packages: matplotlib, pygist (for xplt -- and I will work to get any improvements for xplt into pygist itself), gnuplot, etc. 3) Having everything under a scipy namespace is not necessary, nor worth worrying about at this point. My scipy-related focus over the next 5-6 months will be to get scipy_core to the point that most can agree it effectively replaces the basic tools of Numeric and numarray. -Travis
Hey Travis, It sounds like the Berkeley meeting went well. I am glad that the Numeric3 project is going well and looks like it has a good chance to unify the Numeric/Numarray communities. I really appreciate you putting in so much effort intto its implementation. I also appreciate all the work by Perry, Todd, and the others at StSci have done building Numarray. We've all learned a ton from it. Most of the plans sound right to me (several questions/comments below). Much of SciPy has been structured in this way already, but we really have never worked to make the core useful as a stand alone package. Supporting lite and full versions of fft, linalg, and stats sounds potentially painful, but also worthwhile given the circumstances. Now: 1. How much of stats do we loose from removing fortran dependencies? 2. I do question whether weave really be in this core? I think it was in scipy_core before because it was needed to build some of scipy. 3. Now that I think about it, I also wonder if f2py should really be there -- especially since we are explicitly removing any fortran dependencies from the core. 4. I think keeping scipy a algorithms library and leaving plotting to other libraries is a good plan. At one point, the setup_xplt.py file was more than 1000 lines. It is much cleaner now, but dealing with X11, etc. does take maintenance work. Removing these libraries from scipy would decrease the maintenance effort and leave the plotting to matplotlib, chaco, and others. 5. I think having all the generic algorithm packages (signal, ga, stats, etc. -- basically all the packages that are there now) under the scipy namespace is a good idea. It prevents worry about colliding with other peoples packages. However, I think domain specific libraries (such as astropy) should be in their own namespace and shouldn't be in scipy. thanks, eric Travis Oliphant wrote:
I had a lengthy discussion with Eric today and clarified some things in my mind about the future directions of scipy. The following is basically what we have decided. We are still interested in input so don't think the issues are closed, but I'm just giving people an idea of my (and Eric's as far as I understand it) thinking on scipy.
1) There will be a scipy_core package which will be essentially what Numeric has always been (plus a few easy to install extras already in current scipy_core). It will likely contain the functionality of (the names and placements will be similar to current scipy_core). Numeric3 (actually called ndarray or narray or numstar or numerix or something....) fft (based on c-only code -- no fortran dependency) linalg (a lite version -- no fortran or ATLAS dependency) stats (a lite version --- no fortran dependency) special (only c-code --- no fortran dependency) weave f2py? (still need to ask Pearu about this) scipy_distutils and testing matrix and polynomial classes
...others...?
We will push to make this an easy-to-install effective replacement for Numeric and hopefully for numarray users as well. Therefore community input and assistance will be particularly important.
2) The rest of scipy will be a package (or a series of packages) of algorithms. We will not try to do plotting as part of scipy. The current plotting in scipy will be supported for a time, but users will be weaned off to other packages: matplotlib, pygist (for xplt -- and I will work to get any improvements for xplt into pygist itself), gnuplot, etc.
3) Having everything under a scipy namespace is not necessary, nor worth worrying about at this point.
My scipy-related focus over the next 5-6 months will be to get scipy_core to the point that most can agree it effectively replaces the basic tools of Numeric and numarray.
-Travis
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.net http://www.scipy.net/mailman/listinfo/scipy-user
On Mar 9, 2005, at 11:41 PM, eric jones wrote:
2. I do question whether weave really be in this core? I think it was in scipy_core before because it was needed to build some of scipy. 3. Now that I think about it, I also wonder if f2py should really be there -- especially since we are explicitly removing any fortran dependencies from the core.
It would seem to me that so long as: 1) both these tools have very general usefulness (and I think they do), and 2) are not installation problems (I don't believe they are since they themselves don't require any compilation of Fortran, C++ or whatever--am I wrong on that?) That they are perfectly fine to go into the core. In fact, if they are used by any of the extra packages, they should be in the core to eliminate the extra step in the installation of those packages. Perry
Can I put in a good word for Fortran? Not the language itself, but the available packages for it. I've always thought that one of the really good things about Scipy was the effort put into getting all those powerful, well tested, robust Fortran routines from Netlib inside Scipy. Without them, it seems to me that folks who just install the new scipy_base are going to re-invent a lot of wheels. Is it really that hard to install g77 on non-Linux platforms? Steve Walton
There are no plans to abandon Fortran. SciPy will still rely on it heavily. We're only talking about making some core features here available as python + C to make them easily buildable by the widest group of people. Everything else will still use Fortran the same way it does now. eric Stephen Walton wrote:
Can I put in a good word for Fortran? Not the language itself, but the available packages for it. I've always thought that one of the really good things about Scipy was the effort put into getting all those powerful, well tested, robust Fortran routines from Netlib inside Scipy. Without them, it seems to me that folks who just install the new scipy_base are going to re-invent a lot of wheels.
Is it really that hard to install g77 on non-Linux platforms?
Steve Walton
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.net http://www.scipy.net/mailman/listinfo/scipy-user
"Travis" == Travis Oliphant <oliphant@ee.byu.edu> writes:
Travis> It would seem that while the scipy conference demonstrates Travis> a continuing and even increasing use of Python for Travis> scientific computing, not as many of these users are scipy Travis> devotees. Why? Hi Travis, I like a lot of your proposal, and I want to throw a couple of additional ideas into the mix. There are two ideas about what scipy is: a collection of scientific algorithms and a general purpose scientific computing environment. On the first front, scipy has been a great success; on the second, less so. I think the following would be crucial to make such an effort a success (some of these are just restatements of your ideas with additional comments) * Easy to install: - it would be probably be important to have a fault-tolerant install so that even if a component fails, the parts that don't depend on that can continue. Matthew Knepley's build system might be an important tool to make this work right for source installs, rather than trying to push distutils too hard. * A package repository and a way of specifying dependencies between the packages and allow automated recursive downloads ala apt-get, yum, etc.... So basically we have to come up with a package manager, and probably one that supports src as well as binary installs. Everyone knows this is a significant problem in python, and we're in a good place to tackle it in that we have experience distributing complex packages across platforms which are a mixture of python/C/C++/FORTRAN, so if we can make it work, it will probably work for all of python. I think we would want contributions from people who do packaging on OSX and win32, eg Bob Ippolito, Joe Cooper, Robert Kern, and others. * Transparent support for Numeric, numarray and Numeric3 built into a compatibility layer, eg something like matplotlib.numerix which enables the user to be shielded from past and future changes in the array package. If you and the numarray developers can agree on that interface, that is an important start, because no matter how much success you have with Numeric3, Numeric 23.x and numarray will be in the wild for some time to come. Having all the major players come together and agree on a core interface layer would be a win. In practice, it works well in matplotlib.numerix. * Buy-in from the developers of all the major packages that people want and need to have the CVS / SVN live on a single site which also has mailing lists etc. I think this is a possibility, actually; I'm open to it at least. * Good tutorial, printable documentation, perhaps following a "dive into python" model with a "just-in-time" model of teaching the language; ie, task oriented. A question I think should be addressed is whether scipy is the right vehicle for this aggregation. I know this has been a long-standing goal of yours and appreciate your efforts to continue to make it happen. But there is a lot of residual belief that scipy is hard to install, and this is founded in an old memory that refuses, sometimes irrationally, to die, and in part from people's continued difficulties. If we make a grand effort to unify into a coherent whole, we might be better off with a new name that doesn't carry the difficult-to-install connotation. And easy-to-install should be our #1 priority. Another reason to consider a neutral name is that it wouldn't scare off a lot of people who want to use these tools but don't consider themselves to be scientists. In matplotlib, there are people who just want to make bar and pie charts, and in talks I've given many people are very happy when I tell them that I am interested in providing plotting capabilities outside the realm of scientific plotting. This is obviously a lot to bite off but it could be made viable with some dedicated effort; python is like that. Another concern I have, though, is that it seems to duplicate a lot of the enthought effort to build a scientific python bundle -- they do a great job already for win32 and I think an enthought edition for linux and OSX are in the works. The advantage of your approach is that it is modular rather than monolithic. To really make this work, I think enthought would need to be on board with it. Eg mayavi2 and traits2 are both natural candidates for inclusion into this beast, but both live in the enthought subversion tree. Much of what you describe seems to be parallel to the enthought python, which also provides scipy, numeric, ipython, mayavi, plotting, and so on. I am hesitant to get too involved in the packaging game -- it's really hard and would take a lot of work. We might be better off each making little focused pieces, and let packagers (pythonmac, fink, yum, debian, enthought, ...) do what they do well. Not totally opposed, mind you, just hesitant.... JDH
participants (11)
-
Alan G Isaac -
Bob Ippolito -
Colin J. Williams -
Cory Davis -
eric jones -
John Hunter -
Perry Greenfield -
Prabhu Ramachandran -
Robert Kern -
Stephen Walton -
Travis Oliphant