Restructuring of SciPy
After the 0.3.2 release, scipy will be getting a face-lift. As part of that, I'm wondering about the restructuring process. For a long time, we've been operating under the "big-scipy-package" with capability for small packages to be released separately model. It has had limited success and I don't know of any small packages actually being released like that. With things like Enthon out there that act as the "batteries included super distribution" anyway. It makes me wonder if we shouldn't move scipy to the "scipy-modules" perspective directly and develop and think about scipy has a collection of modules that work together smoothly (i.e. naming conventions, etc.) scipy_core + available instructions could provide all the infrastructure someone needs to contribute packages to scipy. This would also allow people to develop (and maintain) their own modules (which they seem to like to do anyway). If we could just get them to play nice with the rest of scipy_core, it would all be for benefit. We already have the infrastructure in place for modules to play nicely with the overall package. I'm just wondering if we shouldn't emphasize more the modular approach by the way the svn tree is layed out (so individual modules can be checked out independently, for example). I've also spoken to people who find it hard to rely on their scipy in their code they pass on because you have to get the whole package, even if you only use a few parts. Matlab also uses modules effectively. I think we should take advantage of the restructuring process to think about this. Comments? -Travis
On Wed, 2005-08-17 at 14:31 -0600, Travis Oliphant wrote:
After the 0.3.2 release, scipy will be getting a face-lift.
As part of that, I'm wondering about the restructuring process. For a long time, we've been operating under the "big-scipy-package" with capability for small packages to be released separately model. It has had limited success and I don't know of any small packages actually being released like that.
With things like Enthon out there that act as the "batteries included super distribution" anyway. It makes me wonder if we shouldn't move scipy to the "scipy-modules" perspective directly and develop and think about scipy has a collection of modules that work together smoothly (i.e. naming conventions, etc.)
scipy_core + available instructions could provide all the infrastructure someone needs to contribute packages to scipy. This would also allow people to develop (and maintain) their own modules (which they seem to like to do anyway). If we could just get them to play nice with the rest of scipy_core, it would all be for benefit.
We already have the infrastructure in place for modules to play nicely with the overall package. I'm just wondering if we shouldn't emphasize more the modular approach by the way the svn tree is layed out (so individual modules can be checked out independently, for example).
I've also spoken to people who find it hard to rely on their scipy in their code they pass on because you have to get the whole package, even if you only use a few parts.
Matlab also uses modules effectively.
I think we should take advantage of the restructuring process to think about this.
Comments?
-Travis
I think it is a good idea if: 1) There is good documentation so folks know which modules to get. For instance, as far as I know, the zero finders are still in Optimization and that is not the obvious place to look. Perhaps a package approach combined with modules would work better. Or at least some sort of index. 2) It is possible to deal with module dependences. The increasing number of programs also makes this desirable, as we can include more and worry less about suitability. -Chuck
On Wed, 2005-08-17 at 14:31 -0600, Travis Oliphant wrote:
scipy_core + available instructions could provide all the infrastructure someone needs to contribute packages to scipy. This would also allow people to develop (and maintain) their own modules (which they seem to like to do anyway). If we could just get them to play nice with the rest of scipy_core, it would all be for benefit.
I think the user's perspective is: what we really need is a preconfigured package manager. From this I conclude that any restructuring will benefit by enabling easy access with a package manager such as PackMan or pypan. http://www.python.org/packman/ http://starship.python.net/crew/theller/pypan/ A user's perspective: as long as - installation is simple, and - the core of what I need is one install that is easy to update then I'll be happy. SciPy has largely met that demand for me, except that I had to flail about to learn how to best approach simple data driven graphics. Message from *that* experience: make smart choices for the new user! Do NOT just give me a bunch of options when I am starting out, as I will lack the information to choose among them. That may suggest a fairly large SciPy, since lots of initial choices should be made for the new users. (Of course it should also be easy to deviate from these if desired.) Question for the longer term: what structure will make it as easy as possible for people with very focused interests to add modules and submit patches that will be rapidly incorprated in SciPy? Surely some of the recent open source research literature must offer some help here? (But I do not know what it is.) I suspect the answer is: it must be possible for a person in charge of one module/package not to worry about what is happening with other modules/packages. Is this affected by the proposal? fwiw, Alan Isaac
On Aug 17, 2005, at 4:31 PM, Travis Oliphant wrote:
After the 0.3.2 release, scipy will be getting a face-lift.
As part of that, I'm wondering about the restructuring process. For a long time, we've been operating under the "big-scipy-package" with capability for small packages to be released separately model. It has had limited success and I don't know of any small packages actually being released like that.
With things like Enthon out there that act as the "batteries included super distribution" anyway. It makes me wonder if we shouldn't move scipy to the "scipy-modules" perspective directly and develop and think about scipy has a collection of modules that work together smoothly (i.e. naming conventions, etc.)
scipy_core + available instructions could provide all the infrastructure someone needs to contribute packages to scipy. This would also allow people to develop (and maintain) their own modules (which they seem to like to do anyway). If we could just get them to play nice with the rest of scipy_core, it would all be for benefit.
We already have the infrastructure in place for modules to play nicely with the overall package. I'm just wondering if we shouldn't emphasize more the modular approach by the way the svn tree is layed out (so individual modules can be checked out independently, for example).
I've also spoken to people who find it hard to rely on their scipy in their code they pass on because you have to get the whole package, even if you only use a few parts. Matlab also uses modules effectively. I think we should take advantage of the restructuring process to think about this.
Comments?
-Travis
It's probably already obvious what I think but I don't see any harm in reiterating it. I think the biggest obstacle to scipy in its current form is that many users find it hard to install. Yes, some of you may have no problems, but I've run into many that have (and some of them are pretty experienced at doing installations). It is essential that whatever the core is, it be an easy install for the vast majority of users. I think Travis is on the right track here. My only variance is that I wouldn't object to including more popular modules in the core distribution *so long as they are easy installs on all popular platforms*. Anything that isn't shouldn't be in the core. But there are other good reasons for splitting them that Travis alludes to. Some module may have a faster pace of development and don't want to be synced to scipy.core releases. Not everything will be at the same level of maturity, testing, or documentation. The core should have higher standards than what optional models have in that respect. Also, nothing prevents larger binary bundles (assuming someone is willing to put the work into making them). But there is no denying that doing this introduces dependency issues that must be managed as well. Perry
I think the biggest obstacle to scipy in its current form is that many users find it hard to install. Yes, some of you may have no problems, but I've run into many that have (and some of them are pretty experienced at doing installations). It is essential that whatever the core is, it be an easy install for the vast majority of users. I think Travis is on the right track here. My only variance is that I wouldn't object to including more popular modules in the core distribution *so long as they are easy installs on all popular platforms*. Anything that isn't shouldn't be in the core.
Yes, scipy_core will be easy to install (right now that translates to no Fortran code). That is a very important goal. I won't be happy if people do not find scipy_core to be a useful and better replacement for Numeric and numarray. -Travis
participants (4)
-
Alan G Isaac -
Charles R Harris -
Perry Greenfield -
Travis Oliphant