Hello all, Is there a way to get probability values for the various families of distributions in numpy? I.e. ala R:
pnorm(1.96, mean = 0 , sd = 1)
[1] 0.9750021 # for the normal
pt(1.65, df=100)
[1] 0.9489597 # for student t Any suggestions would be greatly appreciated. Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 9097932853 (2563) mjanikas@esri.com
Mark Janikas wrote:
Hello all,
Is there a way to get probability values for the various families of distributions in numpy? I.e. ala R:
We have a full complement of PDFs, CDFs, etc. in scipy. In [1]: from scipy import stats In [2]: stats.norm.pdf(1.96, loc=0.0, scale=1.0) Out[2]: array(0.058440944333451469) In [3]: stats.norm.cdf(1.96, loc=0.0, scale=1.0) Out[3]: array(0.97500210485177952)  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
On Wed, 20 Dec 2006, Robert Kern apparently wrote:
We have a full complement of PDFs, CDFs, etc. in scipy.
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.) Although it is a slippery slope, and I definitely do not want NumPy to slide down it, I would certainly not complain if this basic functionaltiy were moved to NumPy... Cheers, Alan Isaac
On Dec 20, 2006, at 8:41 PM, Alan G Isaac wrote:
On Wed, 20 Dec 2006, Robert Kern apparently wrote:
We have a full complement of PDFs, CDFs, etc. in scipy.
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.)
If they're already installing numpy, isn't 98% of the work already done at that point? I'm pretty sure you can install scipy w/o the fortran dependency if you're not concerned about speed and what not, right? It should be a pretty easy install. Besides ... what else do they have to do during the first weekanda half of the semester anyway, right? ;) steve
On 20/12/06, Alan G Isaac <aisaac@american.edu> wrote:
On Wed, 20 Dec 2006, Robert Kern apparently wrote:
We have a full complement of PDFs, CDFs, etc. in scipy.
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.) Although it is a slippery slope, and I definitely do not want NumPy to slide down it, I would certainly not complain if this basic functionaltiy were moved to NumPy...
This is silly. If it were up to me I would rip out much of the fancy features from numpy and put them in scipy. It's really not very difficult to install, particularly if you don't much care how fast it is, or are using (say) a Linux distribution that packages it. It seems to me that numpy should include only tools for basic calculations on arrays of numbers. The ufuncs, simple wrappers (dot, for example). Anything that requires nontrivial amounts of math (matrix inversion, statistical functions, generating random numbers from exponential distributions, and so on) should go in scipy. If numpy were to satisfy everyone who says, "I like numpy, but I wish it included [their favourite feature from scipy] because I don't want to install scipy", numpy would grow to include everything in scipy. Perhaps an alternative criterion would be "it can go in numpy if it has no external requirements". I think this is a mistake, since it means users have a monstrous headache figuring out what is in which package (for example, some of scipy.integrate depends on external tools and some does not). Moreover it damages the performance of numpy. For example, dot would be faster (for arrays that happen to be matrixshaped, and possibly in general) if it could use ATLAS' routine from BLAS. Of course, numpy is currently fettered by the need to maintain some sort of compatibility with Numeric and numarray; shortly it will have to worry about compatibility with previous versions of numpy as well. A. M. Archibald
A. M. Archibald schrieb:
On 20/12/06, Alan G Isaac <aisaac@american.edu> wrote:
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.) Although it is a slippery slope, and I definitely do not want NumPy to slide down it, I would certainly not complain if this basic functionaltiy were moved to NumPy... ... If numpy were to satisfy everyone who says, "I like numpy, but I wish it included [their favourite feature from scipy] because I don't want to install scipy", numpy would grow to include everything in scipy.
Well my package manager just reported something like 800K for numpy and 20M for scipy, so I think we're not quite at the point of numpy taking over everything yet (if those numbers are actually meaningful, probably I'm missing something ?). I would also welcome if some functionality could be moved to numpy if the size requirements are reasonably small. Currently I try to avoid to depend on the scipy package to make my programs more portable, and I'm mostly successful, but not always. The pvalue stuff in numpy would be helpful here, as Alan already said. Now I don't know if that stuff passes the size criterion, some expert would know that. But if it does, it would be nice if you could consider moving it over eventually. Of course you need to strike a balance, and the optimum is debatable. But again, if scipy is really more than 20 times the size of numpy, and some frequently used things are not in numpy, is there really an urgent need to freeze numpy's set of functionality? just a user's thought, sven
Thanks for all the input so far. The only thing that seems odd about the omission of probability or quantile functions in NumPy is that all the random number generators are present in RandomArray. At any rate, hopefully this bit of functionality will be present in the future, but for now, IMO the library is awesome..... I am used to using R for math routines, and all my sparse matrix stuff is WAAAAAAY faster using the PythonNumPy Combo! Thanks to all for their insight, MJ Original Message From: numpydiscussionbounces@scipy.org [mailto:numpydiscussionbounces@scipy.org] On Behalf Of Sven Schreiber Sent: Thursday, December 21, 2006 7:10 AM To: Discussion of Numerical Python Subject: Re: [Numpydiscussion] Newbie Question, Probability A. M. Archibald schrieb:
On 20/12/06, Alan G Isaac <aisaac@american.edu> wrote:
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.) Although it is a slippery slope, and I definitely do not want NumPy to slide down it, I would certainly not complain if this basic functionaltiy were moved to NumPy... ... If numpy were to satisfy everyone who says, "I like numpy, but I wish it included [their favourite feature from scipy] because I don't want to install scipy", numpy would grow to include everything in scipy.
Well my package manager just reported something like 800K for numpy and 20M for scipy, so I think we're not quite at the point of numpy taking over everything yet (if those numbers are actually meaningful, probably I'm missing something ?). I would also welcome if some functionality could be moved to numpy if the size requirements are reasonably small. Currently I try to avoid to depend on the scipy package to make my programs more portable, and I'm mostly successful, but not always. The pvalue stuff in numpy would be helpful here, as Alan already said. Now I don't know if that stuff passes the size criterion, some expert would know that. But if it does, it would be nice if you could consider moving it over eventually. Of course you need to strike a balance, and the optimum is debatable. But again, if scipy is really more than 20 times the size of numpy, and some frequently used things are not in numpy, is there really an urgent need to freeze numpy's set of functionality? just a user's thought, sven _______________________________________________ Numpydiscussion mailing list Numpydiscussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpydiscussion
Mark Janikas wrote:
Thanks for all the input so far. The only thing that seems odd about the omission of probability or quantile functions in NumPy is that all the random number generators are present in RandomArray. A big part of the issue is that getting many of those pdfs into NumPy would require putting many special functions into NumPy (some of which are actually coded in Fortran).
I much prefer to make SciPy an easy install for as many people as possible and/or work on breaking up SciPy into modular components that can be installed separately if needed. This was my original intention  to make NumPy as small as possible. It's current size is driven by backwards compatibility, only. Travis
On Thursday 21 December 2006 16:10, Travis Oliphant wrote:
I much prefer to make SciPy an easy install for as many people as possible and/or work on breaking up SciPy into modular components that can be installed separately if needed.
Talking about that, what happened to these projects of modular installation of scipy ? Robert promised us last month to explain what went wrong with his approach, but never had the time...
Pierre GM wrote:
On Thursday 21 December 2006 16:10, Travis Oliphant wrote:
I much prefer to make SciPy an easy install for as many people as possible and/or work on breaking up SciPy into modular components that can be installed separately if needed.
Talking about that, what happened to these projects of modular installation of scipy ? Robert promised us last month to explain what went wrong with his approach, but never had the time...
I created a module (scipy_subpackages.py, IIRC) next to setup.py that essentially just served as a global configuration to inform all of the setup.py's what subpackages they were supposed to build (mostly just Lib/setup.py, actually). I then had a script run through the various collections of subpackages that I wanted to build, set the appropriate values in scipy_subpackages, and run setup() with the appropriate parameters to build an egg for each collection. However, build/ apparently needs to be cleaned out between each egg, otherwise you contaminate later eggs.  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
Robert Kern schrieb:
Pierre GM wrote:
Talking about that, what happened to these projects of modular installation of scipy ? Robert promised us last month to explain what went wrong with his approach, but never had the time...
I created a module (scipy_subpackages.py, IIRC) next to setup.py that essentially just served as a global configuration to inform all of the setup.py's what subpackages they were supposed to build (mostly just Lib/setup.py, actually). I then had a script run through the various collections of subpackages that I wanted to build, set the appropriate values in scipy_subpackages, and run setup() with the appropriate parameters to build an egg for each collection.
However, build/ apparently needs to be cleaned out between each egg, otherwise you contaminate later eggs.
So, to put it "pointedly" (if that's the right word...?): Numpy should not get small functions from scipy > because the size of scipy doesn't matter > because scipy's modules will be installable as addons separately (and because there will be readytouse installers); however, nobody knows how to actually do that in practice ? <irony>Well that will convince my colleagues!</irony> Please don't feel offended, I just want to make the point (as usual) that this way numpy is going to be a good library for other software projects, but not superattractive for direct users (aka "matlab converts", although I personally don't come from matlab). cheers, sven
On Fri, Dec 22, 2006 at 10:47:33AM +0100, Sven Schreiber wrote:
Please don't feel offended, I just want to make the point (as usual) that this way numpy is going to be a good library for other software projects, but not superattractive for direct users (aka "matlab converts", although I personally don't come from matlab).
I think that the equivalent of MatLab is more scipy than numpy. I thing there is a missunderstanding here. Gaël
Sven Schreiber wrote:
So, to put it "pointedly" (if that's the right word...?): Numpy should not get small functions from scipy > because the size of scipy doesn't matter > because scipy's modules will be installable as addons separately (and because there will be readytouse installers); however, nobody knows how to actually do that in practice ?
I just want to make the point (as usual) that this way numpy is going to be a good library for other software projects, but not superattractive for direct users (aka "matlab converts"
No one is denying that there is still work to do. I also think there are a lot of us (from "just users" to the major contributers), that WOULD like to see an easy to install package that can do everything (and more, and better) that MATLAB can. The only questions are: A) How best to accomplish this (or work toward it anyway)? B) Who's going to do the work? As for (A), I think the consensus is pretty clear  keep numpy focused on the basic array package, with some extras for backwards compatibility, and work towards a SciPy that has all the bells and whistles, preferably installable as separate packages (like Matlab "toolboxes" I suppose). As for the above comments, if we are looking at the "Matlab converts", or more to the point, people looking for a comprehensive scientific/engineering computation package:  "The size of SciPy doesn't matter"  True, after all, how big is MATLAB?  "Scipy's modules will be installable as addons separately"  this is a good goal, and I think there has been progress there.  "Nobody knows how to actually do that in practice"  well, it's not so much that nobody knows how to do it, as that nobody has done it  it's going to take work, but adding extra stuff to numpy takes work too, it's a matter of where you're going to focus the work. Given the size of disk drives and the speed of Internet connections these days, I'm not sure it's that important to have the "core" part of SciPy very small  but it does need to have easy installers. That approach provides opportunity though  breaking SciPy down into smaller packages requires expertise and consensus among the developers. However, building an installer requires only one person to take the time to do it. Yes, SciPy is too hard to build and install for an average newbie  but it's gotten better, and it's not too hard for a savvy user that is willing to put some time it. The kind of person who is willing to put the time in to post to discussions on this group, for instance. Please, rather than complaining that core developers aren't putting your personally desired "small function" into numpy , just take the time to build the installer you need  we need one for OSX, build it and put it up on pythonmac  it's not that hard, and there are a lot of people here and on the scipy and pythonmac lists that will help. Now my rant: Please, please, please, could at least a few of the people that build packages take the time to make a simple installer for them and put them up on the web somewhere? Now a question: One of the key difficulties to building SciPy is that parts of it depend on Fortran and LAPACK. We'd all like LAPACK to be built to support our particular hardware for best performance. However, would it be that hard to have Scipy build by default with generic LAPACK (kind of like numpy), and put installers built that way up on the web, along with instructions for those that want to rebuild and optimize? For that matter, is it possible (let's say on Windows) to deliver SciPy with a set of multiple dlls for LAPACK/BLAS, and have the appropriate ones chosen at install or run time? Chris  Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 5266959 voice 7600 Sand Point Way NE (206) 5266329 fax Seattle, WA 98115 (206) 5266317 main reception Chris.Barker@noaa.gov
Sven Schreiber wrote:
So, to put it "pointedly" (if that's the right word...?): Numpy should not get small functions from scipy > because the size of scipy doesn't matter > because scipy's modules will be installable as addons separately (and because there will be readytouse installers); however, nobody knows how to actually do that in practice ?
Rather, to put it accurately, numpy should not get large chunks of scipy functionality that require FORTRAN dependencies for reasons that should be obvious from that description. scipy.stats.distributions is just such a chunk. The ancillary point is that I think that, for those who do find the largeness and difficulttoinstallness of scipy onerous, the best path forward is to work on the build process of scipy. And it will take *work* not wishes nor complaints nor <irony/> tags. And honestly, the more I see the latter, the less motivated I am to bother with the former.  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
Robert Kern schrieb:
Rather, to put it accurately, numpy should not get large chunks of scipy functionality that require FORTRAN dependencies for reasons that should be obvious from that description. scipy.stats.distributions is just such a chunk.
I was probably not very clear, I was referring to "small" functions. As you and others have pointed out, for pvalue stuff this doesn't apply apparently. Ok.
The ancillary point is that I think that, for those who do find the largeness and difficulttoinstallness of scipy onerous, the best path forward is to work on the build process of scipy. And it will take *work* not wishes nor complaints nor <irony/> tags. And honestly, the more I see the latter, the less motivated I am to bother with the former.
My impression of the discussion was that many people said _nothing_ at all should be added into numpy ever, which sounded kind of fundamentalistic to me. And partly the given justification were some features of scipy that don't exist yet (fair enough), and that nobody is even working on. Of course I understand the lack of manpower, but then I think this state of affairs should be properly taken into account when arguing against moving (only small!) features from scipy into numpy. Hence my earlier post. I also try to contribute to opensource projects where I can, and believe me, it would probably help my career more to just have my faculty pay for matlab and forget about numpy et al. You know, users' time is valuable, too. Unfortunately I don't have the skills to help with modularizing scipy (nor the time to acquire those skills). Btw, that's why I like the idea of paying for stuff like documentation and other things that opensource projects often forget about because it's not fun for the developers. (Hey, I'm an economist...) So I would be willing to donate money for some of the dull tasks, for example. (I'm fully aware that would not cover the real cost of the work, just like in Travis' case with the numpy guide.) Ok, that's enough, happy holidays, Sven
Sven Schreiber wrote:
Robert Kern schrieb:
Pierre GM wrote:
So, to put it "pointedly" (if that's the right word...?): Numpy should not get small functions from scipy > because the size of scipy doesn't matter > because scipy's modules will be installable as addons separately (and because there will be readytouse installers); however, nobody knows how to actually do that in practice ?
<irony>Well that will convince my colleagues!</irony>
Please don't feel offended, I just want to make the point (as usual) that this way numpy is going to be a good library for other software projects, but not superattractive for direct users (aka "matlab converts", although I personally don't come from matlab).
Don't worry about offending, we all recognize the weaknesses. I think you are pointing out something most of us already see. It is the combination of SciPy+NumPy+Matplotlib+IPython (+ perhaps a good IDE) that can succeed at being a MATLAB/IDL replacement for a lot of people. NumPy by itself isn't usually enough, and it's also important to keep NumPy as a library that can be used for other development. What is also needed is a good "package" of it all  like the Enthon distribution. This requires quite a bit of thankless work. Enthought has done quite a bit in this direction for Windows but they have not had client demand to get it wrapped up for other platforms. I think people are hoping that eggs will help here but it hasn't come to fruition. This is an area that SciPy could really use someone stepping up and taking charge. I like the discussions that have taken place regarding documentation issues, as well as the work that went into making all of SciPy compile with gfortran (though perhaps not bug free...). These are important steps that are much appreciated. Best regards, Travis
Travis Oliphant wrote:
It is the combination of SciPy+NumPy+Matplotlib+IPython (+ perhaps a good IDE) that can succeed at being a MATLAB/IDL replacement for a lot of people.
What is also needed is a good "package" of it all  like the Enthon distribution. This requires quite a bit of thankless work.
I know Robert put some serious effort into "MacEnthon" a while back, but is no longer maintaining that, which doesn't surprise me a bit  that looked like a LOT of work. However, MacEnthon was much bigger than that just the packages Travis listed above, and I think Travis has that right  those are the key ones to do. Let's "just do it!"  first we need to solve the Fortran+Universal binary problems though  that seems to be the technical sticking point on OSX Also, while the Enthon distribution is fabulous, they do tend to stay behind the bleeding edge a fair bit  it would be nice to have the core packages with the latest and greatest on Windows and Linux too, all as one easy installer. (or rpm or .deb or whatever for Linux) Chris  Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 5266959 voice 7600 Sand Point Way NE (206) 5266329 fax Seattle, WA 98115 (206) 5266317 main reception Chris.Barker@noaa.gov
On Wed, 27 Dec 2006, Christopher Barker wrote:
Travis Oliphant wrote:
It is the combination of SciPy+NumPy+Matplotlib+IPython (+ perhaps a good IDE) that can succeed at being a MATLAB/IDL replacement for a lot of people.
What is also needed is a good "package" of it all  like the Enthon distribution. This requires quite a bit of thankless work.
I know Robert put some serious effort into "MacEnthon" a while back, but is no longer maintaining that, which doesn't surprise me a bit  that looked like a LOT of work.
However, MacEnthon was much bigger than that just the packages Travis listed above, and I think Travis has that right  those are the key ones to do. Let's "just do it!"  first we need to solve the Fortran+Universal binary problems though  that seems to be the technical sticking point on OSX
Let me add a comment on the Fortran problem (which I assume to be the (lack of) Fortran compiler problem, right?). I have been working on f2py rewrite to support wrapping Fortran 90 types among other F90 constructs and as a result we have almost a complete Fortran parser in Python. It is relatively easy to use this parser to automatically convert Fortran 77 codes that we have in scipy to C codes whenever no Fortran compiler is available. Due to lack of funding this work has been freezed for now but I'd say that there is a hope to resolve the Fortran compiler issues for any platform in future. Pearu
pearu@cens.ioc.ee wrote:
I have been working on f2py rewrite to support wrapping Fortran 90 types among other F90 constructs and as a result we have almost a complete Fortran parser in Python. It is relatively easy to use this parser to automatically convert Fortran 77 codes that we have in scipy to C codes whenever no Fortran compiler is available.
Cool! How is the different/better than the old standby f2c? One issue with f2c is that it required a pretty good set of libs to support stuff that Fortran had that C didn't  complex numbers come to mind, I'm not sure what else was is in libf2c. In fact, I've often wondered why scipy doesn't use f2c. Chris  Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 5266959 voice 7600 Sand Point Way NE (206) 5266329 fax Seattle, WA 98115 (206) 5266317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
pearu@cens.ioc.ee wrote:
I have been working on f2py rewrite to support wrapping Fortran 90 types among other F90 constructs and as a result we have almost a complete Fortran parser in Python. It is relatively easy to use this parser to automatically convert Fortran 77 codes that we have in scipy to C codes whenever no Fortran compiler is available.
Cool!
How is the different/better than the old standby f2c?
One issue with f2c is that it required a pretty good set of libs to support stuff that Fortran had that C didn't  complex numbers come to mind, I'm not sure what else was is in libf2c.
In fact, I've often wondered why scipy doesn't use f2c.
Generally speaking, g77 was always more likely to work on more platforms with less hassle.  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
I just discovered the: Scipy Superpack for OS X http://trichech.us/?page_id=4 Maybe this will help folks looking for an OS_X Scipy build. Chris  Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 5266959 voice 7600 Sand Point Way NE (206) 5266329 fax Seattle, WA 98115 (206) 5266317 main reception Chris.Barker@noaa.gov
On 12/20/06, A. M. Archibald <peridot.faceted@gmail.com> wrote:
Moreover it damages the performance of numpy. For example, dot would be faster (for arrays that happen to be matrixshaped, and possibly in general) if it could use ATLAS' routine from BLAS.
I thought numpy uses ATLAS. Matrix multiplication in numpy is about as fast as in Octave. So it must be using ATLAS.
A Dijous 21 Desembre 2006 05:59, A. M. Archibald escrigué:
On 20/12/06, Alan G Isaac <aisaac@american.edu> wrote:
On Wed, 20 Dec 2006, Robert Kern apparently wrote:
We have a full complement of PDFs, CDFs, etc. in scipy.
This is my "most missed" functionality in NumPy. (For now I feel cannot ask students to install SciPy.) Although it is a slippery slope, and I definitely do not want NumPy to slide down it, I would certainly not complain if this basic functionaltiy were moved to NumPy...
This is silly.
If it were up to me I would rip out much of the fancy features from numpy and put them in scipy. It's really not very difficult to install, particularly if you don't much care how fast it is, or are using (say) a Linux distribution that packages it.
It seems to me that numpy should include only tools for basic calculations on arrays of numbers. The ufuncs, simple wrappers (dot, for example). Anything that requires nontrivial amounts of math (matrix inversion, statistical functions, generating random numbers from exponential distributions, and so on) should go in scipy.
If numpy were to satisfy everyone who says, "I like numpy, but I wish it included [their favourite feature from scipy] because I don't want to install scipy", numpy would grow to include everything in scipy.
Perhaps an alternative criterion would be "it can go in numpy if it has no external requirements". I think this is a mistake, since it means users have a monstrous headache figuring out what is in which package (for example, some of scipy.integrate depends on external tools and some does not). Moreover it damages the performance of numpy. For example, dot would be faster (for arrays that happen to be matrixshaped, and possibly in general) if it could use ATLAS' routine from BLAS.
Of course, numpy is currently fettered by the need to maintain some sort of compatibility with Numeric and numarray; shortly it will have to worry about compatibility with previous versions of numpy as well.
I agree with most of the arguments above, so +1 
0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data ""
A Dijous 21 Desembre 2006 05:59, A. M. Archibald escrigué:
It seems to me that numpy should include only tools for basic calculations on arrays of numbers. The ufuncs, simple wrappers (dot, for example). Anything that requires nontrivial amounts of math (matrix inversion, statistical functions, generating random numbers from exponential distributions, and so on) should go in scipy.
As a user, I suggest that this becomes a reasonable goal when up to date SciPy installers are maintained for all target platforms. Unless you wish to exclude everyone who is intimidated when installation is less than trivial... Until then, I suggest, the question of the proper functionality bundle with NumPy remains open. Of course as a user I do not pretend to resolve such a questionrecall that I mentioned the slippery slope in my postbut I do object to it being dismissed as "silly" when I offered a straightforward explanation. It is well understood that the current view of the developers is that if anything too much is already in NumPy. Any user comments are taking place within that context. Alan Isaac PS A question: is it a good thing if more students start using NumPy *now*? It looks to me like building community size is an important current goal for NumPy. Strip it down like you suggest and aside from Windows users (and Macs are increasingly popular among my students) you'll have only the few that are not intimidated by building SciPy (which still has no intaller for Python 2.5).
On 21/12/06, Alan G Isaac <aisaac@american.edu> wrote:
A Dijous 21 Desembre 2006 05:59, A. M. Archibald escrigué:
It seems to me that numpy should include only tools for basic calculations on arrays of numbers. The ufuncs, simple wrappers (dot, for example). Anything that requires nontrivial amounts of math (matrix inversion, statistical functions, generating random numbers from exponential distributions, and so on) should go in scipy.
As a user, I suggest that this becomes a reasonable goal when up to date SciPy installers are maintained for all target platforms. Unless you wish to exclude everyone who is intimidated when installation is less than trivial...
Until then, I suggest, the question of the proper functionality bundle with NumPy remains open. Of course as a user I do not pretend to resolve such a questionrecall that I mentioned the slippery slope in my postbut I do object to it being dismissed as "silly" when I offered a straightforward explanation.
It is well understood that the current view of the developers is that if anything too much is already in NumPy. Any user comments are taking place within that context.
Just to be clear: I am not a developer. I am a user who is frustrated with the difficulty of telling whether to look for a given feature in numpy or in scipy. (I have also never really had much difficulty installing scipy either from the packages in one of several linux distribution or compiling it from scratch.) I suppose the basic difference of opinions here is that I think numpy has already taken too many steps down the slippery slope. Also I don't think 20 megabytes is enough disk space to care about, and I think it is better in the long term to encourage the scipy developers to get the installers working than it is to jam all kinds of scientific functionality into this array package to avoid having to install the scientific computing package.
PS A question: is it a good thing if more students start using NumPy *now*? It looks to me like building community size is an important current goal for NumPy. Strip it down like you suggest and aside from Windows users (and Macs are increasingly popular among my students) you'll have only the few that are not intimidated by building SciPy (which still has no intaller for Python 2.5).
I didn't have to build scipy (though I have, it's not hard), and I don't use Windows. But no, I don't think it can be stripped down yet; the backward compatibility issue is currently important. I think moving scientific functionality from scipy to numpy is a step in the wrong direction, though. A. M. Archibald
A key thing to remember here is that each user has their particular set of "small things" that are all they need from scipy  put us all together, and you have SciPy  that's what it is for.
As a user, I suggest that this becomes a reasonable goal when up to date SciPy installers are maintained for all target platforms.
All it takes is someone to do it. Also, there was talk of "modularizing" scipy so that it would be easy to install only those bits you need  in particular, the nonFortran stuff should be trivial to build.
Macs are increasingly popular among my students
It can be a pain to build this kind of thing on OSX, as Apple has not supported a Fortran compiler yet, but it can (and has) been done. IN fact, the Mac is a great target for prebuilt binaries as there is only a small variety of hardware to support, and Apple supplies LAPACK/BLAS libs with the system. As for distributing it, the archive at: pythonmac.org/packages takes submissions from anyone  just send a note to the pythonmac list  that list is a great help in figuring out how to build stuff too. Chris  Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 5266959 voice 7600 Sand Point Way NE (206) 5266329 fax Seattle, WA 98115 (206) 5266317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
It can be a pain to build this kind of thing on OSX, as Apple has not supported a Fortran compiler yet, but it can (and has) been done. IN fact, the Mac is a great target for prebuilt binaries as there is only a small variety of hardware to support, and Apple supplies LAPACK/BLAS libs with the system. As for distributing it, the archive at:
I'm always confused about how to distribute something like SciPy for the MAC. What exactly should be distributed? Is it possible to use distutils to get it done? I'd love to provide MAC binaries of SciPy / NumPy. Right now, we rely on people like Chris for that. Travis
Travis Oliphant wrote:
I'm always confused about how to distribute something like SciPy for the MAC. What exactly should be distributed? Is it possible to use distutils to get it done?
To get a package format that is actually useful (bdist_dumb just doesn't cut it on any platform, really), you need to install something else. I prefer building eggs instead of mpkgs. So this is what I do: Install setuptools. I then create a script (I usually call it ./be) in the numpy/ and scipy/ directories to hold all of the options I use for building: #!/bin/sh pythonw2.5 c "import setuptools; execfile('setup.py')" build_src build_clib fcompiler=gnu95 build_ext fcompiler=gnu95 build "$@" Then, to build an egg: $ ./be bdist_egg You can then upload it to the Package Index (maybe. I had trouble uploading the Windows scipy binary that Gary Pajer sent me. I suspect that the Index rejected it because it was too large). Here are the outstanding issues as I see them: * Using scipy requires that the FORTRAN runtime libraries that you compiled against be installed in the appropriate place, i.e. /usr/local/lib. This is annoying, since there are currently only tarballs available, so the user needs root access to install them. If an enterprising individual wants to make this situation better, he might try to make a framework out of the necessary libraries such that we can simply link against those. Frameworks are easier to install to different locations with less hassle. http://hpc.sourceforge.net/ * g77 does not work with the Universal Python build process, so we are stuck with gfortran. * The GNU FORTRAN compilers that are available are architecturespecific. For us, that means that we cannot build Universal scipy binaries. If you build on an Intel Mac, the binaries will only work on Intel Macs; if you build on a PPC Mac, likewise, the resulting binaries only work on PPC Macs. * In a related problem, I cannot link against ATLAS on Intels, and possibly not on PPCs, either (I haven't tried building with a Universal Python on PPC). The Universal compile flags (notably "arch ppc arch intel") are used when compiling the numpy.distutils test programs for discovering ATLAS's version. Using a singlearchitecture ATLAS library causes those test programs to not link (since they are missing the _ATL_buildinfo symbol for the missing architecture). I've tried using lipo(1) to assemble a Universal ATLAS library from a PPCbuilt library and an Intelbuilt library, but this did not change the symptomology. Fortunately, part of ATLAS is already built into the Accelerate.framework provided with the OS and is automatically recognized by numpy.distutils. It's missing the C versions of LAPACK functions, so scipy.linalg.clapack will be empty. Also, I think numpy.distutils won't recognize that it is otherwise an ATLAS (_ATL_buildinfo is also missing from the framework), so it may not try to compile the C BLAS interfaces, either.  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
Alan G Isaac wrote:
PS A question: is it a good thing if more students start using NumPy *now*? It looks to me like building community size is an important current goal for NumPy. Strip it down like you suggest and aside from Windows users (and Macs are increasingly popular among my students) you'll have only the few that are not intimidated by building SciPy (which still has no intaller for Python 2.5).
You mean a Windows installer? Yes, it does. http://sourceforge.net/project/showfiles.php?group_id=27747  Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco
Alan G Isaac wrote:
Strip it down like you suggest and aside from Windows users (and Macs are increasingly popular among my students) you'll have only the few that are not intimidated by building SciPy (which still has no intaller for Python 2.5).
On Fri, 22 Dec 2006, Robert Kern apparently wrote:
You mean a Windows installer? Yes, it does. http://sourceforge.net/project/showfiles.php?group_id=27747
1. No, I meant a Mac installer. Sorry that was unclear. And let me be clear that I understand that if I really want one for my students that I should learn how to build one. (And if I get a moment to breathe, I'd like to learn how.) My point was only that lack of availability does have implications. 2. Re: other message, I was not aware that moving scipy.stats.distributions into NumPy would complicate the Numpy build process (which is currently delightfully easy). Thank you, Alan Isaac
participants (13)

A. M. Archibald

Alan G Isaac

Christopher Barker

Francesc Altet

Gael Varoquaux

Keith Goodman

Mark Janikas

pearu＠cens.ioc.ee

Pierre GM

Robert Kern

Steve Lianoglou

Sven Schreiber

Travis Oliphant