In the interest of improving the quality of the scipy.stats package, I hereby declare April and May of 2006 to be Statistics Review Months. I propose that we set ourselves a goal to review each function in stats.py and morestats.py (and a few others) for correctness and completeness of implementation by the end of May. By my count, that's about 2.5 functions every day. Surely this is a reasonable amount of effort for a rather large payoff: a robust, well-tested and thorough statistics library. I have added a Wiki page describing the details: http://projects.scipy.org/scipy/scipy/wiki/StatisticsReview Barring any objections, I will be irretrievably creating the ~150 tickets or so for all of the functions to be reviewed later tonight. So if you object, act fast! [Disclosure: this idea isn't mine. Eric Jones mentioned it to me once, and I'm just running with it.] -- Robert Kern robert.kern@gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
on this topic, as an honest-to-goodness statistician it might be nice to see more statistical modelling in scipy. i know Rpy exists, but the interface is not very pythonic. i have some "home-brew" modules for linear regression, formula building (something like R's) and a few other things. if it went into something like scipy, it might gain from the criticisms of others.... is there any interest in making the equivalent of a scipy.stats.models module? i think an easily (medium-term) achievable goal is: i) linear (least-squares) regression models with/without weights or non-diagonal covariance matrices (in R: lm + more) ii) generalized linear models (in R: glm) iii) iteratively reweighted least squares algorithms (glm is a special case), i.e. robust regression (in R: rlm). iv) ordinary least squares multivariate linear models (i.e. multivariate responses) some of these models can easily be "broadcasted", others not so easily.... further goals are more general models: classification, constrained model fitting, model selection.... for some of these things, it may not be worth duplicating R's (or other packages') efforts. -- jonathan Robert Kern wrote:
In the interest of improving the quality of the scipy.stats package, I hereby declare April and May of 2006 to be Statistics Review Months. I propose that we set ourselves a goal to review each function in stats.py and morestats.py (and a few others) for correctness and completeness of implementation by the end of May. By my count, that's about 2.5 functions every day. Surely this is a reasonable amount of effort for a rather large payoff: a robust, well-tested and thorough statistics library.
I have added a Wiki page describing the details:
http://projects.scipy.org/scipy/scipy/wiki/StatisticsReview
Barring any objections, I will be irretrievably creating the ~150 tickets or so for all of the functions to be reviewed later tonight. So if you object, act fast!
[Disclosure: this idea isn't mine. Eric Jones mentioned it to me once, and I'm just running with it.]
-- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305
On Sat, 01 Apr 2006 20:20:02 -0800 Jonathan Taylor <jonathan.taylor@stanford.edu> wrote:
on this topic, as an honest-to-goodness statistician it might be nice to see more statistical modelling in scipy. i know Rpy exists, but the interface is not very pythonic.
i have some "home-brew" modules for linear regression, formula building (something like R's) and a few other things. if it went into something like scipy, it might gain from the criticisms of others....
is there any interest in making the equivalent of a
scipy.stats.models
module?
i think an easily (medium-term) achievable goal is:
i) linear (least-squares) regression models with/without weights or non-diagonal covariance matrices (in R: lm + more)
ii) generalized linear models (in R: glm)
iii) iteratively reweighted least squares algorithms (glm is a special case), i.e. robust regression (in R: rlm).
I'm a big fan of R and of rlm in particular. I have to agree with your comments about Rpy, though I think their plans for it head in the right direction and give hope of a better interface. But, yes, I would support adding those capabilities to SciPy. I have Rpy accessing rlm in a little product right now, and it would certainly simply life!
-- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan@ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | -----------------------------------------------------------------------
Hi, On 2 Apr 2006, at 05:20, Jonathan Taylor wrote:
on this topic, as an honest-to-goodness statistician it might be nice to see more statistical modelling in scipy. i know Rpy exists, but the interface is not very pythonic.
I am not a statistician, but I do a fair amount of stats nonetheless. Enough that I wish there were some basic stats functions in "vanilla" python!
i have some "home-brew" modules for linear regression, formula building (something like R's) and a few other things. if it went into something like scipy, it might gain from the criticisms of others....
is there any interest in making the equivalent of a
scipy.stats.models
module?
My concern is that this is effort re-inventing a wheel. Would it not be better to improve the R interface rather than effectively duplicating functionality that's already in R (a Free, and high quality piece of software)? Matthew -- Matthew Vernon MA VetMB LGSM MRCVS Farm Animal Epidemiology and Informatics Unit Department of Veterinary Medicine, University of Cambridge http://www.cus.cam.ac.uk/~mcv21/
Hi,
My concern is that this is effort re-inventing a wheel. Would it not be better to improve the R interface rather than effectively duplicating functionality that's already in R (a Free, and high quality piece of software)?
Well, I think the argument would be that if you are writing software in Python that needs this functionality, then making R and RPy a dependency is a burden, even if RPy is working perfectly on all platforms. It also means that NumPy will be less likely to attract statisticians, if you still require R for your statistical processing. Best, Matthew
My concern is that this is effort re-inventing a wheel. Would it not be better to improve the R interface rather than effectively duplicating functionality that's already in R (a Free, and high quality piece of software)?
On Mon, 3 Apr 2006, Matthew Brett apparently wrote:
Well, I think the argument would be that if you are writing software in Python that needs this functionality, then making R and RPy a dependency is a burden, even if RPy is working perfectly on all platforms. It also means that NumPy will be less likely to attract statisticians, if you still require R for your statistical processing.
It would also mean that using numpy for stats would involve GPL'd rather than free software. ;-) Cheers, Alan Isaac
On 4/3/06, Matthew Vernon <matthew@sel.cam.ac.uk> wrote:
My concern is that this is effort re-inventing a wheel. Would it not be better to improve the R interface rather than effectively duplicating functionality that's already in R (a Free, and high quality piece of software)?
I strongly disagree with this approach. I'm not interested in dropping back to a 3rd party application to do sophisticated statistical analysis. We should be able to do it in python, and do it well. I specifically chose python to implement my Bayesian analysis software because it is faster, more flexible, and more user-friendly than R. In fact, all of scipy could have been replicated in R if desired, but that would be a bad choice. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us
Chris Fonnesbeck wrote:
On 4/3/06, *Matthew Vernon* <matthew@sel.cam.ac.uk <mailto:matthew@sel.cam.ac.uk>> wrote:
My concern is that this is effort re-inventing a wheel. Would it not be better to improve the R interface rather than effectively duplicating functionality that's already in R (a Free, and high quality piece of software)?
I strongly disagree with this approach. I'm not interested in dropping back to a 3rd party application to do sophisticated statistical analysis. We should be able to do it in python, and do it well. I specifically chose python to implement my Bayesian analysis software because it is faster, more flexible, and more user-friendly than R. In fact, all of scipy could have been replicated in R if desired, but that would be a bad choice.
I'm of this opinion as well. Certainly we can learn from R (especially regarding interfaces and useful functions to implement). -Travis
participants (8)
-
Alan G Isaac -
Alan Jackson -
Chris Fonnesbeck -
Jonathan Taylor -
Matthew Brett -
Matthew Vernon -
Robert Kern -
Travis Oliphant