lstsq: masked arrays, weights, scaling, and covariance
Hi All, I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run. Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal http://tinyurl.com/3sbwgxy, something we might want to be consistent with. Thoughts? Chuck
On Sat, Sep 17, 2011 at 2:52 PM, Charles R Harris
Hi All,
I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run.
Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal, something we might want to be consistent with.
Thoughts?
What's the speed penalty if we just want to use numpy/scipy linalg as a library and don't need any of the extra features? As some of the discussions have shown it can be pretty expensive to use linalg in loops. Josef
Chuck
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sat, Sep 17, 2011 at 1:40 PM,
Hi All,
I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run.
Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but
On Sat, Sep 17, 2011 at 2:52 PM, Charles R Harris
wrote: they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal, something we might want to be consistent with.
Thoughts?
What's the speed penalty if we just want to use numpy/scipy linalg as a library and don't need any of the extra features?
As some of the discussions have shown it can be pretty expensive to use linalg in loops.
I think the features should be optional and not the defaults, although masked arrays will always need some special treatment. Chuck
I think this sounds like a great idea. The lowest level that makes sense is the correct place for them. -Travis On Sep 17, 2011, at 1:52 PM, Charles R Harris wrote:
Hi All,
I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run.
Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal, something we might want to be consistent with.
Thoughts?
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
--- Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
participants (3)
-
Charles R Harris
-
josef.pktd@gmail.com
-
Travis Oliphant