Guys, Dave and I talked last night about standard deviation of profiles. Can anybody see how to convert this one-pass algorithm: http://www.cs.berkeley.edu/~mhoemmen/cs194/Tutorials/variance.pdf into a weighted-average method? My brain is not working today. -Matt
I can make it an upper triangular matrix multiply. You want a weighted sum because the numpy.array.sum function is super optimized, over looping which is slow? Are matrix multiplies similarly optimized? d. On Wed, Feb 4, 2009 at 8:35 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Guys,
Dave and I talked last night about standard deviation of profiles. Can anybody see how to convert this one-pass algorithm:
http://www.cs.berkeley.edu/~mhoemmen/cs194/Tutorials/variance.pdf
into a weighted-average method? My brain is not working today.
-Matt _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I can make it an upper triangular matrix multiply.
Okay, so that should work for decomposing across grids, right? We want to be able to cut this up and then do a single combine step at the end, after each grid is handled.
You want a weighted sum because the numpy.array.sum function is super optimized, over looping which is slow?
Well, sure, but more because what we're trying to do is: sum ( w_i * v_i ) / sum (w_i) so we'll need to weight the values to calculate std-dev, yes? -Matt
I vote do it the 2 pass way. I've been looking at this this morning, and what I keep coming up with (though I don't have a full solution) is that you're going to need to compute partial sums of the weight field. I think this pass through the weight field almost outweighs the advantage of using this single pass approach. d. On Wed, Feb 4, 2009 at 10:23 AM, david collins <antpuncher@gmail.com> wrote:
so we'll need to weight the values to calculate std-dev, yes?
I think I answered the wrong question. Ignore my first post, the UT matrix is irrelevant.
d.
Well, nuts. Okay, I'll take a look at this later today or tomorrow. Maybe we should make a special "calculate error" function? On Wed, Feb 4, 2009 at 12:47 PM, david collins <antpuncher@gmail.com> wrote:
I vote do it the 2 pass way. I've been looking at this this morning, and what I keep coming up with (though I don't have a full solution) is that you're going to need to compute partial sums of the weight field. I think this pass through the weight field almost outweighs the advantage of using this single pass approach.
d.
On Wed, Feb 4, 2009 at 10:23 AM, david collins <antpuncher@gmail.com> wrote:
so we'll need to weight the values to calculate std-dev, yes?
I think I answered the wrong question. Ignore my first post, the UT matrix is irrelevant.
d.
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Well, nuts. Okay, I'll take a look at this later today or tomorrow. Maybe we should make a special "calculate error" function?
Meh. I vote do the simple way, and if it looks like the person-hours in savings to increase the speed are worth the person-hours cost in figuring out a faster way (keeping in mind that two of us are trying to graduate) then do it. It's at *best* a factor of 2 in cost for something that's been pretty quick for me, so I don't see it as worth your time to speed it up. It's not an increase of accuracy to do sum( w_i * ( v_i - M )^2) / sum(w_i), I don't think, since you do the difference in the square. d.
On Wed, Feb 4, 2009 at 12:47 PM, david collins <antpuncher@gmail.com> wrote:
I vote do it the 2 pass way. I've been looking at this this morning, and what I keep coming up with (though I don't have a full solution) is that you're going to need to compute partial sums of the weight field. I think this pass through the weight field almost outweighs the advantage of using this single pass approach.
d.
On Wed, Feb 4, 2009 at 10:23 AM, david collins <antpuncher@gmail.com> wrote:
so we'll need to weight the values to calculate std-dev, yes?
I think I answered the wrong question. Ignore my first post, the UT matrix is irrelevant.
d.
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Okay, sounds good. I'll add it in tonight or tomorrow. On Wed, Feb 4, 2009 at 1:35 PM, david collins <antpuncher@gmail.com> wrote:
Well, nuts. Okay, I'll take a look at this later today or tomorrow. Maybe we should make a special "calculate error" function?
Meh. I vote do the simple way, and if it looks like the person-hours in savings to increase the speed are worth the person-hours cost in figuring out a faster way (keeping in mind that two of us are trying to graduate) then do it. It's at *best* a factor of 2 in cost for something that's been pretty quick for me, so I don't see it as worth your time to speed it up. It's not an increase of accuracy to do sum( w_i * ( v_i - M )^2) / sum(w_i), I don't think, since you do the difference in the square.
d.
On Wed, Feb 4, 2009 at 12:47 PM, david collins <antpuncher@gmail.com> wrote:
I vote do it the 2 pass way. I've been looking at this this morning, and what I keep coming up with (though I don't have a full solution) is that you're going to need to compute partial sums of the weight field. I think this pass through the weight field almost outweighs the advantage of using this single pass approach.
d.
On Wed, Feb 4, 2009 at 10:23 AM, david collins <antpuncher@gmail.com> wrote:
so we'll need to weight the values to calculate std-dev, yes?
I think I answered the wrong question. Ignore my first post, the UT matrix is irrelevant.
d.
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (2)
-
david collins -
Matthew Turk