<br><br><div class="gmail_quote">On Tue, Nov 15, 2011 at 10:48 AM, Andreas Müller <span dir="ltr"><<a href="mailto:amueller@ais.uni-bonn.de">amueller@ais.uni-bonn.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<u></u>
<div text="#000000" bgcolor="#ffffff"><div><div></div><div class="h5">
On 11/15/2011 05:46 PM, Andreas Müller wrote:
<blockquote type="cite">
On 11/15/2011 04:28 PM, Bruce Southey wrote:
<blockquote type="cite">
On 11/14/2011 10:05 AM, Andreas Müller wrote:
<blockquote type="cite">
<pre>On 11/14/2011 04:23 PM, David Cournapeau wrote:
</pre>
<blockquote type="cite">
<pre>On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller
<a href="mailto:amueller@ais.uni-bonn.de" target="_blank"><amueller@ais.uni-bonn.de></a> wrote:
</pre>
<blockquote type="cite">
<pre>Hi everybody.
When I did some normalization using numpy, I noticed that numpy.std uses
more ram than I was expecting.
A quick google search gave me this:
<a href="http://luispedro.org/software/ncreduce" target="_blank">http://luispedro.org/software/ncreduce</a>
The site claims that std and other reduce operations are implemented
naively with many temporaries.
Is that true? And if so, is there a particular reason for that?
This issues seems quite easy to fix.
In particular the link I gave above provides code.
</pre>
</blockquote>
<pre>The code provided only implements a few special cases: being more
efficient in those cases only is indeed easy.
</pre>
</blockquote>
<pre>I am particularly interested in the std function.
Is this implemented as a separate function or an instantiation
of a general reduce operations?
_______________________________________________
NumPy-Discussion mailing list
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a>
</pre>
</blockquote>
The<span> 'On-line
algorithm</span>' (<a href="http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm" target="_blank">http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm)</a>
could save you storage. I would presume if you know cython that
you can probably make it quick as well (to address the loop over
the data).<br>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:sans-serif;font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-size:13px;line-height:19px"></span></span><br>
</blockquote>
<br>
My question was more along the lines of "why doesn't numpy do the
online algorithm".<br>
<br>
</blockquote></div></div>
To be more precise, even not using the online version but computing
E(X^2) and E(X)^2 would be good.<br>
It seems numpy centers the whole dataset. Otherwise I can't explain
why the memory needed should depend<br>
on the number of examples.<br></div></blockquote><div><br><br>Yes, that is what it is doing. See line 63 in the function _var(), which is called by _std():<br> <a href="https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py">https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py</a><br>
<br><br>Warren<br><br> <br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div text="#000000" bgcolor="#ffffff">
</div>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>