[Numpy-discussion] Memory hungry reduce ops in Numpy
amueller at ais.uni-bonn.de
Tue Nov 15 13:17:58 EST 2011
On 11/15/2011 07:03 PM, Gael Varoquaux wrote:
> On Tue, Nov 15, 2011 at 05:57:14PM +0000, Robert Kern wrote:
>> Actually, last time I suggested it, it was brought up that the online
>> algorithms can be worse numerically. I'll try to find the thread.
> Indeed, especially for smallish datasets where the memory overhead is not
> an issue. I think that this is a common situation for which there is not
> _one_ good algorithm to solve the problem. IMHO the best way forward is
> to propose several options to the user, either with several function, or
> with a keyword argument.
I thought it would be possible to do the "two pass algorithm"
(as wikipedia calls it) without copying the dataset.
I guess I didn't really think it through.
That should be possible doing a custom reduce operation, right?
I don't know in how far this would have numerical problems.
More information about the NumPy-Discussion