[Numpy-discussion] convolving (or correlating) with sliding windows

Jonathan Hilmer jkhilmer at gmail.com
Tue Feb 15 13:17:54 EST 2011


I'm sorry that I don't have some example code for you, but you
probably need to break down the problem if you can't fit it into
memory: http://en.wikipedia.org/wiki/Overlap-add_method

Jonathan

On Tue, Feb 15, 2011 at 10:27 AM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 15, 2011 at 11:42 AM, Davide Cittaro
> <davide.cittaro at ifom-ieo-campus.it> wrote:
>> Hi all,
>> I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those.
>> The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this:
>>
>> - given a window W and a step size S
>> - given data1 and data2
>> - pad with zeros data1 and data2 by adding W/2 0-arrays
>> - get the np.correlation like
>>
>> y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in  np.arange(0, len(data1) - W, S)]).ravel()
>>
>> instead of np.correlate(data1, data2, mode='same')
>
> If data2 is of similar length as data1, then you should use
> fftconvolve which is much faster for long arrays than np.correlate or
> convolve. I'm not sure about the rest.
>
> Josef
>
>
>> - interpolate the correlation using scipy.interpolate
>> ?
>>
>> Thanks
>> d
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



More information about the NumPy-Discussion mailing list