Re: [Numpy-discussion] numpy FFT memory accumulation
At 11:55 PM 10/31/2007, Travis wrote:
Ray S wrote:
I am using fftRes = abs(fft.rfft(data_array[end-2**15:end]))
At first glance, I would say that I don't expect memory to be growing here, so it looks like a problem with rfft that deserves looking into.
I saw that Numeric did also (I still use Numeric for smaller array speed) but much more slowly. I will try to repeat with a small demo and post.
Does data_array keep growing?
no, it is a 64k circular buffer Which reminds me, I've been wanting to try to build a Numeric circular buffer object; one that, when sliced or assigned to, auto-magically wraps as needed, exposing only an extra pointer attribute for the "end". I currently always do it with Python if-s and pointer vars for these products that only do "real time" data analysis.
From: "Anne Archibald" <peridot.faceted@gmail.com> If the range is *really* small, you can try using a DFT - sometimes that is fast enough, and gives you just the bins you're curious about.
I've considered weave'ing a simple sine transform with specified range, but until I do and test I won't know if my own implementation is any faster than just the FFTPACK.
If the range is bigger than that, but still a small fraction of the FFT size, you can do some tricks where you band-pass filter the data ... There are also "zoom fft" and "chirp-z" techniques which are supposed to give you only part of the FFT, but the wisdom is that unless you want less than a few percent of the data points you're better just FFTing and throwing lots of data away.
I've tried zoom before; the issue was just that - 2 FFTs and a shift or convolution eats a lot of CPU cycles and falls behind the real time data. The range of interest in the Fourrier domain is small, 3kHz-7kHz. The sample rate is high for precise phase information. I've got some more testing to do, it seems. Thanks, Ray -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.15.15/1101 - Release Date: 10/31/2007 10:06 AM
On 11/1/07, Ray Schumacher <subscriber100@rjs.org> wrote:
At 11:55 PM 10/31/2007, Travis wrote:
Ray S wrote:
I am using fftRes = abs(fft.rfft(data_array[end-2**15:end]))
At first glance, I would say that I don't expect memory to be growing here, so it looks like a problem with rfft that deserves looking into.
I saw that Numeric did also (I still use Numeric for smaller array speed) but much more slowly. I will try to repeat with a small demo and post.
Does data_array keep growing?
no, it is a 64k circular buffer Which reminds me, I've been wanting to try to build a Numeric circular buffer object; one that, when sliced or assigned to, auto-magically wraps as needed, exposing only an extra pointer attribute for the "end". I currently always do it with Python if-s and pointer vars for these products that only do "real time" data analysis.
In Python, collections.deque makes a pretty good circular buffer. Numpy will make an array out of it, which involves a copy, but it might be better than what you are doing now. In [14]: from collections import deque In [15]: a = deque([1,2,3,4]) In [16]: fft(a) Out[16]: array([ 10.+0.j, -2.+2.j, -2.+0.j, -2.-2.j]) In [17]: a.append(5) In [18]: a.popleft() Out[18]: 1 In [19]: fft(a) Out[19]: array([ 14.+0.j, -2.+2.j, -2.+0.j, -2.-2.j]) Chuck
participants (2)
-
Charles R Harris
-
Ray Schumacher