numpy FFT memory accumulation
I am using fftRes = abs(fft.rfft(data_array[end2**15:end])) to do running analysis on streaming data. The N never changes. It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec. (Interestingly, Numeric FFT accumulates much slower..) (Commenting out that one line stops memory growth.) What can one do to alleviate this? Can I del() some numpy object or such? It's a bit of an issue for a program that needs to run for weeks. It's purpose is simply to argmax() the largest bin, which always falls within a small range  do I have another, better option than fft? Cheers, Ray Schumacher Blue Cove Interactive
Ray S wrote:
I am using fftRes = abs(fft.rfft(data_array[end2**15:end])) to do running analysis on streaming data. The N never changes. It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec. (Interestingly, Numeric FFT accumulates much slower..) (Commenting out that one line stops memory growth.)
At first glance, I would say that I don't expect memory to be growing here, so it looks like a problem with rfft that deserves looking into. Does data_array keep growing? Travis O.
On 31/10/2007, Ray S <subscriber100@rjs.org> wrote:
I am using fftRes = abs(fft.rfft(data_array[end2**15:end])) to do running analysis on streaming data. The N never changes. It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec. (Interestingly, Numeric FFT accumulates much slower..) (Commenting out that one line stops memory growth.)
What can one do to alleviate this? Can I del() some numpy object or such? It's a bit of an issue for a program that needs to run for weeks.
It's purpose is simply to argmax() the largest bin, which always falls within a small range  do I have another, better option than fft?
If the range is *really* small, you can try using a DFT  sometimes that is fast enough, and gives you just the bins you're curious about. If the range is bigger than that, but still a small fraction of the FFT size, you can do some tricks where you bandpass filter the data (with a FIR filter, say), downsample (which aliases frequencies downward), and then FFT. Concretely, if your data is sampled at 80ksamp/s and you're taking 160ksamp FFTs, you're getting the spectrum up to 35 kHz with a resolution of 0.5 Hz. If all you care about is the spectral region from 5 kHz to 6 kHz but you still need the 0.5 Hz spectral resolution, you can run a bandpass filter to throw out everything outside (say) 4 to 7 kHz, then downsample to 8 ksamp/s; the 4 to 7kHz region will appear mirrored in the 1 to 4 kHz region of the new FFTs, which will each be a tenth the size. There are also "zoom fft" and "chirpz" techniques which are supposed to give you only part of the FFT, but the wisdom is that unless you want less than a few percent of the data points you're better just FFTing and throwing lots of data away. Anne
participants (3)

Anne Archibald

Ray S

Travis E. Oliphant