[Numpy-discussion] numpy ring buffer

Robert Kern robert.kern at gmail.com
Mon May 6 05:01:37 EDT 2013


On Mon, May 6, 2013 at 9:51 AM, Daniele Nicolodi <daniele at grinta.net> wrote:
> Hello,
>
> numpy arrays are great for interfacing python with libraries that expect
> continuous memory buffers for data passing.  However, libraries
> interfacing to data acquisition hardware often use those buffers as ring
> buffers where, once the buffer has been filled with data, new data will
> be written overwriting the data at the beginning of the buffer.  The
> application is supposed to have read this data meanwhile.
>
> Efficiently processing the data in the ring buffer requires addressing
> efficiently the content of the ring buffer.  In C or Cython this is very
> easy, just compute the wrap around when accessing the single elements of
> the array:
>
>   buffer = np.empty(size)
>   adc.setup(buffer)
>   start = 0
>   for n in adc.read():
>       # data processing
>       for i in range(start, start + n):
>           element = buffer[start % size]
>           ....
>       start += n
>
> My current approach to do the same thing in Python is to use the
> np.roll() function to "linearize" the buffer access:
>
>   buffer = np.empty(size)
>   adc.setup(buffer)
>   start = 0
>   for n in adc.read():
>       data = np.roll(buffer, -start)[:n]
>       start += n
>       # data processing
>       process(data)
>
> Since np.roll() returns a view on the array i suppose this is very
> efficient.  Does anyone have a better idea on how to do this?

np.roll() copies all of the data every time. It does not return a
view. Try a function like this instead:

[~]
|5> def ring_window(buffer, start, n):
..>     length = len(buffer)
..>     start %= length
..>     end = start + n
..>     if end <= length:
..>         window = buffer[start:end]
..>     else:
..>         end %= length
..>         window = np.concatenate((buffer[start:length], buffer[0:end]))
..>     return end, window
..>

[~]
|6> buffer = np.arange(20)

[~]
|7> start = 0

[~]
|8> for i in range(5):
..>     start, window = ring_window(buffer, start, 6)
..>     print start, window
..>
6 [0 1 2 3 4 5]
12 [ 6  7  8  9 10 11]
18 [12 13 14 15 16 17]
4 [18 19  0  1  2  3]
10 [4 5 6 7 8 9]

--
Robert Kern



More information about the NumPy-Discussion mailing list