[Python-ideas] Marvelous movable memoryview

[Christian Heimes]

Am 27.07.2012 03:37, schrieb Nick Coghlan:

> Thus, using views instead of copying really only starts to pay off
> once you're talking about comparatively large chunks of data:
>
>>>> x *= 1000
>>>> sys.getsizeof(x)
> 3033
>>>> sys.getsizeof(memoryview(x))
> 184

It can pay off A LOT. I've recently added PEP 3118 buffer support to my
smc.freeimage library (Cython wrapper of FreeImage and LCMS).

smc.freeimage needs 0.006 sec for the loop as it returns a writable
buffer of the internal pixel data (zero copy). PIL runs 1.165 sec as it
uses the old buffer interface and copies its data every time.

Benchmark code:
https://bitbucket.org/tiran/smc.freeimage/src/b97e13e9f04c/contrib/benchmark.py

RW_COUNT = 300
tiff = Image(TIFF)
start = time()
for i in xrange(RW_COUNT):
    arr = asarray(tiff)
    # change last BGR -> RGB
    arr = arr[..., ::-1]
    bytescale(arr, 64, 192)
end = time() - start

tiff = pil_open(TIFF)
tiff.load()
start = time()
for i in xrange(RW_COUNT):
    arr = asarray(tiff)
    bytescale(arr, 64, 192)
end = time() - start


> No, because making memorviews mutable would be a huge increase in
> complexity (and they're complex enough already - only with Stefan
> Krah's work in 3.3 have we finally worked most of the kinks out of the
> implementation).

+1 for Nick's -1

The new buffer interface and memoryview's are already very complex. I
haven't found a library yet that supports all edge cases. Even NumPy
doesn't support multidimensional and non-contiguous buffer with
suboffsets. I wanted to use a sub-offset of 3 on the last dimension with
a negative stride on the last dimension to convert BGR to RGB inside the
buffer code. It didn't work because NumPy ignore the information.

Christian