PyArray_BASE equivalent in python
Hi, I as the title says, I am looking for a way to set in python the base of an ndarray to an object. Use case is porting qimage2ndarray to PySide where I want to do something like: In [1]: from PySide import QtGui In [2]: image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg') In [3]: import numpy as np In [4]: a = np.frombuffer(image.bits()) --> I would like to do something like: In [5]: a.base = image --> to avoid situations such as: In [6]: del image In [7]: a Segmentation fault (core dumped) The current implementation of qimage2ndarray uses a C function to do PyArray_BASE(sipRes) = image; Py_INCREF(image); But I want to avoid having to install compilers, headers etc on target machines of my code just for these two lines of code. Thanks, P
On Tue, Nov 26, 2013 at 11:54 AM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
Hi,
I as the title says, I am looking for a way to set in python the base of an ndarray to an object.
Use case is porting qimage2ndarray to PySide where I want to do something like:
In [1]: from PySide import QtGui
In [2]: image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
In [3]: import numpy as np
In [4]: a = np.frombuffer(image.bits())
--> I would like to do something like: In [5]: a.base = image
--> to avoid situations such as: In [6]: del image
In [7]: a Segmentation fault (core dumped)
This is a bug in PySide -- the buffer object returned by image.bits() needs to hold a reference to the original image. Please report a bug to them. You will also get a segfault from code that doesn't use numpy at all, by doing things like: bits = image.bits() del image <anything involving the bits object> As a workaround, you can write a little class with an __array_interface__ attribute that points to the image's contents, and then call np.asarray() on this object. The resulting array will have your object as its .base, and then your object can hold onto whatever references it wants. -n
Brilliant thanks, I will try out the "little class" approach. On 11/26/2013 08:03 PM, Nathaniel Smith wrote:
On Tue, Nov 26, 2013 at 11:54 AM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
Hi,
I as the title says, I am looking for a way to set in python the base of an ndarray to an object.
Use case is porting qimage2ndarray to PySide where I want to do something like:
In [1]: from PySide import QtGui
In [2]: image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
In [3]: import numpy as np
In [4]: a = np.frombuffer(image.bits())
--> I would like to do something like: In [5]: a.base = image
--> to avoid situations such as: In [6]: del image
In [7]: a Segmentation fault (core dumped) This is a bug in PySide -- the buffer object returned by image.bits() needs to hold a reference to the original image. Please report a bug to them. You will also get a segfault from code that doesn't use numpy at all, by doing things like:
bits = image.bits() del image <anything involving the bits object>
As a workaround, you can write a little class with an __array_interface__ attribute that points to the image's contents, and then call np.asarray() on this object. The resulting array will have your object as its .base, and then your object can hold onto whatever references it wants.
-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
I probably did something wrong, but it does not work how I tried it. I am not sure if you meant it like this, but I tried to subclass from ndarray first, but then I do not have access to __array_interface__. Is this what you had in mind? from PySide import QtGui import numpy as np class myArray(): def __init__(self, shape, bits, strides): self.__array_interface__ = \ {'data': bits, 'typestr': '<i32', 'descr': [('', '<f8')], 'shape': shape, 'strides': strides, 'version': 3} image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg') b = myArray((image.width(), image.height()), image.bits(), (image.bytesPerLine(), 4)) b = np.asarray(b) b.base #<read-write buffer ptr 0x7fd744c4b010, size 1478400 at 0x264e9f0> del image b # booom # On 11/26/2013 08:12 PM, Peter Rennert wrote:
Brilliant thanks, I will try out the "little class" approach.
On 11/26/2013 08:03 PM, Nathaniel Smith wrote:
On Tue, Nov 26, 2013 at 11:54 AM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
Hi,
I as the title says, I am looking for a way to set in python the base of an ndarray to an object.
Use case is porting qimage2ndarray to PySide where I want to do something like:
In [1]: from PySide import QtGui
In [2]: image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
In [3]: import numpy as np
In [4]: a = np.frombuffer(image.bits())
--> I would like to do something like: In [5]: a.base = image
--> to avoid situations such as: In [6]: del image
In [7]: a Segmentation fault (core dumped) This is a bug in PySide -- the buffer object returned by image.bits() needs to hold a reference to the original image. Please report a bug to them. You will also get a segfault from code that doesn't use numpy at all, by doing things like:
bits = image.bits() del image <anything involving the bits object>
As a workaround, you can write a little class with an __array_interface__ attribute that points to the image's contents, and then call np.asarray() on this object. The resulting array will have your object as its .base, and then your object can hold onto whatever references it wants.
-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 26, 2013 at 9:37 PM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
I probably did something wrong, but it does not work how I tried it. I am not sure if you meant it like this, but I tried to subclass from ndarray first, but then I do not have access to __array_interface__. Is this what you had in mind?
from PySide import QtGui import numpy as np
class myArray(): def __init__(self, shape, bits, strides):
You need to pass in the image as well and keep a reference to it.
self.__array_interface__ = \ {'data': bits, 'typestr': '<i32', 'descr': [('', '<f8')], 'shape': shape, 'strides': strides, 'version': 3}
Most of these are wrong. Something like the following should suffice: class QImageArray(object): def __init__(self, qimage): shape = (qimage.height(), qimage.width(), -1) # Generate an ndarray from the image bits and steal its # __array_interface__ information. arr = np.frombuffer(qimage.bits(), dtype=np.uint8).reshape(shape) self.__array_interface__ = arr.__array_interface__ # Keep the QImage alive. self.qimage = qimage -- Robert Kern
On Tue, Nov 26, 2013 at 1:37 PM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
I probably did something wrong, but it does not work how I tried it. I am not sure if you meant it like this, but I tried to subclass from ndarray first, but then I do not have access to __array_interface__. Is this what you had in mind?
from PySide import QtGui import numpy as np
class myArray(): def __init__(self, shape, bits, strides): self.__array_interface__ = \ {'data': bits, 'typestr': '<i32', 'descr': [('', '<f8')], 'shape': shape, 'strides': strides, 'version': 3}
You need this object to also hold a reference to the image object -- the idea is that so long as the array lives it will hold a ref to this object in .base, and then this object holds the image alive. But...
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
b = myArray((image.width(), image.height()), image.bits(), (image.bytesPerLine(), 4)) b = np.asarray(b)
b.base #<read-write buffer ptr 0x7fd744c4b010, size 1478400 at 0x264e9f0>
...this isn't promising, since it suggests that numpy is cleverly cutting out the middle-man when you give it a buffer object, since it knows that buffer objects are supposed to actually take care of memory management. You might have better luck using the raw pointer two-tuple form for the "data" field. You can't get these pointers directly from a buffer object, but numpy will give them to you. So you can use something like "data": np.asarray(bits).__array_interface__["data"] -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
Btw, I just wanted to file a bug at PySide, but it might be alright at their end, because I can do this: from PySide import QtGui image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg') a = image.bits() del image a #<read-write buffer ptr 0x7f5fe0034010, size 1478400 at 0x3c1a6b0> On 11/26/2013 09:37 PM, Peter Rennert wrote:
I probably did something wrong, but it does not work how I tried it. I am not sure if you meant it like this, but I tried to subclass from ndarray first, but then I do not have access to __array_interface__. Is this what you had in mind?
from PySide import QtGui import numpy as np
class myArray(): def __init__(self, shape, bits, strides): self.__array_interface__ = \ {'data': bits, 'typestr': '<i32', 'descr': [('', '<f8')], 'shape': shape, 'strides': strides, 'version': 3}
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
b = myArray((image.width(), image.height()), image.bits(), (image.bytesPerLine(), 4)) b = np.asarray(b)
b.base #<read-write buffer ptr 0x7fd744c4b010, size 1478400 at 0x264e9f0>
del image
b # booom #
On 11/26/2013 08:12 PM, Peter Rennert wrote:
Brilliant thanks, I will try out the "little class" approach.
On 11/26/2013 08:03 PM, Nathaniel Smith wrote:
On Tue, Nov 26, 2013 at 11:54 AM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
Hi,
I as the title says, I am looking for a way to set in python the base of an ndarray to an object.
Use case is porting qimage2ndarray to PySide where I want to do something like:
In [1]: from PySide import QtGui
In [2]: image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
In [3]: import numpy as np
In [4]: a = np.frombuffer(image.bits())
--> I would like to do something like: In [5]: a.base = image
--> to avoid situations such as: In [6]: del image
In [7]: a Segmentation fault (core dumped) This is a bug in PySide -- the buffer object returned by image.bits() needs to hold a reference to the original image. Please report a bug to them. You will also get a segfault from code that doesn't use numpy at all, by doing things like:
bits = image.bits() del image <anything involving the bits object>
As a workaround, you can write a little class with an __array_interface__ attribute that points to the image's contents, and then call np.asarray() on this object. The resulting array will have your object as its .base, and then your object can hold onto whatever references it wants.
-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 26, 2013 at 2:55 PM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
Btw, I just wanted to file a bug at PySide, but it might be alright at their end, because I can do this:
from PySide import QtGui
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
a = image.bits()
del image
a #<read-write buffer ptr 0x7f5fe0034010, size 1478400 at 0x3c1a6b0>
That just means that the buffer still has a pointer to the QImage's old memory. It doesn't mean that following that pointer won't crash. Try str(a) or something that actually touches the buffer contents... -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
First, sorry for not responding to your other replies, there was a jam in Thunderbird and I did not receive your answers. The bits() seem to stay alive after deleting the image: from PySide import QtGui image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg') a = image.bits() del image b = str(a) Works. But it might still be a Garbage collector thing and it might break later. I do not know enough about Python and its buffers to know what to expect here. As a solution I have done something similar as it was proposed earlier, just that I derived from ndarray and kept the QImage reference it it: from PySide import QtGui as _qt import numpy as _np class MemoryTie(np.ndarray): def __new__(cls, image): # Retrieving parameters for _np.ndarray() dims = [image.height(), image.width()] strides = [[] for i in range(2)] strides[0] = image.bytesPerLine() bits = image.bits() if image.format() == _qt.QImage.Format_Indexed8: dtype = _np.uint8 strides[1] = 1 elif image.format() == _qt.QImage.Format_RGB32 \ or image.format() == _qt.QImage.Format_ARGB32 \ or image.format() == _qt.QImage.Format_ARGB32_Premultiplied: dtype = _np.uint32 strides[1] = 4 elif image.format() == _qt.QImage.Format_Invalid: raise ValueError("qimageview got invalid QImage") else: raise ValueError("qimageview can only handle 8- or 32-bit QImages") # creation of ndarray obj = _np.ndarray(dims, _np.uint32, bits, 0, strides, 'C').view(cls) obj._image = image return obj Thanks all for your help, P On 11/26/2013 10:58 PM, Nathaniel Smith wrote:
Btw, I just wanted to file a bug at PySide, but it might be alright at their end, because I can do this:
from PySide import QtGui
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
a = image.bits()
del image
a #<read-write buffer ptr 0x7f5fe0034010, size 1478400 at 0x3c1a6b0> That just means that the buffer still has a pointer to the QImage's
On Tue, Nov 26, 2013 at 2:55 PM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote: old memory. It doesn't mean that following that pointer won't crash. Try str(a) or something that actually touches the buffer contents...
On Wed, Nov 27, 2013 at 2:16 PM, Peter Rennert <p.rennert@cs.ucl.ac.uk> wrote:
As a solution I have done something similar as it was proposed earlier, just that I derived from ndarray and kept the QImage reference it it:
from PySide import QtGui as _qt import numpy as _np
class MemoryTie(np.ndarray): def __new__(cls, image): # Retrieving parameters for _np.ndarray() dims = [image.height(), image.width()]
strides = [[] for i in range(2)] strides[0] = image.bytesPerLine()
bits = image.bits()
if image.format() == _qt.QImage.Format_Indexed8: dtype = _np.uint8 strides[1] = 1 elif image.format() == _qt.QImage.Format_RGB32 \ or image.format() == _qt.QImage.Format_ARGB32 \ or image.format() == _qt.QImage.Format_ARGB32_Premultiplied: dtype = _np.uint32 strides[1] = 4 elif image.format() == _qt.QImage.Format_Invalid: raise ValueError("qimageview got invalid QImage") else: raise ValueError("qimageview can only handle 8- or 32-bit QImages")
# creation of ndarray obj = _np.ndarray(dims, _np.uint32, bits, 0, strides, 'C').view(cls) obj._image = image
return obj
Don't do that. Use my recipe. You don't really want your RGBA tuples interpreted as uint32s, do you? from PySide import QtGui import numpy as np class QImageArray(object): def __init__(self, qimage): shape = (qimage.height(), qimage.width(), -1) # Generate an ndarray from the image bits and steal its # __array_interface__ information. arr = np.frombuffer(qimage.bits(), dtype=np.uint8).reshape(shape) self.__array_interface__ = arr.__array_interface__ # Keep the QImage alive. self.qimage = qimage def qimage2ndarray(qimage): return np.asarray(QImageArray(qimage)) qimage = QtGui.QImage('/Users/rkern/git/scipy/scipy/misc/tests/data/icon.png') arr = qimage2ndarray(qimage) del qimage print arr -- Robert Kern
participants (3)
-
Nathaniel Smith -
Peter Rennert -
Robert Kern