Re: [Python-Dev] Extended Buffer Interface/Protocol
(cc'ing back to Python-dev; the original reply was intended for it by I had an email malfunction.) Travis Oliphant wrote:
Carl Banks wrote:
3. Allow getbuffer to return an array of "derefence offsets", one for each dimension. For a given dimension i, if derefoff[i] is nonnegative, it's assumed that the current position (base pointer + indexing so far) is a pointer to a subarray, and derefoff[i] is the offest in that array where the current position goes for the next dimension. If derefoff[i] is negative, there is no dereferencing. Here is an example of how it'd work:
This sounds interesting, but I'm not sure I totally see it. I probably need a picture to figure out what you are proposing.
I'll get on it sometime. For now I hope an example will do.
The derefoff sounds like some-kind of offset. Is that enough? Why not just make derefoff[i] == 0 instead of negative?
I may have misunderstood something. I had thought the values exported by getbuffer could change as the view narrowed, but I'm not sure if it's the case now. I'll assume it isn't for now, because it simplifies things and demonstrates the concept better. Let's start from the beginning. First, change the prototype to this: typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) "isptr" is a flag indicating whether, for a certain dimension, the positision we've strided to so far is a pointer that should be followed before proceeding with the rest of the strides. Now here's what a general "get_item_pointer" function would look like, given a set of indices: void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides, Py_ssize_t* derefoff, Py_ssize_t *indices) { char* pointer = (char*)buf; int i; for (i = 0; i < ndim; i++) { pointer += strides[i]*indices[i]; if (isptr[i]) { pointer = *(char**)pointer; } } return (void*)pointer; }
I don't fully understand the PIL example you gave.
Yeah. How about more details. Here is a hypothetical image data object structure: struct rgba { unsigned char r, g, b, a; }; struct ImageObject { PyObject_HEAD; ... struct rgba** lines; Py_ssize_t height; Py_ssize_t width; Py_ssize_t shape_array[2]; Py_ssize_t stride_array[2]; Py_ssize_t view_count; }; "lines" points to malloced 1-D array of (struct rgba*). Each pointer in THAT block points to a seperately malloced array of (struct rgba). Got that? In order to access, say, the red value of the pixel at x=30, y=50, you'd use "lines[50][30].r". So what does ImageObject's getbuffer do? Leaving error checking out: PyObject* getbuffer(PyObject *self, void **buf, Py_ssize_t *len, int *writeable, char **format, int *ndims, Py_ssize_t **shape, Py_ssize_t **strides, int **isptr) { static int _isptr[2] = { 1, 0 }; *buf = self->lines; *len = self->height*self->width; *writable = 1; *ndims = 2; self->shape_array[0] = height; self->shape_array[1] = width; *shape = &self->shape_array; self->stride_array[0] = sizeof(struct rgba*); /* yep */ self->stride_array[1] = sizeof(struct rgba); *strides = &self->stride_array; *isptr = _isptr; self->view_count ++; /* create and return view object here, but for what? */ } There are three essential differences from a regular, contiguous array. 1. buf is set to point at the array of pointers, not directly to the data. 2. The isptr thing. isptr[0] is true to indicate that the first dimension is an array of pointers, not the actual data. 3. stride[0] is sizeof(struct rgba*), not self->width*sizeof(struct rgba) like it would be for a contiguous array. This is because your first stride is through an array of pointers, not the data itself. So let's examine what "get_item_pointer" above will do given these values. Once again, we're looking for the pixel at x=30, y=50. First, we set pointer to buf, that is, self->lines. Then we take the first stride: we add index[0]+strides[0], that is, 50*4=200, to poitner. pointer now equals &self->lines[50]. Now, we check isptr[0]. We see that it is true. Thus, the position we've strided to is, in fact, a pointer to a subarray where the actual data is. So we follow it: pointer = *pointer. pointer now equals self->lines[50] which equals &self->lines[50][0]. Next dimension. We take the second stride: we add index[1]+strides[1], that is, 30*4=120, to pointer. pointer now equals &self->lines[50][30]. Now, we check isptr[1]. It's false. No dereferencing this step. We're done. Return pointer.
By the way, has anyone signed up to modify the standard library modules? I could do those when the protocol is finalized. And if you're implementing the new buffer protocol in 2.6 (while deprecating but not removing the old protocol, I presume), will the modules also be updated for 2.6?
Nobody has signed up for anything. I'm willing for anyone to help. Many of the standard library modules will need to be modified. And yes, I do want to implement the new protocol for 2.6 (adding it to the current one). Updating the modules for 2.6 would not be high priority (except the struct module), but is a desirable.
Ok, then, consider me available for it.
Thanks for the ideas.
Ok, I have two questions, now. First, I'm not sure why getbuffer needs to return a view object. I expect most views of data to be created separately--for instance, a view of an image is likely to be created in Python using something like this: imgview = ImageView(image,(left,right),(top,bottom)) I'd expect the ImageView object would call getbuffer and use the data returned in buf, len, writable, etc., and would have no need for a type-specific view object. Furthermore, I would expect in many cases different views are desirable, and some cases where the viewer is unknown to the exporter. And, if it does have to return a view for some reason, why bother returning buf, len, and friends in the function? Just return those values in the view object. Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export? Naively, I'd expect the "strides" and "shape" array to have rearranged indices, but it looks like you might be trying to get rid of this complexity. The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?) Carl Banks
Carl Banks wrote:
We're done. Return pointer.
Thank you for this detailed example. I will have to parse it in more depth but I think I can see what you are suggesting.
First, I'm not sure why getbuffer needs to return a view object.
The view object in your case would just be the ImageObject. The reason I was thinking the function should return "something" is to provide more flexibility in what a view object actually is. I've also been going back and forth between explicitly passing all this information around or placing it in an actual view-object. In some sense, a view object is a NumPy array in my mind. But, with the addition of isptr we are actually expanding the memory abstraction of the view object beyond an explicit NumPy array. In the most common case, I envisioned the view object would just be the object itself in which case it doesn't actually have to be returned. While returning the view object would allow unspecified flexibilty in the future, it really adds nothing to the current vision. We could add a view object separately as an abstract API on top of the buffer interface.
Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export?
Basically, you could not alter the internal representation of the object while views which depended on those values were around. In NumPy, a transposed array actually creates a new NumPy object that refers to the same data but has its own shape and strides arrays. With the new buffer protocol, the NumPy array would not be able to alter it's shape/strides/or reallocate its data areas while views were being held by other objects. With the shape and strides information, the format information, and the data buffer itself, there are actually several pieces of memory that may need to be protected because they may be shared with other objects. This makes me wonder if releasebuffer should contain an argument which states whether or not to release the memory, the shape and strides information, the format information, or all of it. Having such a thing as a view object would actually be nice because it could hold on to a particular view of data with a given set of shape and strides (whose memory it owns itself) and then the exporting object would be free to change it's shape/strides information as long as the data did not change.
The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?)
I'm not sure what you mean, offsets are handled by changing the starting location of the pointer to the buffer. -Travis
Travis Oliphant wrote:
Carl Banks wrote:
We're done. Return pointer.
Thank you for this detailed example. I will have to parse it in more depth but I think I can see what you are suggesting.
First, I'm not sure why getbuffer needs to return a view object.
The view object in your case would just be the ImageObject.
ITSM that we are using the word "view" very differently. Consider this example: A = zeros((100,100)) B = A.transpose() In this scenario, A would be the exporter object, I think we both would call it that. When I use the word "view", I'm referring to B. However, you seem to be referring to the object returned by A.getbuffer, right? What term have you been using to refer to B? Obviously, it would help the discussion if we could get our terminology straight. (Frankly, I don't agree with your usage; it doesn't agree with other uses of the word "view". For example, consider the proposed Python 3000 dictionary views: D = dict() V = D.items() Here, V is the view, and it's analogous to B in the above example.) I'd suggest the object returned by A.getbuffer should be called the "buffer provider" or something like that. For the sake of discussion, I'm going to avoid the word "view" altogether. I'll call A the exporter, as before. B I'll refer to as the requestor. The object returned by A.getbuffer is the provider.
The reason I was thinking the function should return "something" is to provide more flexibility in what a view object actually is.
I've also been going back and forth between explicitly passing all this information around or placing it in an actual view-object. In some sense, a view object is a NumPy array in my mind. But, with the addition of isptr we are actually expanding the memory abstraction of the view object beyond an explicit NumPy array.
In the most common case, I envisioned the view object would just be the object itself in which case it doesn't actually have to be returned. While returning the view object would allow unspecified flexibilty in the future, it really adds nothing to the current vision.
We could add a view object separately as an abstract API on top of the buffer interface.
Having thought quite a bit about it, and having written several abortive replies, I now understand it and see the importance of it. getbuffer returns the object that you are to call releasebuffer on. It may or may not be the same object as exporter. Makes sense, is easy to explain. It's easy to see possible use cases for returning a different object. A hypothetical future incarnation of NumPy might shift the responsibility of managing buffers from NumPy array object to a hidden raw buffer object. In this scenario, the NumPy object is the exporter, but the raw buffer object the provider. Considering this use case, it's clear that getbuffer should return the shape and stride data independently of the provider. The raw buffer object wouldn't have that information; all it does is store a pointer and keep a reference count. Shape and stride is defined by the exporter.
Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export?
Basically, you could not alter the internal representation of the object while views which depended on those values were around.
In NumPy, a transposed array actually creates a new NumPy object that refers to the same data but has its own shape and strides arrays.
With the new buffer protocol, the NumPy array would not be able to alter it's shape/strides/or reallocate its data areas while views were being held by other objects.
But requestors could alter their own copies of the data, no? Back to the transpose example: B itself obviously can't use the same "strides" array as A uses. It would have to create its own strides, right? So, what if someone takes a slice out of B? When calling B.getbuffer, does it get B's strides, or A's? I think it should get B's. After all, if you're taking a slice of B, don't you care about the slicing relative to B's axes? I can't really think of a use case for exporting A's stride data when you take a slice of B, and it doesn't seem to simplify memory management, because B has to make it's own copies anyways.
With the shape and strides information, the format information, and the data buffer itself, there are actually several pieces of memory that may need to be protected because they may be shared with other objects. This makes me wonder if releasebuffer should contain an argument which states whether or not to release the memory, the shape and strides information, the format information, or all of it.
Here's what I think: the lock should only apply to the buffer itself, and not to shape and stride data at all. If the requestor wants to keep its own copies of the data, it would have to malloc its own storage for it. I expect that this would be very rare. As for the provider; I think that's between it the exporter. If the exporter and provider know about each other, they shouldn't have any problems managing memory together.
Having such a thing as a view object would actually be nice because it could hold on to a particular view of data with a given set of shape and strides (whose memory it owns itself) and then the exporting object would be free to change it's shape/strides information as long as the data did not change.
What I don't undestand is why it's important for the provider to retain this data. The requestor only needs the information when it's created; it will calculate its own versions of the data, and will not need the originals again, so no need to the provider to keep them around. Indeed, in the use case I described of the raw buffer object, the provider doesn't even know about the exporter's shape and strides.
The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?)
I'm not sure what you mean, offsets are handled by changing the starting location of the pointer to the buffer.
Yes, well, the single buffer problem isn't as well solved as I originally thought, so maybe it's best to focus on that first. But to anwser your question: you can't just change the starting location because there's hundreds of buffers. You'd either have to change the starting location of each one of them, which is highly undesirable, or to factor in an offset somehow. (This was, in fact, the point of the "derefoff" term in my original suggestion.) Anyways, despite the miscommunications so far, I now have a very good idea of what's going on. We definitely need to get terms straight. I agree that getbuffer should return an object. I think we need to think harder about the case when requestors re-export the buffer. (Maybe it's time to whip up some experimental objects?) Carl Banks
Carl Banks wrote:
It's easy to see possible use cases for returning a different object. A hypothetical future incarnation of NumPy might shift the responsibility of managing buffers from NumPy array object to a hidden raw buffer object. In this scenario, the NumPy object is the exporter, but the raw buffer object the provider.
But since the NumPy object has to know about the provider, it can simply pass the release call on to it if appropriate. I don't see how this case necessitates making the release call on a different object. I'm -1 on involving any other objects or returning object references from the buffer interface, unless someone can come up with a use case which actually *requires* this (as opposed to it just being something which might be "nice to have"). The buffer interface should be Blazingly Fast(tm), and messing with PyObject*s is not the way to get that.
This makes me wonder if releasebuffer should contain an argument which states whether or not to release the memory, the shape and strides information, the format information, or all of it.
Here's what I think: the lock should only apply to the buffer itself, and not to shape and stride data at all. If the requestor wants to keep its own copies of the data, it would have to malloc its own storage for it. I expect that this would be very rare.
Seems to me the lock should apply to *everything* returned by getbuffer. If the requestor is going to iterate over the data, and there are multiple dimensions, surely it's going to want to refer to the shape and stride info at regular intervals while it's doing that. Requiring it to make its own copy would be a burden. -- Greg
Greg Ewing wrote:
But since the NumPy object has to know about the provider, it can simply pass the release call on to it if appropriate. I don't see how this case necessitates making the release call on a different object.
I'm -1 on involving any other objects or returning object references from the buffer interface, unless someone can come up with a use case which actually *requires* this (as opposed to it just being something which might be "nice to have"). The buffer interface should be Blazingly Fast(tm), and messing with PyObject*s is not the way to get that.
The current proposal would be fast but would be more flexible for objects that don't have a memory representation that can be shared unless they create their own "sharing object" that perhaps copies the data into a contiguous chunk first. Objects which have memory which can be shared perfectly through the interface would simply pass themselves as the return value (after incrementing their "extant buffers" count by one).
Seems to me the lock should apply to *everything* returned by getbuffer. If the requestor is going to iterate over the data, and there are multiple dimensions, surely it's going to want to refer to the shape and stride info at regular intervals while it's doing that. Requiring it to make its own copy would be a burden.
There are two use cases that seem to be under discussion. 1) When you want to apply an algorithm to an arbitrary object that exposes the buffer interface 2) When you want to create an object that shares memory with another object exposing the buffer interface. These two use cases have slightly different needs. What I want to avoid is forcing the exporting object to be unable to change its shape and strides just because an object is using the memory for use case #2. I think the solution that states the shape and strides information are only guaranteed valid until the GIL is released is sufficent. Alternatively, one could release the shape and strides and format separately from the memory with a flag as a second argument to releasebuffer. -Travis
-- Greg
Carl Banks wrote:
Tr ITSM that we are using the word "view" very differently. Consider this example:
A = zeros((100,100)) B = A.transpose()
You are thinking of NumPy's particular use case. I'm thinking of a generic use case. So, yes I'm using the word view in two different contexts. In this scenario, NumPy does not even use the buffer interface. It knows how to transpose it's own objects and does so by creating a new NumPy object (with it's own shape and strides space) with a data buffer pointed to by "A". Yes, I use the word "view" for this NumPy usage, but only in the context of NumPy. In the PEP, I've been using the word "view" quite a bit more generically. So, I don't think this is a good example because A.transpose() will never call getbuffer of the A object (it will instead use the known structure of NumPy directly). So, let's talk about the generic situation instead of the NumPy specific one.
I'd suggest the object returned by A.getbuffer should be called the "buffer provider" or something like that.
I don't care what we call it. I've been using the word "view" because of the obvious analogy to my use of view in NumPy. When I had envisioned returning an actual object very similar to a NumPy array from the buffer interface it made a lot of sense to call it a view. Now, I'm fine to call it "buffer provider"
For the sake of discussion, I'm going to avoid the word "view" altogether. I'll call A the exporter, as before. B I'll refer to as the requestor. The object returned by A.getbuffer is the provider.
Fine. Let's use that terminology since it is new and not cluttered by other uses in other contexts.
Having thought quite a bit about it, and having written several abortive replies, I now understand it and see the importance of it. getbuffer returns the object that you are to call releasebuffer on. It may or may not be the same object as exporter. Makes sense, is easy to explain.
Yes, that's exactly all I had considered it to be. Only now, I'm wondering if we need to explicitly release a lock on the shape, strides, and format information as well as the buffer location information.
It's easy to see possible use cases for returning a different object. A hypothetical future incarnation of NumPy might shift the responsibility of managing buffers from NumPy array object to a hidden raw buffer object. In this scenario, the NumPy object is the exporter, but the raw buffer object the provider.
Considering this use case, it's clear that getbuffer should return the shape and stride data independently of the provider. The raw buffer object wouldn't have that information; all it does is store a pointer and keep a reference count. Shape and stride is defined by the exporter.
So, who manages the memory to the shape and strides and isptr arrays? When a provider is created do these need to be created so that the shape and strides arrays are never deallocated when in use. The situation I'm considering is if you have a NumPy array of shape (2,3,3) which you then obtain a provider of (presumably from another package) and it retains a lock on the memory for a while. Should it also retain a lock on the shape and strides array? Can the NumPy array re-assign the shape and strides while the provider has still not been released? I would like to say yes, which means that the provider must supply it's own copy of shape and strides arrays. This could be the policy. Namely, that the provider must supply the memory for the shape, strides, and format arrays which is guaranteed for as long as a lock is held. In the case of NumPy, that provider could create it's own copy of the shape and strides arrays (or do it when the shape and strides arrays are re-assigned).
Second question: what happens if a view wants to re-export the buffer? Do the views of the buffer ever change? Example, say you create a transposed view of a Numpy array. Now you want a slice of the transposed array. What does the transposed view's getbuffer export?
Basically, you could not alter the internal representation of the object while views which depended on those values were around.
In NumPy, a transposed array actually creates a new NumPy object that refers to the same data but has its own shape and strides arrays.
With the new buffer protocol, the NumPy array would not be able to alter it's shape/strides/or reallocate its data areas while views were being held by other objects.
But requestors could alter their own copies of the data, no? Back to the transpose example: B itself obviously can't use the same "strides" array as A uses. It would have to create its own strides, right?
I don't like this example because B does have it's own strides because it is a complete NumPy array. I think we are talking about the same thing and that is "who manages the memory" for the shape and strides (and format). I think the easiest solution is to say that the "provider" does. It will manage that memory until the lock is released. How the exporter object handles that is up to the exporter.
So, what if someone takes a slice out of B? When calling B.getbuffer, does it get B's strides, or A's?
I think it should get B's. After all, if you're taking a slice of B, don't you care about the slicing relative to B's axes? I can't really think of a use case for exporting A's stride data when you take a slice of B, and it doesn't seem to simplify memory management, because B has to make it's own copies anyways.
We already have an implementation in NumPy for memory sharing and in that implementation each NumPy array controls it's own shape and strides and only the data-location is shared. I don't like the terminology of "re-exporting" because it is unnecessarily confusing to me. Naturally an object can both consume and export the buffer interface. If it does this, then it will want to make it's own copies of the shape and strides arrays so as not to rely on the "provider" for these.
Here's what I think: the lock should only apply to the buffer itself, and not to shape and stride data at all. If the requestor wants to keep its own copies of the data, it would have to malloc its own storage for it. I expect that this would be very rare.
What I'm worried about is un-necessarily preventing an exporter from changing its shape and strides because a consumer is holding a lock to it's memory (but wouldn't need a lock to it's shape and strides because it made it's own copy). How does the consumer signal that situation to the provider? So, we can't avoid thinking about a lock on the shape and stride data. We can make a policy that the provider always owns the memory for the shape and stride and will not alter it until the lock is released. Or we can say that the shape and stride information could change before the lock on the memory is released if the GIL is released (so make your own copies if you need to keep track of shape and stride). Or we could have a separate lock on shape/stride. But, we have to make some policy.
As for the provider; I think that's between it the exporter. If the exporter and provider know about each other, they shouldn't have any problems managing memory together.
Sure, what the provider and exporter do is up to the exporting object.
Having such a thing as a view object would actually be nice because it could hold on to a particular view of data with a given set of shape and strides (whose memory it owns itself) and then the exporting object would be free to change it's shape/strides information as long as the data did not change.
What I don't undestand is why it's important for the provider to retain this data. The requestor only needs the information when it's created; it will calculate its own versions of the data, and will not need the originals again, so no need to the provider to keep them around.
That is certainly a policy we could enforce (and pretty much what I've been thinking). We just need to make it explicit that the shape and strides provided is only guaranteed up until a GIL release (i.e. arbitrary Python code could change these memory areas both their content and location) and so if you need them later, make your own copies. If this were the policy, then NumPy could simply pass pointers to its stored shape and strides arrays when the buffer interface is called but then not worry about re-allocating these arrays before the "buffer" lock is released. Another object could hold on to the memory area of the NumPy array but would have to store shape and strides information if it wanted to keep it. NumPy could also just pass a pointer to the char * representation of the format (which in NumPy would be stored in the dtype object) and would not have to worry about the dtype being re-assigned later.
The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?)
I'm not sure what you mean, offsets are handled by changing the starting location of the pointer to the buffer.
But to anwser your question: you can't just change the starting location because there's hundreds of buffers. You'd either have to change the starting location of each one of them, which is highly undesirable, or to factor in an offset somehow. (This was, in fact, the point of the "derefoff" term in my original suggestion.)
I get better what your derefoff was doing now. I was missing the de-referencing that was going on. Couldn't you still just store a pointer to the start of the array. In other words, isn't your **isptr suggestion sufficient? It seems to be.
Anyways, despite the miscommunications so far, I now have a very good idea of what's going on. We definitely need to get terms straight. I agree that getbuffer should return an object. I think we need to think harder about the case when requestors re-export the buffer. (Maybe it's time to whip up some experimental objects?)
I'm still not clear what you are concerned about. If an object consumes the buffer interface and then wants to be able to later export it to another, then from our discussion about the shape/strides and format information, it would have to maintain it's own copies of these things, because it could not rely on the original provider (or exporter) to keep them around once the GIL is released. This is the reason we would have to be very clear about the guaranteed persistance of the shape/strides and format memory whose pointers are returned through the proposed buffer interface. Thanks for the discussion. It is nice to have someone to talk with about these things. A conversation always results in a better implementation. -Travis
Travis Oliphant wrote:
Carl Banks wrote:
Tr ITSM that we are using the word "view" very differently. Consider this example:
A = zeros((100,100)) B = A.transpose()
You are thinking of NumPy's particular use case. I'm thinking of a generic use case. So, yes I'm using the word view in two different contexts.
In this scenario, NumPy does not even use the buffer interface. It knows how to transpose it's own objects and does so by creating a new NumPy object (with it's own shape and strides space) with a data buffer pointed to by "A".
I realized that as soon as I tried a simple Python demonstration of it. So it's a poor example. But I hope it's obvious how it would generalize to a different type.
Having such a thing as a view object would actually be nice because it could hold on to a particular view of data with a given set of shape and strides (whose memory it owns itself) and then the exporting object would be free to change it's shape/strides information as long as the data did not change.
What I don't undestand is why it's important for the provider to retain this data. The requestor only needs the information when it's created; it will calculate its own versions of the data, and will not need the originals again, so no need to the provider to keep them around.
That is certainly a policy we could enforce (and pretty much what I've been thinking). We just need to make it explicit that the shape and strides provided is only guaranteed up until a GIL release (i.e. arbitrary Python code could change these memory areas both their content and location) and so if you need them later, make your own copies.
If this were the policy, then NumPy could simply pass pointers to its stored shape and strides arrays when the buffer interface is called but then not worry about re-allocating these arrays before the "buffer" lock is released. Another object could hold on to the memory area of the NumPy array but would have to store shape and strides information if it wanted to keep it. NumPy could also just pass a pointer to the char * representation of the format (which in NumPy would be stored in the dtype object) and would not have to worry about the dtype being re-assigned later.
Bingo! This is my preference.
The reason I ask is: if things like "buf" and "strides" and "shape" could change when a buffer is re-exported, then it can complicate things for PIL-like buffers. (How would you account for offsets in a dimension that's in a subarray?)
I'm not sure what you mean, offsets are handled by changing the starting location of the pointer to the buffer.
But to anwser your question: you can't just change the starting location because there's hundreds of buffers. You'd either have to change the starting location of each one of them, which is highly undesirable, or to factor in an offset somehow. (This was, in fact, the point of the "derefoff" term in my original suggestion.)
I get better what your derefoff was doing now. I was missing the de-referencing that was going on. Couldn't you still just store a pointer to the start of the array. In other words, isn't your **isptr suggestion sufficient? It seems to be.
No. The problem arises when slicing. In a single buffer, you would adjust the base pointer to point at the element [0,0] of the slice. But you can't do that with multiple buffers. Instead, you have to add an offset after deferencing the pointer to the subarray. Hence my derefoff proposal. It dereferenced the pointer, then added an offset to get you to the 0 position in that dimension.
Anyways, despite the miscommunications so far, I now have a very good idea of what's going on. We definitely need to get terms straight. I agree that getbuffer should return an object. I think we need to think harder about the case when requestors re-export the buffer. (Maybe it's time to whip up some experimental objects?)
I'm still not clear what you are concerned about. If an object consumes the buffer interface and then wants to be able to later export it to another, then from our discussion about the shape/strides and format information, it would have to maintain it's own copies of these things, because it could not rely on the original provider (or exporter) to keep them around once the GIL is released.
Right. So, if someone calls getbuffer, it would send its own copies of the buffer information, and not the original exporter's. The values returned by getbuffer can vary for a given buffer, depending on the exporter. Which means the data returned by getbuffer could reflect slicing. Which means the isptr array is not sufficient for the PIL-style multiple buffers.
This is the reason we would have to be very clear about the guaranteed persistance of the shape/strides and format memory whose pointers are returned through the proposed buffer interface.
Thanks for the discussion. It is nice to have someone to talk with about these things. A conversation always results in a better implementation.
I just hope it wasn't all due to terminological misunderstading. Carl Banks
participants (5)
-
Carl Banks -
Carl Banks -
Greg Ewing -
Travis Oliphant -
Travis Oliphant