On 1/11/07, Charles R Harris charlesr.harris@gmail.com wrote:
On 1/11/07, Torgil Svensson torgil.svensson@gmail.com wrote:
Sure. I'm not objecting the memory model, what I mean is that data access between modules has a wider scope than just a memory model. Maybe i'm completely out-of-scope here, I thought this was worth considering for the inter-module-data-sharing - scope.
This is where separating the memory block from the API starts to show advantages. OTOH, we should try to keep this all as simple and basic as possible. Trying to design for every potential use will lead to over design, it is a fine line to walk.
I Agree. I'm trying to look after a use case of my own here where I have a huge array (won't fit memory) with data that is very easy to compress (easily fit in memory). OTOH, I have yet no need to share this between modules but a simple data access API opens up a variety of options.
I my mindset, I can slice and dice my huge array and the implementation behind the data access API will choose between having the views represented internally as intervals or lists of indexes.
So i'm +1 for having all information concerning nd-array access on a logical level (shapes) in one API and let the memory-layout-details (strides, FORTRAN, C etc) live in another API and a module that wants to try to skip the api overhead (numpy) can always do something like:
memory_interface=array_interface->get_memory_layout() if (memory_interface) { ... use memory_interface->strides ... etc } else { ... use array_interface->get_item_fom_index() ... etc }
I'm guessing that most of the modules trying to access an array will choose to go through numpy for fast operations.
Another use of an api is to do things like give an "RGB"-view of an image regardless of which weird image format lying below without having to convert the whole image in-memory and loose precision or memory. If we want the whole in-memory-RGB-copy we could just take the RGB-view, pass it to numpy and force numpy to do a copy. The module can then, in either case, operate on the image through numpy or return a numpy object to the user. (numpy is of course integrated in python by then)