Find location of slice in it's base
Hi everyone, I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion. The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy... <https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy...>. Although I am not sure if it is the exact same one. So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible. Kind regards, Dg —Nothing ever dies, just enters the state of deferred evaluation—
In principle this can be reconstructed from the strides, shape, and base memory address (a.ctypes.data) of the view and base arrays. However, not all views can be reconstructed using slices alone, for example, views from reshape operations or stride tricks. I don't know if it's possible to just construct a view directly given a base, offset, shape, and strides, but I think ideally that's what you'd want. Aaron Meurer On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis <dom.grigonis@gmail.com> wrote:
Hi everyone,
I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion.
The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy.... Although I am not sure if it is the exact same one.
So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible.
Kind regards, Dg
—Nothing ever dies, just enters the state of deferred evaluation—
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
I am looking for something that would work for stride tricks too. Essentially, any possible view.
On 31 Aug 2023, at 23:03, Aaron Meurer <asmeurer@gmail.com> wrote:
In principle this can be reconstructed from the strides, shape, and base memory address (a.ctypes.data) of the view and base arrays. However, not all views can be reconstructed using slices alone, for example, views from reshape operations or stride tricks. I don't know if it's possible to just construct a view directly given a base, offset, shape, and strides, but I think ideally that's what you'd want.
Aaron Meurer
On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis <dom.grigonis@gmail.com> wrote:
Hi everyone,
I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion.
The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy.... Although I am not sure if it is the exact same one.
So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible.
Kind regards, Dg
—Nothing ever dies, just enters the state of deferred evaluation—
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: dom.grigonis@gmail.com
On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis <dom.grigonis@gmail.com> wrote:
Hi everyone,
I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion.
The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy.... Although I am not sure if it is the exact same one.
So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible.
If you chase the `.base` chain all the way down, you get to a `SharedArray.map_owner` object with its `.addr` attribute giving the memory address that's assigned to it in the current process. This will be the same address that's in the `'data'` key of that first `ndarray` returned from `SharedArray.create()` that you are making your views from. In your view `ndarray` (which may be a view of views, with slices, transposes, reshapes, and arbitrary `stride_tricks` manipulations in between). If you store the difference between the view `ndarray`'s `'data'` address from the `map_owner.addr` address, I think that's all you need to recreate the array in a different process which gets assigned a different memory address for the same `shm` name. Just add that offset to the `map_owner.addr` and restore the rest of the information in `__array_interface__`, and I think you should be good to go. I don't think you'll need to infer or represent the precise path of Python-level operations (slices, transposes, reshapes, etc.) to which it got to that point. -- Robert Kern
Thank you, sounds just what I need. Will work on this in due time.
On 1 Sep 2023, at 00:38, Robert Kern <robert.kern@gmail.com> wrote:
On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis <dom.grigonis@gmail.com <mailto:dom.grigonis@gmail.com>> wrote: Hi everyone,
I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion.
The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy... <https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy...>. Although I am not sure if it is the exact same one.
So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible.
If you chase the `.base` chain all the way down, you get to a `SharedArray.map_owner` object with its `.addr` attribute giving the memory address that's assigned to it in the current process. This will be the same address that's in the `'data'` key of that first `ndarray` returned from `SharedArray.create()` that you are making your views from. In your view `ndarray` (which may be a view of views, with slices, transposes, reshapes, and arbitrary `stride_tricks` manipulations in between). If you store the difference between the view `ndarray`'s `'data'` address from the `map_owner.addr` address, I think that's all you need to recreate the array in a different process which gets assigned a different memory address for the same `shm` name. Just add that offset to the `map_owner.addr` and restore the rest of the information in `__array_interface__`, and I think you should be good to go. I don't think you'll need to infer or represent the precise path of Python-level operations (slices, transposes, reshapes, etc.) to which it got to that point.
-- Robert Kern _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org <mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org <mailto:numpy-discussion-leave@python.org> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ <https://mail.python.org/mailman3/lists/numpy-discussion.python.org/> Member address: dom.grigonis@gmail.com <mailto:dom.grigonis@gmail.com>
participants (3)
-
Aaron Meurer -
Dom Grigonis -
Robert Kern