On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis <dom.grigonis@gmail.com> wrote:
Hi everyone,
I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue I ran into is that when I pickle & unpickle slice of memory mapped array, I need to store the spec of the subset so that I can retrieve the same portion.
The issue I am working on has been briefly discussed on stack: https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy.... Although I am not sure if it is the exact same one.
So, in short, is there a way to determine location of a slice in original array. Ideally, it would be indexing which works for any non-copy slice/subset possible.
If you chase the `.base` chain all the way down, you get to a `SharedArray.map_owner` object with its `.addr` attribute giving the memory address that's assigned to it in the current process. This will be the same address that's in the `'data'` key of that first `ndarray` returned from `SharedArray.create()` that you are making your views from. In your view `ndarray` (which may be a view of views, with slices, transposes, reshapes, and arbitrary `stride_tricks` manipulations in between). If you store the difference between the view `ndarray`'s `'data'` address from the `map_owner.addr` address, I think that's all you need to recreate the array in a different process which gets assigned a different memory address for the same `shm` name. Just add that offset to the `map_owner.addr` and restore the rest of the information in `__array_interface__`, and I think you should be good to go. I don't think you'll need to infer or represent the precise path of Python-level operations (slices, transposes, reshapes, etc.) to which it got to that point. -- Robert Kern