Feng Yu <rainwoodman@gmail.com> wrote:
In most (half?) situations the result can be directly write back via preallocated shared array before works are spawned. Then there is no need to pass data back with named segments.
You can work around it in various ways, this being one of them. Personally I prefer a parallel programming style with queues – either to scatter arrays to workers and collecting arrays from workers, or to chain workers together in a pipeline (without using coroutines). But exactly how you program is a matter of taste. I want to make it as inexpensive as possible to pass a NumPy array through a queue. If anyone else wants to help improve parallel programming with NumPy using a different paradigm, that is fine too. I just wanted to clarify why I stopped working on shared memory arrays. (As for the implementation, I am also experimenting with platform dependent asynchronous I/O (IOCP, GCD or kqueue, epoll) to pass NumPy arrays though a queue as inexpensively and scalably as possible. And no, there is no public repo, as I like to experiment with my pet project undisturbed before I let it out in the wild.) Sturla