Personally I prefer a parallel programming style with queues – either to scatter arrays to workers and collecting arrays from workers, or to chain workers together in a pipeline (without using coroutines). But exactly how you program is a matter of taste. I want to make it as inexpensive as possible to pass a NumPy array through a queue. If anyone else wants to help improve parallel programming with NumPy using a different paradigm, that is fine too. I just wanted to clarify why I stopped working on shared memory arrays.
Even I am not very obsessed with functional and queues, I still have to agree with you queues tend to produce more readable and less verbose code -- if there is the right tool.
(As for the implementation, I am also experimenting with platform dependent asynchronous I/O (IOCP, GCD or kqueue, epoll) to pass NumPy arrays though a queue as inexpensively and scalably as possible. And no, there is no public repo, as I like to experiment with my pet project undisturbed before I let it out in the wild.)
It will be wonderful if there is a way to pass numpy array around without a huge dependency list. After all, we know the address of the array and, in principle we are able to find the physical pages and map them in the receiver side. Also, did you checkout http://zeromq.org/blog:zero-copy ? ZeroMQ is a dependency of Jupyter, so it is quite available. - Yu
Sturla
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion