Again, not everyone uses Unix.
And on Unix it is not trival to pass data back from the child process. I solved that problem with Sys V IPC (pickling the name of the segment).
I wonder if it is neccessary insist being able to pass large amount of data back from child to the parent process. In most (half?) situations the result can be directly write back via preallocated shared array before works are spawned. Then there is no need to pass data back with named segments. Here I am just doodling some possible use cases along the OpenMP line. The sample would just copy the data from s to r, in two different ways. On systems that does not support multiprocess + fork, the semantics is still well preserved if threading is used. ``` import ...... as mp # the access attribute of inherited variables is at least 'privatecopy' # but with threading backend it becomes 'shared' s = numpy.arange(10000) with mp.parallel(num_threads=8) as section: r = section.empty(10000) # variables defined via section.empty will always be 'shared' def work(): # variables defined in the body is 'private' tid = section.get_thread_num() size = section.get_num_threads() sl = slice(tid * r.size // size, (tid + 1) * r.size // size) r[sl] = s[sl] status = section.run(work) assert not any(status.errors) # the support to the following could be implemented with section.run chunksize = 1000 def work(i): sl = slice(i, i + chunksize) r[sl] = s[sl] return s[sl].sum() status = section.loop(work, range(0, r.size, chunksize), schedule='static') assert not any(status.errors) total = sum(status.results) ```
6. If we are to define a set of operations I would recommend take a look at OpenMP as a reference -- It has been out there for decades and used widely. An equiavlant to the 'omp parallel for' construct in Python will be a very good starting point and immediately useful.
If you are on Unix, you can just use a context manager. Call os.fork in __enter__ and os.waitpid in __exit__.
Sturla
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion