On 01/10/15 02:32, Juha Jeronen wrote:
Sounds good. Out of curiosity, are there any standard fork-safe threadpools, or would this imply rolling our own?
You have to roll your own. Basically use pthreads_atfork to register a callback that shuts down the threadpool before a fork and another callback that restarts it. Python's threading module does not expose the pthreads_atfork function, so you must call it from Cython. I believe there is also a tiny atfork module in PyPI.
So maybe it would be better, at least at first, to make a pure-Cython version with no attempt at multithreading?
I would start by making a pure Cython version that works correctly. The next step would be to ensure that it releases the GIL. After that you can worry about parallel processing, or just tell the user to use threads or joblib. Sturla