On Sat, 26 May 2012 17:34:52 +0200
Masklinn
On 2012-05-25, at 23:37 , Mike Meyer wrote:
Is it possible to place limitations on the function such that this kind of controlled concurrent operation is safe? This would mean ideally only having pure functions, and at the very least having functions which can't share state (not easily anyway).
I'm not sure pure functions is good enough for cPython. If the function involves looking through a tree of state (shared via the arguments, even), then the changing reference counts as the code goes through the key will hose you, unless the function evaluations are serialized via the GIL.
Python, as a language, has no such provision that I know of beyond "be careful" and "you're on your own".
Generally true for your code, but I think it tries to keep the interpreter from tripping over it's own feet (via the GIL, etc.).
A possible option, though, would be to use `multiprocessing` rather than threads: multiprocessing.pool already provides a `map` operation, and processes can't share state by default (doing so is quite an explicit — and some would say involved — operation). Going through multiprocessing puts other limitations/complexities on the function implementations, but at the very least it wouldn't be possible to *unknowingly* share state.
I'm familiar with that option, but was hoping to avoid it.
Though adding reduce (and maybe a mapreduce?) method to something like
concurrent.futures might be nice.
Thanks