Nesting concurrent.futures.ThreadPoolExecutor
Andrew McLean
lists at andros.org.uk
Thu Jul 20 15:44:40 EDT 2017
I have a program where I am currently using a
concurrent.futures.ThreadPoolExecutor to run multiple tasks
concurrently. These tasks are typically I/O bound, involving access to
local databases and remote REST APIs. However, these tasks could
themselves be split into subtasks, which would also benefit from
concurrency.
What I am hoping is that it is safe to use a
concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up
a toy example, which seems to work. However, I'd like some confidence
that this is intentional. Concurrency is notoriously tricky.
I very much hope this is safe, because otherwise it would not be safe to
use a ThreadPoolExecutor to execute arbitrary code, in case it also used
concurrent.futures to exploit concurrency.
Here is the toy example:
> |importconcurrent.futures definner(i,j):returni,j,i**j
> defouter(i):withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
> ={executor.submit(inner,i,j):j forj inrange(5)}results =[]forfuture
> inconcurrent.futures.as_completed(futures):results.append(future.result())returnresults
> defmain():withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
> ={executor.submit(outer,i):i fori inrange(10)}results =[]forfuture
> inconcurrent.futures.as_completed(futures):results.extend(future.result())print(results)if__name__
> =="__main__":main()|
I have previously posted this on Stack Overflow, but didn't get any
replies. Apologies if you are seeing this twice.
https://stackoverflow.com/questions/44989473/nesting-concurrent-futures-threadpoolexecutor
More information about the Python-list
mailing list