Python 3: Launch multiple commands(subprocesses) in parallel (but upto 4 any time at same time) AND store each of their outputs into a variable
Dale Marvin
dmarv at dop.com
Wed Aug 24 00:42:21 EDT 2016
On 8/23/16 8:15 PM, lax.clarke at gmail.com wrote:
> I am trying to:
>
> 1) Use Python 3+ (specifically 3.4 if it matters)
> 2) Launch N commands in background (e.g., like subprocess.call would
for individual commands)
> 3) But only limit P commands to run at same time
> 4) Wait until all N commands are done
> 5) Have an array of N strings with the stdout+stderr of each command
in it.
>
> What is the best way to do this?
The best way is a matter of opinion, I have had success using Celery
with Redis. <http://www.celeryproject.org/>
DAle
More information about the Python-list
mailing list