A question about subprocess
kar1107 at gmail.com
Thu Oct 4 08:12:32 CEST 2007
On Oct 3, 9:46 am, JD <Jiandong... at gmail.com> wrote:
> I want send my jobs over a whole bunch of machines (using ssh). The
> jobs will need to be run in the following pattern:
> (Machine A) (Machine B) (Machine C)
> Job A1 Job B1 Job C1
> Job A2 Job B2 etc
> Job A3 etc
> Jobs runing on machine A, B, C should be in parallel, however, for
> each machine, jobs should run one after another.
> How can I do it with the subprocess?
subprocess is not network aware. What you can do is write a simple
python script say run_jobs.py which can take in a command-line
argument (say A or B or C) and will fire a sequence of subprocesses to
execute a series of jobs. This will ensure the serialization condition
like A2 starting after A1's completion.
Now you can write a load distributer kind of script which uses ssh to
login to the various machines and run run_jobs.py with appropriate
argument (Here I assume all machines have access to run_jobs.py -- say
it may reside on a shared mounted file-system).
e.g. in outer script:
ssh machine-A run_jobs.py A
ssh machine-B run_jobs.py B
ssh machine-B run_jobs.py C
You may want to fire all these at once so that they all execute in
More information about the Python-list