How to schedule system calls with Python

[Jorgen Grahn]

On Fri, 2009-10-16, Jeremy wrote:
> On Oct 15, 6:32 pm, MRAB <pyt... at mrabarnett.plus.com> wrote:
>> TerryP wrote:
>> > On Oct 15, 7:42 pm, Jeremy <jlcon... at gmail.com> wrote:
>> >> I need to write a Python script that will call some command line
>> >> programs (using os.system).  I will have many such calls, but I want
>> >> to control when the calls are made.  I won't know in advance how long
>> >> each program will run and I don't want to have 10 programs running
>> >> when I only have one or two processors.  I want to run one at a time
>> >> (or two if I have two processors), wait until it's finished, and then
>> >> call the next one.
...
>> You could use multithreading: put the commands into a queue; start the
>> same number of worker threads as there are processors; each worker
>> thread repeatedly gets a command from the queue and then runs it using
>> os.system(); if a worker thread finds that the queue is empty when it
>> tries to get a command, then it terminates.
>
> Yes, this is it.  If I have a list of strings which are system
> commands, this seems like a more intelligent way of approaching it.
> My previous response will work, but won't take advantage of multiple
> cpus/cores in a machine without some manual manipulation.  I like this
> idea.

Note that you do not need *need* multithreading for this. To me it
seems silly to have N threads sitting just waiting for one process
each to die -- those threads contribute nothing to the multiprocessing
you want.

In Unix, you can have one process fork() and exec() as many programs
as you like, have them run on whatever CPUs you have, and wait for
them to die and reap them using wait() and related calls.  (Not sure
what the equivalent is in non-Unix OSes or portable Python.)

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .  .
\X/     snipabacken.se>   O  o   .