spawning pyhon apps...

[Jason Scheirer]

On Jan 9, 3:43 pm, "bruce" <bedoug... at earthlink.net> wrote:
> hi jason....
>
> forgive me... but in your sample:
>         my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
>         %i.txt"%x) for x in xrange(10)]
> are you spawning 'foo.py' 10 times? that can't be right!
> so just what is "foo.py" used for? what am i missing...
>
> it looks like the my_popenobjects array is iterated through to check the
> statuscode. is the statuscode the value that would be returned from a child
> python script via something like "return(2)"....
>
> i've seen mention of os.waitpid(..) does this play into waiting for child
> processes to complete, or determine if they've terminated??
>
> thanks
>
> -----Original Message-----
> From: python-list-bounces+bedouglas=earthlink.... at python.org
>
> [mailto:python-list-bounces+bedouglas=earthlink.... at python.org]On Behalf
> Of Jason Scheirer
> Sent: Friday, January 09, 2009 3:19 PM
> To: python-l... at python.org
> Subject: Re: spawning pyhon apps...
>
> On Jan 9, 2:47 pm, "bruce" <bedoug... at earthlink.net> wrote:
> > hi...
>
> > toying with an idea.. trying to figure out a good/best way to spawn
> multiple
> > python scripts from a parent python app. i'm trying to figure out how to
> > determine when all child apps have completed, or to possibly determine if
> > any of the child processes have died/halted..
>
> > parent app
> >  spawn child1
> >  spawn child2
> >  spawn child3
> >  .
> >  .
> >  .
> >  spawn childn
>
> > do i iterate through a os.waitpid(pid) for each pid of the child processes
> i
> > create?
>
> > is there another approach? code samples/tutorial...??
>
> > i've seen various approaches via google, but not just what i'm looking
> for..
>
> > thanks
>
> Investigate the subprocess module, you probably want Popen objects.
> You can do a poll loop like
>
> my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
> %i.txt"%x) for x in xrange(10)]
>
> while any(popenobject.returncode is None for popenobject in
> my_popenobjects):
>   time.sleep(0.25)
>
> If your tasks are more like function calls and less like shell
> scripts, then investigate writing your python as an importable module
> and use the multiprocessing module, which will do threading/
> subprocessing for you.
> --http://mail.python.org/mailman/listinfo/python-list
>
>

Correction: statuscode is wrong. It's returncode.

Foo.py is the hypothetical Python script you want to run in a
subprocess. In this example, I have an external shell script named
foo.py, and I am indeed spawning 10 copies of it, each with a second
argument that varies (foo.py --filename=file0.txt, foo.py --
filename=file1.txt, ... foo.py --filename=file9.txt). You don't need
os.waitpid() with a Popen object, there is a Popen.wait() method you
can call which will accomplish the exact same thing. I'm polling the
Popen.returncode for each process' return code (which is the numeric
code a process returns when it finishes, like sys.exit(x) or return,
or gives None if it's not done yet. What this sample is doing is
opening 10 copies of the script and running them in parallel, if you
want to run it is serial then you can do a simple for loop and .wait()
on each:

for cmd in ('a', 'b', 'c'):
  sp = subprocess.Popen(cmd)
  sp.wait()
  print "Command %r completed with status %i" % (cmd, sp.returncode)

I'm still not 100% sure what you're trying to accomplish. What is the
exact problem you are wishing to solve?