Let child process to run while parent is out (multiprocessing)
kushal.kumaran+python at gmail.com
Fri Jul 20 10:48:51 CEST 2012
On Fri, Jul 20, 2012 at 2:04 AM, John Wong <gokoproject at gmail.com> wrote:
> def main(...):
> build_id = create_build_id(...)
> return build_id
> Suppose build_stuff compiles a C program. It could take days to finish, and
> notify users their builds are ready. I was thinking about using
> mutliprocessing to handle the build_stuff.
> So here is a sample:
> import multiprocessing as mp
> import time
> def build():
> print 'I am building HUGE things'
> def main():
> build_p = mp.Process(name='build process', target=build)
> return 'abcd12345'
> if __name__ == '__main__':
> v = main()
> print v
> print 'done'
> Here is output:
> yeukhon at fermat:~$ python c2.py
> done [now hangs for 10 seconds]
> I build things
> When I looked at `ps -elf|grep python`, I can see two processes running, and
> one of the python c2.py process is `wait`. But this is not ideal,
> especially this is a web app. I can't implement any advanced queue / event
> system right now (I use Pylon, I believe I have gevent installed). But just
> with multiprocessing, is it possible to send the id first, while running
> child in the backgroud?
> Right now it hangs there as long as the child process is alive. Any
>From the documentation, there does not seem to be any way of
"detaching" a multiprocessing Process. But it is doable by using the
underlying os.fork directly (CAUTION: not ready for being invoked from
a web app):
print 'I am building HUGE things'
child_pid = os.fork()
if child_pid == 0:
if __name__ == '__main__':
v = main()
To make it work correctly with web app will require a bit more work.
At the least, you will have to close all file descriptors to make sure
the request processing finishes. You can turn it into a proper
background process (a daemon) using the python-daemon library with
very little code, I think.
More information about the Python-list