Let child process to run while parent is out (multiprocessing)

[Kushal Kumaran]

On Fri, Jul 20, 2012 at 2:04 AM, John Wong <gokoproject at gmail.com> wrote:
> def main(...):
>     build_id = create_build_id(...)
>     build_stuff
>     return build_id
>
> Suppose build_stuff compiles a C program. It could take days to finish, and
> notify users their builds are ready. I was thinking about using
> mutliprocessing to handle the build_stuff.
>
> So here is a sample:
>
> #!/usr/bin/python
>
> import multiprocessing as mp
> import time
>
> def build():
>     print 'I am building HUGE things'
>     time.sleep(10)
>
> def main():
>     build_p = mp.Process(name='build process', target=build)
>     build_p.start()
>     return 'abcd12345'
>
> if __name__ == '__main__':
>
>     v = main()
>     print v
>     print 'done'
>
> Here is output:
> yeukhon at fermat:~$ python c2.py
> abcd12345
> done  [now hangs for 10 seconds]
> I build things
>
> When I looked at `ps -elf|grep python`, I can see two processes running, and
> one of the python c2.py process is `wait`.  But this is not ideal,
> especially this is a web app. I can't implement any advanced queue / event
> system right now (I use Pylon, I believe I have gevent installed). But just
> with multiprocessing, is it possible to send the id first, while running
> child in the backgroud?
>
> Right now it hangs there as long as the child process is alive. Any
> solutions?
>

>From the documentation, there does not seem to be any way of
"detaching" a multiprocessing Process.  But it is doable by using the
underlying os.fork directly (CAUTION: not ready for being invoked from
a web app):

#!/usr/bin/python

import os
import time

def build_and_send_email():
    print 'I am building HUGE things'
    time.sleep(10)

def main():
    child_pid = os.fork()
    if child_pid == 0:
        build_and_send_email()
        os._exit(0)
    return 'abcd12345'

if __name__ == '__main__':
    v = main()
    print v
    print 'done'

To make it work correctly with web app will require a bit more work.
At the least, you will have to close all file descriptors to make sure
the request processing finishes.  You can turn it into a proper
background process (a daemon) using the python-daemon library with
very little code, I think.

-- 
regards,
kushal