executing multiple functions in background simultaneously

Aaron Brady castironpi at gmail.com
Wed Jan 14 10:03:43 CET 2009

On Jan 13, 7:02 pm, Catherine Moroney
<Catherine.M.Moro... at jpl.nasa.gov> wrote:
> Hello everybody,
> I know how to spawn a sub-process and then wait until it
> completes.  I'm wondering if I can do the same thing with
> a Python function.
> I would like to spawn off multiple instances of a function
> and run them simultaneously and then wait until they all complete.
> Currently I'm doing this by calling them as sub-processes
> executable from the command-line.  Is there a way of accomplishing
> the same thing without having to make command-line executables
> of the function call?
> I'm primarily concerned about code readability and ease of
> programming.  The code would look a lot prettier and be shorter
> to boot if I could spawn off function calls rather than
> subprocesses.
> Thanks for any advice,
> Catherine

'multiprocessing' does what you mentioned, as others said.  The
abstraction layer is solid, which makes your code pretty.  However, it
just creates a command line like this:

'"c:\\programs\\python26\\python.exe" "-c" "from
multiprocessing.forking import main; main()" "--multiprocessing-fork"

The handle '1916' is a pipe used to read further instructions.  The
arrive in 'main()' in the form of a pickled (serialized) dictionary.
In it, the 'main_path' key contains the path to your program.  'main
()' calls the 'prepare()' function, which calls 'imp.find_module',
using that path.  Pretty sophisticated.

You can do it yourself by creating your own command line.  Create a
subprocess by this command line (untested & lots of caveats):

'"c:\\programs\\python26\\python.exe" "-c" "from myprogram import
myfunc; myfunc()"'

But you have practically no communication with it.  If you need
parameters, you can include them on the command line, since you're
building it yourself (untested & highly vulnerable):

'"c:\\programs\\python26\\python.exe" "-c" "from myprogram import
myfunc; myfunc( literal1, literal2 )"'

For a return value, unless it can be a simple exit code, you'll need a
communication channel.  For it, a socket wouldn't be bad, or a pipe if
you're not on Windows (include the port or descriptor on the command
line).  (Even with 'multiprocessing', you're limited to pickleable
objects, however, I believe.)

More information about the Python-list mailing list