executing multiple functions in background simultaneously
Aaron Brady
castironpi at gmail.com
Wed Jan 14 04:03:43 EST 2009
On Jan 13, 7:02 pm, Catherine Moroney
<Catherine.M.Moro... at jpl.nasa.gov> wrote:
> Hello everybody,
>
> I know how to spawn a sub-process and then wait until it
> completes. I'm wondering if I can do the same thing with
> a Python function.
>
> I would like to spawn off multiple instances of a function
> and run them simultaneously and then wait until they all complete.
> Currently I'm doing this by calling them as sub-processes
> executable from the command-line. Is there a way of accomplishing
> the same thing without having to make command-line executables
> of the function call?
>
> I'm primarily concerned about code readability and ease of
> programming. The code would look a lot prettier and be shorter
> to boot if I could spawn off function calls rather than
> subprocesses.
>
> Thanks for any advice,
>
> Catherine
'multiprocessing' does what you mentioned, as others said. The
abstraction layer is solid, which makes your code pretty. However, it
just creates a command line like this:
'"c:\\programs\\python26\\python.exe" "-c" "from
multiprocessing.forking import main; main()" "--multiprocessing-fork"
"1916"'
The handle '1916' is a pipe used to read further instructions. The
arrive in 'main()' in the form of a pickled (serialized) dictionary.
In it, the 'main_path' key contains the path to your program. 'main
()' calls the 'prepare()' function, which calls 'imp.find_module',
using that path. Pretty sophisticated.
You can do it yourself by creating your own command line. Create a
subprocess by this command line (untested & lots of caveats):
'"c:\\programs\\python26\\python.exe" "-c" "from myprogram import
myfunc; myfunc()"'
But you have practically no communication with it. If you need
parameters, you can include them on the command line, since you're
building it yourself (untested & highly vulnerable):
'"c:\\programs\\python26\\python.exe" "-c" "from myprogram import
myfunc; myfunc( literal1, literal2 )"'
For a return value, unless it can be a simple exit code, you'll need a
communication channel. For it, a socket wouldn't be bad, or a pipe if
you're not on Windows (include the port or descriptor on the command
line). (Even with 'multiprocessing', you're limited to pickleable
objects, however, I believe.)
More information about the Python-list
mailing list