Python-list Digest, Vol 112, Issue 79
niklas.berliner at gmail.com
Thu Jan 10 15:34:43 CET 2013
> ---------- Weitergeleitete Nachricht ----------
> From: Dave Angel <d at davea.name>
> To: python-list at python.org
> Date: Thu, 10 Jan 2013 00:56:20 -0500
> Subject: Re: subprocess.Popen and multiprocessing fails to execute
> external program
> On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> > I have a pipline that involves processing some data, handing the data to
> > external program (t_coffee used for sequence alignments in
> > and postprocessing the result. Since I have a lot of data, I need to run
> > pipeline in parallel which I implemented using the multiprocessing module
> > following Doug Hellmanns blog (
> > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html
> > My pipeline works perfectly fine when I run it with the multiprocessing
> > implementation and one consumer, i.e. on one core. If I increase the
> > of consumers, i.e. that multiple instances of my pipeline run in parallel
> > the external program fails with a core dump.
> Could it be that the external program is not designed to have multiple
> simultaneous instances? There are many such programs, some of which
> check for an existing process before allowing another one to get far.
> When using the multiprocessing module, always make sure your externals
> are well-behaved before looking for problems in your multi-code.
> To put it more strongly, a well-written program cannot easily be crashed
> by the parent that launched it.
the developers of the external program said that they are using the program
with multiple simultaneous instances. Also, when I execute multiple
simultaneous instances of the external program using a bash wrapper script
on my machine it works (the same wrapper scripts that fail when executed
Before asking here I have contacted the developers of the external program
but they couldn't help me any further.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list