Daemonization / Popen / pipe issue

John O'Hagan research at johnohagan.com
Sun Mar 18 02:35:16 CET 2012

On Sat, 17 Mar 2012 18:17:10 -0400
Lee Clemens <java at leeclemens.net> wrote:

> On 03/16/2012 11:37 PM, John O'Hagan wrote:
> > On Fri, 16 Mar 2012 22:12:14 -0400
> > Lee Clemens <java at leeclemens.net> wrote:
> >
> >> I have a multi-threaded application
> >>
> >> I have provided a test-case here: https://gist.github.com/2054194
> > I haven't looked at your test case yet, but a possible cause is the fact
> > that the main application (or main thread) exits if it has finished
> > executing and only daemon threads remain. This can  abruptly terminate
> > threads which may be busy, for example, communicating via a pipe. The best
> > solution IMO is not to use daemon threads, but to give all threads a way to
> > terminate cleanly before the main thread does, even if this means using a
> > flag or the like.
> >
> > HTH,
> >
> > John
> >
> I call .join() on each spawned thread, whether it is daemonized or not.
> The code is available here:
> https://gist.github.com/2054194

I've tried running your code (which at a glance looks very well-written) but I
get the broken pipe errors whether the DAEMONIZE flag is True or False
(Python 2.7, Debian testing). I don't have time to dig much deeper. 

> Is it generally not recommended to daemonize Python applications using
> this method? 

I'm no guru, but not that I know of. I haven't seen the os.fork method you use
before; out of interest, why not use Thread.setDaemon? 

> Do you have any references or data to support your opinion?
> Do you have any references which specifically state - "Daemonizing
> Python applications cause Python to work in unexpected ways."? 

Not particularly, although there are a few oldish references to "Python daemon
threads considered harmful" out there. You probably know that daemonising can
cover a multitude of sins, and it's always a good idea to ensure clean
termination and not just assume that what happens in a daemon thread on program
exit is moot. I'm not saying you haven't done that, in fact your errors seem to
occur during execution, not termination.

I replied to your post because I've had broken pipe errors before with
daemon threads communicating on sockets which sounded similar to your issue. I
solved them by rewriting so that the main thread told the threads to terminate,
and waited till they had. This made daemonising unnecessary in that case.




More information about the Python-list mailing list