[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly

Ask Solem report at bugs.python.org
Fri Aug 27 21:55:10 CEST 2010


Ask Solem <askh at opera.com> added the comment:

> - A worker removes a job from the queue and is killed before
> sending an ACK.

Yeah, this may be a problem. I was thinking we could make sure the task is acked before child process shutdown. Kill -9 is then not safe, but do we really want to guarantee that in multiprocessing? In celery we're safe by using AMQP's ack trasnaction anyway. The same could be said if there's a problem with the queue though. Maybe using ack timeouts? We know how many worker processes are free already.

> A worker removes a job from the queue, sends an ACK, and then is
> killed.  Due to bad luck with the scheduler, the parent cleans the
> worker before the parent has recorded the worker pid.
Guess we need to consume from the result queue until it's empty.

> You're now reading from self._cache in one thread but writing it in
> another.

Yeah, I'm not sure if SimpleQueue is managed by a lock already. Should maybe use a lock if it isn't.

> What happens if a worker sends a result and then is killed?
In the middle of sending? Or, if not I don't think this matters.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9205>
_______________________________________


More information about the Python-bugs-list mailing list