[New-bugs-announce] [issue25908] ProcessPoolExecutor deadlock on KeyboardInterrupt

[Thomas Jackson]

New submission from Thomas Jackson:

If a KeyboardInterrupt is received while a worker process is grabbing an item off of the queue that worker process dies with an uncaught exception. This means that the ProcessPool now has lost a process, and currently has no mechanism to recover from dead processes. This is especially noticeable if the CallItem is relatively large (as the call_queue.get() includes all the pickle time).

A simple fix is to have the worker process not do anything with the keyboard interrupt-- since it would have no idea what to do. This cannot be implemented with a regular try/except-- as the item will be partially pulled off of the queue and lost. My proposed fix is to disable the SIGINT handler in the worker process while getting items off of the queue.

An alternate approach is to actually change multiprocessing.Queue.get() to leave the item on the queue if it is interrupted with a keyboard interrupt

to this is to catch the KeyboardInterrupt and simply continue on-- then we can rely on the caller to do the cleanup.This cannot be done by simply 


Proposed patch attached

----------
files: worker_ignore_interrupt.patch
keywords: patch
messages: 256703
nosy: jacksontj
priority: normal
severity: normal
status: open
title: ProcessPoolExecutor deadlock on KeyboardInterrupt
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file41358/worker_ignore_interrupt.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25908>
_______________________________________