[New-bugs-announce] [issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[SeungHyun.Hwang]

New submission from SeungHyun.Hwang:

I tested on python3.4.2, windows 7

1. Only proactorEvent issued.
2. ThreadPoolExecutor has many workers (in the attached example file, worker count 20,000)
3. The loop does run_in_executor more call than worker count (in the attached example file, 40,000 calls)
4. After some random seconds, raise BlockingIOError

BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately.
exception calling callback for <Future at 0x1ab89ef0 state=finished returned NoneType>
Traceback (most recent call last):
  File "c:\Python342\Lib\concurrent\futures\_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "c:\Python342\Lib\asyncio\futures.py", line 410, in <lambda>
    new_future._copy_state, fut))
  File "c:\Python342\Lib\asyncio\base_events.py", line 403, in call_soon_threadsafe
    self._write_to_self()
  File "c:\Python342\Lib\asyncio\proactor_events.py", line 449, in _write_to_self
    self._csock.send(b'\0')


I guess that proactor's _write_to_self method misses exception handle.

proactor_events.py
def _write_to_self(self):
        self._csock.send(b'\0')


selector_events.py
def _write_to_self(self):
        # This may be called from a different thread, possibly after
        # _close_self_pipe() has been called or even while it is
        # running.  Guard for self._csock being None or closed.  When
        # a socket is closed, send() raises OSError (with errno set to
        # EBADF, but let's not rely on the exact error code).
        csock = self._csock
        if csock is not None:
            try:
                csock.send(b'\0')
            except OSError:
                if self._debug:
                    logger.debug("Fail to write a null byte into the "
                                 "self-pipe socket",
                                 exc_info=True)


Ps: It's my first publication. Hope you understand my poor comment..

----------
components: asyncio
files: example_thread_executor.py
messages: 239872
nosy: gvanrossum, haypo, kernel0, yselivanov
priority: normal
severity: normal
status: open
title: asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers
type: crash
versions: Python 3.4
Added file: http://bugs.python.org/file38801/example_thread_executor.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23846>
_______________________________________