[New-bugs-announce] [issue34769] _asyncgen_finalizer_hook running in wrong thread

[twisteroid ambassador]

New submission from twisteroid ambassador <twisteroid.ambassador at gmail.com>:

When testing my happy eyeballs library, I occasionally run into issues with async generators seemingly not finalizing. After setting loop.set_debug(True), I have been seeing log entries like these:


Exception ignored in: <async_generator object builtin_resolver at 0x7fca945c8158>
Traceback (most recent call last):
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 466, in _asyncgen_finalizer_hook
    self.create_task(agen.aclose())
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 386, in create_task
    task = tasks.Task(coro, loop=self)
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 674, in call_soon
    self._check_thread()
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 712, in _check_thread
    "Non-thread-safe operation invoked on an event loop other "
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
ERROR    asyncio Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
  File "/opt/Python3.7.0/lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "/opt/Python3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/opt/Python3.7.0/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/Python3.7.0/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
    work_item.run()
  File "/opt/Python3.7.0/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 746, in _getaddrinfo_debug
    msg.append(f'type={type!r}')
  File "/opt/Python3.7.0/lib/python3.7/enum.py", line 572, in __repr__
    self.__class__.__name__, self._name_, self._value_)
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 466, in _asyncgen_finalizer_hook
    self.create_task(agen.aclose())
  File "/opt/Python3.7.0/lib/python3.7/asyncio/base_events.py", line 386, in create_task
    task = tasks.Task(coro, loop=self)
task: <Task pending coro=<<async_generator_athrow without __name__>()> created at /opt/Python3.7.0/lib/python3.7/asyncio/base_events.py:386>



This is a typical instance. Usually several instances like this occur at once.

I'll try to reproduce these errors in a simple program. Meanwhile, here are some details about the actual program, which may or may not be related to the errors:

* I have several nested async generators (async for item in asyncgen: yield do_something(item); ), and when the errors appear, the above error messages and stack traces repeat several times, with the object names mentioned in "Exception ignored in: ..." being each of the nested async generators. Other parts of the error messages, including the stack traces, are exactly the same.

* I never used threads or loop.run_in_executor() explicitly in the program. However, the innermost async generator calls loop.getaddrinfo(), and that is implemented by running a Python function, socket.getaddrinfo(), with loop.run_in_executor(), so the program implicitly uses threads. (socket.getaddrinfo() is a Python function that calls a C function, _socket.getaddrinfo().)

* The outermost async generator is not iterated using `async for`. Instead, it is iterated by calling its `__aiter__` method, saving the returned async iterator object, and then awaiting on the `__anext__` method of the async iterator repeatedly. Of course, all of these are done in the same event loop.


Environment: Python 3.7.0 compiled from source, on Debian stretch.

----------
components: asyncio
messages: 326090
nosy: asvetlov, twisteroid ambassador, yselivanov
priority: normal
severity: normal
status: open
title: _asyncgen_finalizer_hook running in wrong thread
type: behavior
versions: Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34769>
_______________________________________