Mailman 3 July 2017 - Async-sig

question re: loop.shutdown_asyncgens()
by Chris Jerdonek 02 Aug '17

02 Aug '17

I have a question about PEP 525 (Asynchronous Generators) which I'm sure has a simple answer, but I didn't see it in the PEP or final discussion: https://mail.python.org/pipermail/python-dev/2016-September/146265.html Basically, why is the API such that loop.shutdown_asyncgens() must be called manually? For example, why can't it be called automatically as part of close(), which seems like it would be a friendlier API and more helpful to the common case? I was trying asynchronous iterators in my code and getting the following error: Exception ignored in: <generator object Queue.get at 0x7f950a667678> Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/queues.py", line 169, in get getter.cancel() # Just in case getter is not done yet. File "/usr/local/lib/python3.6/asyncio/base_events.py", line 574, in call_soon self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Calling loop.shutdown_asyncgens() made the error go away, but it seems a little obscure that by adding an asynchronous iterator somewhere in your code, you have to remember to check that that line is present before loop.close() is called (and the exception message doesn't provide a good hint). Is there any disadvantage to always calling loop.shutdown_asyncgens() (i.e. even if it's not needed)? And why might someone need to call it at a different time? Thanks, --Chris

2 6

using asyncio in synchronous applications
by Chris Jerdonek 17 Jul '17

17 Jul '17

I have a two-part question. If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap? Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other? Thanks, --Chris [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers

4 10

Optional async method and best practices
by Laurent Mazuel 12 Jul '17

12 Jul '17

Hello, I’m working currently with Brett Cannon to bring asyncio support to our SDK. We wanted to check with you one of the scenario, since we got a loooong discussion on it together 😊. And we want to do it using the best reasonable practice with your opinion. We have an api that is clearly async and will gain a lot to be converted to asyncio. However, it's a two-step operation. Operation 1 asks for the creation of a resource and is not async, operation 2 is *optional* and wait for completion of this creation (with nightmare threads currently and I removed a lot of code moving to asyncio - happiness). There is perfectly legit scenarios where operation 2 is not needed and avoid it is better, but it has to be prepared at the same time of operation 1. Current code looks like this: sync_poller = client.create(**parameters) obj = sync_poller.resource() # Get the initial resource information, but the object is not actually created yet. obj = sync_poller.result() # OPTIONAL. This is a blocking call with thread, if you want to wait for actual creation and get updated metadatas My first prototype was to split and return a tuple (resource, coroutine): obj, optional_poller = client.create(**parameters) obj = await optional_poller # OPTIONAL But I got a warning if I decide to do not use this poller, RuntimeWarning: coroutine 'foo' was never awaited I was surprised honestly that I can't do that, since I feel like I'm not leaking anything. I didn't run the operation, so there is no wasted resource at my knowledge. But I remember wasting time because of a forgotten "yield from", so I guess it's fair 😊. But I would be curious to understand what I did badly. I found 2 solutions to avoid the warning, and I currently prefer solution 2: 1- Return a function to call, and not a coroutine. The "await" statement becomes: obj = await optional_poller() 2- Return my initial object with an async method. This allows me to write (something finally close to the current code): async_poller = client.create(**parameters) obj = async_poller.resource() # Get the initial resource information, but the object is not actually created yet. obj = await async_poller.result() # OPTIONAL My async_poller object being something like: class PollerOperation: async def result(self): ...async version of previous sync result()... So the questions are: - Does this seem a correct pattern? - Is there a simple way to achieve something like this: obj = await async_poller meaning, I can win the "result()" syntax and directly "await" on the object and get the result from magic function. I tried by subclassing some ABC coroutine/awaitable, but wasn't able to find a correct syntax. I'm not even sure this makes sense and respects the zen of Python 😊 If it helps, I'm willing to use 3.5 as minimal requirement to get async behavior. Thank you!! Laurent

5 7

no current event loop in thread 'MainThread'
by Pradip Caulagi 07 Jul '17

07 Jul '17

I am trying to write a test for a function that uses async. Can I mix using unittest.TestCase and asyncio.test_utils.TestCase? When I do that, the loop in test_utils.TestCase seems to affect the other code I have. This is a simplified example - $ cat foo.py import asyncio import unittest from asyncio import test_utils async def foo(): return True class Foo1Test(test_utils.TestCase): def setUp(self): super().setUp() self.loop = self.new_test_loop() self.set_event_loop(self.loop) def test_foo(self): res = self.loop.run_until_complete(foo()) assert res is True class Foo2Test(unittest.TestCase): def test_foo(self): loop = asyncio.get_event_loop() res = loop.run_until_complete(foo()) assert res is True $ python3 Python 3.6.1 (default, Apr 4 2017, 09:40:21) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> $ python3 -m unittest foo.py .E ====================================================================== ERROR: test_foo (foo.Foo2Test) ---------------------------------------------------------------------- Traceback (most recent call last): File "/private/tmp/foo.py", line 26, in test_foo loop = asyncio.get_event_loop() File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/events.py", line 678, in get_event_loop return get_event_loop_policy().get_event_loop() File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/events.py", line 584, in get_event_loop % threading.current_thread().name) RuntimeError: There is no current event loop in thread 'MainThread'. ---------------------------------------------------------------------- Ran 2 tests in 0.001s What am I missing? -- Pradip Caulagi http://caulagi.com

2 1

async documentation methods
by Dima Tisnek 06 Jul '17

06 Jul '17

Hi all, I'm working to improve async docs, and I wonder if/how async methods ought to be marked in the documentation, for example library/async-sync.rst: """ ... It [lock] has two basic methods, `acquire()` and `release()`. ... """ In fact, these methods are not symmetric, the earlier is asynchronous and the latter synchronous: Definitions are `async def acquire()` and `def release()`. Likewise user is expected to call `await .acquire()` and `.release()`. This is user-facing documentation, IMO it should be clearer. Although there are examples for this specific case, I'm concerned with general documentation best practice. Should this example read, e.g.: * two methods, `async acquire()` and `release()` or perhaps * two methods, used `await x.acquire()` and `x.release()` or something else? If there's a good example already Python docs or in some 3rd party docs, please tell. Likewise, should there be marks on iterators? async generators? things that ought to be used as context managers? Cheers, d.

7 15

Go-style generators in asyncio
by Vincent Michel 06 Jul '17

06 Jul '17

Hi all, I've recently been looking into the go concurrency model to see how it compares to asyncio and an interesting concept caught my attention: go generators. It's quite similar to asynchronous generator, with a bit of extra concurrency. You can find a small write-up comparing the two concepts [1] and a possible implementation of go-style generators using asyncio [2]. [1]: https://gist.github.com/vxgmichel/4ea46d3ae5c270260471d304a2c8e97b [2]: https://gist.github.com/vxgmichel/8fc63c02389dc6807206dec7ede9eb99 My conclusion is that go-style generators are quite useful in the context of a pipeline of generators producing and processing values asynchronously. This research was motivated by my work on aiostream [3], an asynchronous version of itertools, that does not benefit from this kind of optimization (yet). [3]: https://github.com/vxgmichel/aiostream Hope you'll find this interesting, Cheers, /Vincent

1 0

Read remaining and ready buffer once it has been closed by your peer
by Pau Freixes 05 Jul '17

05 Jul '17

Hi guys, The current implementation of StreamReader does not allow to read the remaining buffer if has had an exception and there was no waiters before to read the data. IMHO this is not the best way to handle this situation, the developer should be able to access to the remaining and ready buffer in somehow. For example, this data might bring some important stuff related to the reason of the exception. Ive just opened a bug [1] and made a PR with a simple proposal [2] with a very basic rationale: Meanwhile, there is data in the buffer still to be processed, the StreamReader shouldn't raise the final exception. Thoughts ? [1] http://bugs.python.org/issue30861 [2] https://github.com/python/cpython/pull/2593 -- --pau

1 0

async testing question
by Chris Jerdonek 04 Jul '17

04 Jul '17

I have a question about testing async code. Say I have a coroutine: async def do_things(): await do_something() await do_more() await do_even_more() And future: task = ensure_future(do_things()) Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()? In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more(). done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs. Thanks, --Chris

6 11