Mailman 3 - Async-sig

async await as basic paradigm - relation to delimited continuations ?
by Asmodehn Shade 02 Dec '17

02 Dec '17

Hi everyone, I have been working on distributed programming for a while now, however I was away from python land, and didnt get into the twisted/tornado/asyncio train until few years ago. While writing asyncio code and exploring different abstract computing paradigm (my recent hobby seems to be exploring new ways to do computing), I recently discover curio. I read a few github issues, and although I do enjoy experiments, I am also aware of the need for mathematical fundations in order for a project to be able to compose between projects and groups of people and ultimately scale and persist. As well as finding its own roots to support an ecosystem. I was wondering if there was someone on this mailing list versed enough in maths and computer science who could see some kind of association/relation between async/await and delimited continuations... and hopefully fomalize it somehow ? As a side effect, this might give some improvement ideas to curio ... The way I see it so far, very rough and naive : - the event loop is "doing the work", that is a sequence of delimited continuations. - async/await denotes how the work to be done can be composed (wrapping functions that already could compose by themselves to add additional runtime semantics), - tasks are like delimited continuations, and they will be ordered at runtime to execute in a specific sequence. Or am I completely misleaded ? If we scale this up and "distribute" it (actor model style), one actor can match one event-loop, and there are probably many association to draw there, that could indicate for curio (or libraries using it) if it is sensible or not to completely identify the event_loop to a thread... As a broad overview : 1) imperative concepts can be implemented from functional/math concepts (haskell - do notation, relation with category theory) 2) functional concept are currently implemented with imperative languages ( scheme interpreters, haskell compiler, etc. ) 3) looking as async programming and functional/math concepts these days, I am wondering how these relate, and how far down goes the rabbit hole... Thanks a lot for sharing your views on this :-). -- AlexV

1 0

awaiting task is not chaining exception
by Chris Jerdonek 13 Nov '17

13 Nov '17

Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected. I was able to reduce it down to the code below. The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate. The way I would describe it is that (at least in this case) awaiting on a task that raises an exception doesn't set __context__ when awaiting on the task inside an "except" block. Here is the code: import asyncio async def raise_error(): raise RuntimeError('exception #2') async def main(): try: raise RuntimeError('exception #1') except Exception: future = raise_error() # Uncommenting the next line makes exc.__context__ None. # future = asyncio.ensure_future(future) try: await future except Exception as exc: print(f'exc: {exc!r}, context: {exc.__context__!r}') raise asyncio.get_event_loop().run_until_complete(main()) Here is the output of the above, which works as expected: exc: RuntimeError('exception #2',), context: RuntimeError('exception #1',) Traceback (most recent call last): File "test.py", line 8, in main raise RuntimeError('exception #1') RuntimeError: exception #1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 20, in <module> asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 15, in main await future File "test.py", line 4, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 And here is the unexpected output when the commented line is uncommented (exception not getting chained): exc: RuntimeError('exception #2',), context: None Traceback (most recent call last): File "test.py", line 22, in <module> asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 17, in main await future File "test.py", line 6, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 --Chris

3 5

coroutine function vs. function returning awaitable
by Chris Jerdonek 03 Nov '17

03 Nov '17

Hi, I have a 2-part question: 1) Say you have a function that returns an awaitable (but not a coroutine object). Are there any advantages or disadvantages to replacing this with a function for which asyncio.iscoroutinefunction() returns True and asyncio.iscoroutine() returns True for the return value? An advantage might be, say, additional checks that asyncio performs. 2) If there are advantages, is there a straightforward way to construct such a replacement? In the case I have in mind (from an open source project), the "function" you'd be starting with is the class constructor for an async context manager that is also awaitable. So the class has both __aenter__ and __await__ methods. As an example naive attempt, the following _wouldn't_ work: @asyncio.coroutine def connect(): return AsyncConnectManager() The reason this doesn't work is that wrapping connect() in @asyncio.coroutine() means the return value is no longer an async context manager (it no longer has an __aenter__ method). You have to await to get the underlying async context manager. Thanks, --Chris

2 5

Asyncio-tkinter hope this is the right thread for this
by Tim Jones 06 Oct '17

06 Oct '17

Hi I am trying to build an application on the raspberry Pi using python3.6 and tkinter for the GUI. It is an entry system for a community project and so I need to be able to receive FOB swipe data into the application which is running a tkinter user interface. Any who I have all the components working ok but can't seem to get them to work together. I am trying to use asyncio but keep clashing with the tkinter event loop, so my question is can the asyncio loop work cooperatively with the tkinter loop? Do I need to create a seperate thread, or is there some other solution available? Any help with this would be greatly appreciated Regards tim Sent from my iPad

4 3

Feedback, loop.load() function
by Pau Freixes 22 Aug '17

22 Aug '17

Hi guys, I would appreciate any feedback about the idea of implementing a new load function to ask about how saturated is your reactor. I have a proof of concept [1] of how the load function might be implemented in the Asyncio python loop. The idea is to provide a method that can be used to ask about the load of the reactor in a specific time, this implementation returns the load taking into account the last 60 seconds but it can easily return the 5m and 15minutes ones u others. This method can help services built on to of Asyncio to implement back pressure mechanisms that take into account a metric coming from the loop, instead of inferring the load using other metrics provided by external agents such as the CPU, load average u others. Nowadays exists some alternatives for other languages that address this situation using the lag of a scheduler callback, produced by saturated reactors. The most known implementation is toobusy [2] a nodejs implementation. IMHO the solution provided by tobusy has a strong dependency with the hardware needing to tune the maximum lag allowed in terms of milliseconds [3]. in the POF presented the user can use an exact value meaning the percentage of the load, perhaps 0.9 Any comment would be appreciated. [1] https://github.com/pfreixes/cpython/commit/5fef3cae043abd62165ce40b181286e1… [2] https://www.npmjs.com/package/toobusy [3] https://www.npmjs.com/package/toobusy#tunable-parameters -- --pau

4 10

Re: [Async-sig] Async-sig Digest, Vol 14, Issue 5
by manuel miranda 12 Aug '17

12 Aug '17

I have one comment regarding what Nathaniel Smith said: """ Of course, if you really want an end to end measure you can do things like instrument your actual logic, see how fast you're replying to http requests or whatever, which is even more valid but creates complications because some requests are supposed to take longer than others, etc. """ you can't always use http requests or other metrics to measure how busy is your worker. Some examples of invalid metrics: - Your service can depend on external services that may be the ones making you slow. In this case, scaling up because a spike on your http response time doesn't help, its a waste of resources - You can't use metrics like CPU, memory, etc. Of course, they may mean something, but how do you know if its your worker who is using the CPU or any other random process triggered because admin connected manually, redis is also in the same machine, and a large of "please don't do that in PROD" list :) So, IMO how busy is the loop (I don't know what's the correct metric name here) is specific to that worker which will tell you that your service is dying because it is receiving too many asyncio.Tasks to be served inside what you consider a normal window time. For example, if you have an API where more than 1 second response time is not acceptable, if that loop metric is above 2 seconds (stable) you know you have to do something (scale up, improve something, etc). On Sat, Aug 12, 2017 at 6:03 PM <async-sig-request(a)python.org> wrote: > Send Async-sig mailing list submissions to > async-sig(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/async-sig > or, via email, send a message with subject or body 'help' to > async-sig-request(a)python.org > > You can reach the person managing the list at > async-sig-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Async-sig digest..." > > > Today's Topics: > > 1. Re: Feedback, loop.load() function (Nathaniel Smith) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 11 Aug 2017 11:04:43 -0700 > From: Nathaniel Smith <njs(a)pobox.com> > To: Pau Freixes <pfreixes(a)gmail.com> > Cc: async-sig(a)python.org > Subject: Re: [Async-sig] Feedback, loop.load() function > Message-ID: > < > CAPJVwBmEx7UzMtN6WGVpWvhbrOhnhzrv7EkOf3vLseiyRf6dbQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > It looks like your "load average" is computing something very different > than the traditional Unix "load average". If I'm reading right, yours is a > measure of what percentage of the time the loop spent sleeping waiting for > I/O, taken over the last 60 ticks of a 1 second timer (so generally > slightly longer than 60 seconds). The traditional Unix load average is an > exponentially weighted moving average of the length of the run queue. > > Is one of those definitions better for your goal of detecting when to shed > load? I don't know. But calling them the same thing is pretty confusing > :-). The Unix version also has the nice property that it can actually go > above 1; yours doesn't distinguish between a service whose load is at > exactly 100% of capacity and barely keeping up, versus one that's at 200% > of capacity and melting down. But for load shedding maybe you always want > your tripwire to be below that anyway. > > More broadly we might ask what's the best possible metric for this purpose > ? how do we judge? A nice thing about the JavaScript library you mention is > that scheduling delay is a real thing that directly impacts quality of > service ? it's more of an "end to end" measure in a sense. Of course, if > you really want an end to end measure you can do things like instrument > your actual logic, see how fast you're replying to http requests or > whatever, which is even more valid but creates complications because some > requests are supposed to take longer than others, etc. I don't know which > design goals are important for real operations. > > On Aug 6, 2017 3:57 PM, "Pau Freixes" <pfreixes(a)gmail.com> wrote: > > > Hi guys, > > > > I would appreciate any feedback about the idea of implementing a new > > load function to ask about how saturated is your reactor. > > > > I have a proof of concept [1] of how the load function might be > > implemented in the Asyncio python loop. > > > > The idea is to provide a method that can be used to ask about the load > > of the reactor in a specific time, this implementation returns the > > load taking into account the last 60 seconds but it can easily return > > the 5m and 15minutes ones u others. > > > > This method can help services built on to of Asyncio to implement back > > pressure mechanisms that take into account a metric coming from the > > loop, instead of inferring the load using other metrics provided by > > external agents such as the CPU, load average u others. > > > > Nowadays exists some alternatives for other languages that address > > this situation using the lag of a scheduler callback, produced by > > saturated reactors. The most known implementation is toobusy [2] a > > nodejs implementation. > > > > IMHO the solution provided by tobusy has a strong dependency with the > > hardware needing to tune the maximum lag allowed in terms of > > milliseconds [3]. in the POF presented the user can use an exact value > > meaning the percentage of the load, perhaps 0.9 > > > > Any comment would be appreciated. > > > > [1] https://github.com/pfreixes/cpython/commit/ > > 5fef3cae043abd62165ce40b181286e18f5fb19c > > [2] https://www.npmjs.com/package/toobusy > > [3] https://www.npmjs.com/package/toobusy#tunable-parameters > > -- > > --pau > > _______________________________________________ > > Async-sig mailing list > > Async-sig(a)python.org > > https://mail.python.org/mailman/listinfo/async-sig > > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > >

1 0

pattern for handling interrupt signals in asyncio
by Chris Jerdonek 06 Aug '17

06 Aug '17

I want to share a pattern I came up with for handling interrupt signals in asyncio to see if you had any feedback (ways to make it easier, similar approaches, etc). I wanted something that was easy to check and reason about. I'm already familiar with some of the pitfalls in handling signals, for example as described in Nathaniel's Control-C blog post announced here: https://mail.python.org/pipermail/async-sig/2017-April/thread.html The basic idea is to create a Future to run alongside the main coroutine whose only purpose is to "catch" the signal. And then call-- asyncio.wait(futures, return_when=asyncio.FIRST_COMPLETED) When a signal is received, both tasks stop, and then you have access to the main task (which will be pending) for things like cleanup and inspection. One advantage of this approach is that it lets you put all your cleanup logic in the main program instead of putting some of it in the signal handler. You also don't need to worry about things like handling KeyboardInterrupt at arbitrary points in your code. I'm including the code at bottom. On the topic of asyncio.run() that I mentioned in an earlier email [1], it doesn't look like the run() API posted in PR #465 [2] has hooks to support what I'm describing (but I could be wrong). So maybe this is another use case that the future API should contemplate. --Chris [1] https://mail.python.org/pipermail/async-sig/2017-August/000373.html [2] https://github.com/python/asyncio/pull/465 import asyncio import io import signal def _cleanup(loop): try: loop.run_until_complete(loop.shutdown_asyncgens()) finally: loop.close() def handle_sigint(future): future.set_result(signal.SIGINT) async def run(): print('running...') await asyncio.sleep(1000000) def get_message(sig, task): stream = io.StringIO() task.print_stack(file=stream) traceback = stream.getvalue() return f'interrupted by {sig.name}:\n{traceback}' def main(coro): loop = asyncio.new_event_loop() try: # This is made truthy if the loop is interrupted by a signal. interrupted = [] future = asyncio.Future(loop=loop) future.add_done_callback(lambda future: interrupted.append(1)) loop.add_signal_handler(signal.SIGINT, handle_sigint, future) futures = [future, coro] future = asyncio.wait(futures, return_when=asyncio.FIRST_COMPLETED) done, pending = loop.run_until_complete(future) if interrupted: # Do whatever cleanup you want here and/or get the stacktrace # of the interrupted main task. sig = done.pop().result() task = pending.pop() msg = get_message(sig, task) task.cancel() raise KeyboardInterrupt(msg) finally: _cleanup(loop) main(run()) Below is what the code above outputs if you run it and then press Control-C: running... ^CTraceback (most recent call last): File "test-signal.py", line 54, in <module> main(run()) File "test-signal.py", line 49, in main raise KeyboardInterrupt(msg) KeyboardInterrupt: interrupted by SIGINT: Stack for <Task pending coro=<run() running at test-signal.py:17> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10fe9b9a8>()]>> (most recent call last): File "test-signal.py", line 17, in run await asyncio.sleep(1000000)

1 1

question re: loop.shutdown_asyncgens()
by Chris Jerdonek 02 Aug '17

02 Aug '17

I have a question about PEP 525 (Asynchronous Generators) which I'm sure has a simple answer, but I didn't see it in the PEP or final discussion: https://mail.python.org/pipermail/python-dev/2016-September/146265.html Basically, why is the API such that loop.shutdown_asyncgens() must be called manually? For example, why can't it be called automatically as part of close(), which seems like it would be a friendlier API and more helpful to the common case? I was trying asynchronous iterators in my code and getting the following error: Exception ignored in: <generator object Queue.get at 0x7f950a667678> Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/queues.py", line 169, in get getter.cancel() # Just in case getter is not done yet. File "/usr/local/lib/python3.6/asyncio/base_events.py", line 574, in call_soon self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Calling loop.shutdown_asyncgens() made the error go away, but it seems a little obscure that by adding an asynchronous iterator somewhere in your code, you have to remember to check that that line is present before loop.close() is called (and the exception message doesn't provide a good hint). Is there any disadvantage to always calling loop.shutdown_asyncgens() (i.e. even if it's not needed)? And why might someone need to call it at a different time? Thanks, --Chris

2 6

using asyncio in synchronous applications
by Chris Jerdonek 17 Jul '17

17 Jul '17

I have a two-part question. If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap? Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other? Thanks, --Chris [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers

4 10

Optional async method and best practices
by Laurent Mazuel 12 Jul '17

12 Jul '17

Hello, I’m working currently with Brett Cannon to bring asyncio support to our SDK. We wanted to check with you one of the scenario, since we got a loooong discussion on it together 😊. And we want to do it using the best reasonable practice with your opinion. We have an api that is clearly async and will gain a lot to be converted to asyncio. However, it's a two-step operation. Operation 1 asks for the creation of a resource and is not async, operation 2 is *optional* and wait for completion of this creation (with nightmare threads currently and I removed a lot of code moving to asyncio - happiness). There is perfectly legit scenarios where operation 2 is not needed and avoid it is better, but it has to be prepared at the same time of operation 1. Current code looks like this: sync_poller = client.create(**parameters) obj = sync_poller.resource() # Get the initial resource information, but the object is not actually created yet. obj = sync_poller.result() # OPTIONAL. This is a blocking call with thread, if you want to wait for actual creation and get updated metadatas My first prototype was to split and return a tuple (resource, coroutine): obj, optional_poller = client.create(**parameters) obj = await optional_poller # OPTIONAL But I got a warning if I decide to do not use this poller, RuntimeWarning: coroutine 'foo' was never awaited I was surprised honestly that I can't do that, since I feel like I'm not leaking anything. I didn't run the operation, so there is no wasted resource at my knowledge. But I remember wasting time because of a forgotten "yield from", so I guess it's fair 😊. But I would be curious to understand what I did badly. I found 2 solutions to avoid the warning, and I currently prefer solution 2: 1- Return a function to call, and not a coroutine. The "await" statement becomes: obj = await optional_poller() 2- Return my initial object with an async method. This allows me to write (something finally close to the current code): async_poller = client.create(**parameters) obj = async_poller.resource() # Get the initial resource information, but the object is not actually created yet. obj = await async_poller.result() # OPTIONAL My async_poller object being something like: class PollerOperation: async def result(self): ...async version of previous sync result()... So the questions are: - Does this seem a correct pattern? - Is there a simple way to achieve something like this: obj = await async_poller meaning, I can win the "result()" syntax and directly "await" on the object and get the result from magic function. I tried by subclassing some ABC coroutine/awaitable, but wasn't able to find a correct syntax. I'm not even sure this makes sense and respects the zen of Python 😊 If it helps, I'm willing to use 3.5 as minimal requirement to get async behavior. Thank you!! Laurent

5 7