[Python-ideas] async/await in Python (C API ?)

Andrew Svetlov andrew.svetlov at gmail.com
Mon Apr 20 20:16:29 CEST 2015


Marc-Andre, I see your need but we should write C Accelerators for
asyncio first.
The task is not related to the PEP directly, while I like to working on it soon.
The problem is: accelerator should be backward compatibility and
optional (no problem with it).
But making C API requires much more: we need to implement the most
critical parts of asyncio in C and then invent C API for that.
Anyway, thank you for inspiration for desired C Async API.

On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 20.04.2015 17:17, Yury Selivanov wrote:
>>
>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote:
>>> On 20.04.2015 15:40, Yury Selivanov wrote:
>>>> Hi Marc-Andre,
>>>>
>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:
>>>>> Hi Yury,
>>>>>
>>>>> do you have any plans of also exposing an easy to use C API
>>>>> for async and await ?
>>>>>
>>>>> I think this would be useful to e.g. implement asynchronous
>>>>> database access. Most of those adapters are written in C and
>>>>> need so a C API would help a lot with this.
>>>> I think one way to have a convenient C API is to implement
>>>> a Future-like object in C -- an object with __await__ method
>>>> that should return an iterator, which should also be
>>>> implemented in C and provide a way to attach callback
>>>> or to *pipeline* C functions.
>>>>
>>>> This way C code should seamlessly integrate with asyncio.
>>>>
>>>> I doubt that we have time to include this kind of API in
>>>> the PEP, but it can be developed as a separate library.
>>>>
>>>> Your thoughts?
>>> I'm just sketching here, since I don't have much experience
>>> with writing async code (I'm using threads and multiple
>>> processes instead) and it probably shows ;-)
>>>
>>>  From the C side of things, I think using callbacks
>>> and a way to register/unregister polling functions
>>> would be great to have.
>>>
>>> The C code could then run like this in polling mode:
>>>
>>> long_running_query(...) {
>>>      /* Start long running operation... creating a handle for it */
>>>      PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);
>>>      PyAsync_ReturnAwait();
>>> }
>>>
>>> mypolling_fct(int handle) {
>>>      /* Check completion of operation */
>>>      if (completed) {
>>>          /* Fetch results into value */
>>>          PyAsync_UnregisterPoll(mypolling_fct, handle);
>>>          PyAsync_ReturnReady(value);
>>>      }
>>>      else
>>>          PyAsync_ReturnContinue();
>>> }
>>>
>>> and like this in callback mode:
>>>
>>> long_running_query(...) {
>>>      /* Create a handle for a long running operation... */
>>>      PyAsync_RegisterCallback(myquery_done, handle, timeout);
>>>      /* Start long running operation... */
>>>      PyAsync_ReturnAwait();
>>> }
>>>
>>> myquery_done(int handle, PyObject *value) {
>>>      PyAsync_UnregisterCallback(myquery_done, handle);
>>>      PyAsync_ReturnReady(value);
>>> }
>>>
>>> Both mechanisms will likely have to move the long running
>>> operation into a separate (blocking) thread, unless there's also
>>> a way to register an event loop selector somewhere and the C lib
>>> your interfacing to provides this mechanism as well.
>>>
>>
>> I see what you want to do and why now.
>>
>> However, I can't imagine how this could be implemented
>> without asyncio event loop being implemented in C.
>> Coroutines from PEP 492 aren't an indepenedant concept,
>> they require a library/framework with an event loop
>> to make them work.
>>
>> The only way to go right now is to get 'get_event_loop()'
>> and 'loop.call_soon()' working in C using Python C API.
>>
>> From that point you can implement Future object in C,
>> see 'asyncio/futures.py'; Future._schedule_callbacks,
>> Future.set_result and Future.__iter__ methods in
>> particular.
>>
>> With all that, you can easily return your future object
>> from C methods and everything should work just fine.
>>
>> I know this isn't ideal, and we actually should start a
>> separate discussion on how we can improve things on
>> our asyncio C side.
>
> I'm not expecting this to be ready anytime soon,
> just wanted to bring this up, since it may be something
> that has to be considered in the async/await API design
> early rather than later to not make it too complex to
> add later on.
>
> I'm not sure, but going through the Python API from C
> can potentially result in the code to reenter which sounds
> like it could cause some unwanted race conditions. A C API
> would avoid this.
>
> Thanks,
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Apr 20 2015)
>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Thanks,
Andrew Svetlov


More information about the Python-ideas mailing list