[Python-ideas] async/await in Python (C API ?)

[M.-A. Lemburg]

On 20.04.2015 15:40, Yury Selivanov wrote:
> Hi Marc-Andre,
> 
> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:
>> Hi Yury,
>>
>> do you have any plans of also exposing an easy to use C API
>> for async and await ?
>>
>> I think this would be useful to e.g. implement asynchronous
>> database access. Most of those adapters are written in C and
>> need so a C API would help a lot with this.
> 
> I think one way to have a convenient C API is to implement
> a Future-like object in C -- an object with __await__ method
> that should return an iterator, which should also be
> implemented in C and provide a way to attach callback
> or to *pipeline* C functions.
> 
> This way C code should seamlessly integrate with asyncio.
> 
> I doubt that we have time to include this kind of API in
> the PEP, but it can be developed as a separate library.
> 
> Your thoughts?

I'm just sketching here, since I don't have much experience
with writing async code (I'm using threads and multiple
processes instead) and it probably shows ;-)

>From the C side of things, I think using callbacks
and a way to register/unregister polling functions
would be great to have.

The C code could then run like this in polling mode:

long_running_query(...) {
    /* Start long running operation... creating a handle for it */
    PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);
    PyAsync_ReturnAwait();
}

mypolling_fct(int handle) {
    /* Check completion of operation */
    if (completed) {
        /* Fetch results into value */
        PyAsync_UnregisterPoll(mypolling_fct, handle);
        PyAsync_ReturnReady(value);
    }
    else
        PyAsync_ReturnContinue();
}

and like this in callback mode:

long_running_query(...) {
    /* Create a handle for a long running operation... */
    PyAsync_RegisterCallback(myquery_done, handle, timeout);
    /* Start long running operation... */
    PyAsync_ReturnAwait();
}

myquery_done(int handle, PyObject *value) {
    PyAsync_UnregisterCallback(myquery_done, handle);
    PyAsync_ReturnReady(value);
}

Both mechanisms will likely have to move the long running
operation into a separate (blocking) thread, unless there's also
a way to register an event loop selector somewhere and the C lib
your interfacing to provides this mechanism as well.

>> A problem I ran to with e.g. gevent is that it's not easy
>> to use the available async APIs from C, even less so for
>> systems like the ODBC API which require the use of polling
>> for async processing and which don't give you access to e.g.
>> tap into long running parts of the communication (sockets, etc.)
>> which gevent does at the Python level.
>>
>> Thanks,
> 
> The other problem is that asyncio itself is written in
> Python, so it won't be easy to use it from C.  I think
> it might be a good idea to speed up asyncio by implementing
> core event loop and selectors in C and to have an easy
> to use C API on top of that, but it's not in the scope
> of the PEP.

Agreed. Having the loop in C would make this easier.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 20 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/