[Python-ideas] async/await in Python (C API ?)
M.-A. Lemburg
mal at egenix.com
Mon Apr 20 20:46:28 CEST 2015
On 20.04.2015 20:16, Andrew Svetlov wrote:
> Marc-Andre, I see your need but we should write C Accelerators for
> asyncio first.
> The task is not related to the PEP directly, while I like to working on it soon.
> The problem is: accelerator should be backward compatibility and
> optional (no problem with it).
> But making C API requires much more: we need to implement the most
> critical parts of asyncio in C and then invent C API for that.
> Anyway, thank you for inspiration for desired C Async API.
Understood, and thanks for pushing all this. Looking forward to
some future PEP which outlines the C API for async/await :-)
BTW: If you need some financial backup for these things, please
consider writing a grant application to the PSF:
https://www.python.org/psf/grants/
Cheers,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Apr 20 2015)
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
> On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 20.04.2015 17:17, Yury Selivanov wrote:
>>>
>>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote:
>>>> On 20.04.2015 15:40, Yury Selivanov wrote:
>>>>> Hi Marc-Andre,
>>>>>
>>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:
>>>>>> Hi Yury,
>>>>>>
>>>>>> do you have any plans of also exposing an easy to use C API
>>>>>> for async and await ?
>>>>>>
>>>>>> I think this would be useful to e.g. implement asynchronous
>>>>>> database access. Most of those adapters are written in C and
>>>>>> need so a C API would help a lot with this.
>>>>> I think one way to have a convenient C API is to implement
>>>>> a Future-like object in C -- an object with __await__ method
>>>>> that should return an iterator, which should also be
>>>>> implemented in C and provide a way to attach callback
>>>>> or to *pipeline* C functions.
>>>>>
>>>>> This way C code should seamlessly integrate with asyncio.
>>>>>
>>>>> I doubt that we have time to include this kind of API in
>>>>> the PEP, but it can be developed as a separate library.
>>>>>
>>>>> Your thoughts?
>>>> I'm just sketching here, since I don't have much experience
>>>> with writing async code (I'm using threads and multiple
>>>> processes instead) and it probably shows ;-)
>>>>
>>>> From the C side of things, I think using callbacks
>>>> and a way to register/unregister polling functions
>>>> would be great to have.
>>>>
>>>> The C code could then run like this in polling mode:
>>>>
>>>> long_running_query(...) {
>>>> /* Start long running operation... creating a handle for it */
>>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);
>>>> PyAsync_ReturnAwait();
>>>> }
>>>>
>>>> mypolling_fct(int handle) {
>>>> /* Check completion of operation */
>>>> if (completed) {
>>>> /* Fetch results into value */
>>>> PyAsync_UnregisterPoll(mypolling_fct, handle);
>>>> PyAsync_ReturnReady(value);
>>>> }
>>>> else
>>>> PyAsync_ReturnContinue();
>>>> }
>>>>
>>>> and like this in callback mode:
>>>>
>>>> long_running_query(...) {
>>>> /* Create a handle for a long running operation... */
>>>> PyAsync_RegisterCallback(myquery_done, handle, timeout);
>>>> /* Start long running operation... */
>>>> PyAsync_ReturnAwait();
>>>> }
>>>>
>>>> myquery_done(int handle, PyObject *value) {
>>>> PyAsync_UnregisterCallback(myquery_done, handle);
>>>> PyAsync_ReturnReady(value);
>>>> }
>>>>
>>>> Both mechanisms will likely have to move the long running
>>>> operation into a separate (blocking) thread, unless there's also
>>>> a way to register an event loop selector somewhere and the C lib
>>>> your interfacing to provides this mechanism as well.
>>>>
>>>
>>> I see what you want to do and why now.
>>>
>>> However, I can't imagine how this could be implemented
>>> without asyncio event loop being implemented in C.
>>> Coroutines from PEP 492 aren't an indepenedant concept,
>>> they require a library/framework with an event loop
>>> to make them work.
>>>
>>> The only way to go right now is to get 'get_event_loop()'
>>> and 'loop.call_soon()' working in C using Python C API.
>>>
>>> From that point you can implement Future object in C,
>>> see 'asyncio/futures.py'; Future._schedule_callbacks,
>>> Future.set_result and Future.__iter__ methods in
>>> particular.
>>>
>>> With all that, you can easily return your future object
>>> from C methods and everything should work just fine.
>>>
>>> I know this isn't ideal, and we actually should start a
>>> separate discussion on how we can improve things on
>>> our asyncio C side.
>>
>> I'm not expecting this to be ready anytime soon,
>> just wanted to bring this up, since it may be something
>> that has to be considered in the async/await API design
>> early rather than later to not make it too complex to
>> add later on.
>>
>> I'm not sure, but going through the Python API from C
>> can potentially result in the code to reenter which sounds
>> like it could cause some unwanted race conditions. A C API
>> would avoid this.
>>
>> Thanks,
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source (#1, Apr 20 2015)
>>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
>> ________________________________________________________________________
>>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>> Registered at Amtsgericht Duesseldorf: HRB 46611
>> http://www.egenix.com/company/contact/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
More information about the Python-ideas
mailing list