[Python-ideas] async/await in Python (C API ?)
INADA Naoki
songofacandy at gmail.com
Tue Apr 21 09:42:21 CEST 2015
I'm +1 for C API.
Writing server or client in C
To write fast networking application, low-level I/O should be done without
any Python call.
For example, fast HTTP server should accept, read and parse incoming
request in C.
To achieve it, minimum part of eventloop should provide Capsule for
watching fds
https://docs.python.org/3.4/library/asyncio-eventloop.html#watch-file-descriptors
I think we can prototyping in third party event loop like aiouv
<https://github.com/saghul/aiouv>.
Accelerating asyncio application written in Python
I've found bottleneck of aiohttp is stream.
aiohttp has own FlowControlStreamReader.
It extends aiohttp.streams.StreamReader and aiohttp.streams.StreamReader
extends asyncio.StreamReader.
It makes reading from stream slow.
To make it faster, imports some feature from aiohttp's stream to asyncio
stream, and rewrite stream in C like
io module was rewritten.
But this is hard job because stream provides/uses coroutines.
Before rewriting asyncio.stream, Python should provide way to use and
provide coroutine in C.
And aiohttp may be able to use Cython to prototype faster stream.
On Tue, Apr 21, 2015 at 4:39 AM, Ludovic Gasc <gmludo at gmail.com> wrote:
> +1 for C Accelerators for AsyncIO.
> I don't have the level to implement that, nevertheless I can contribute
> freely with two other manners: benchmarks and launch that on several
> daemons to track bugs.
>
> --
> Ludovic Gasc (GMLudo)
> http://www.gmludo.eu/
>
> 2015-04-20 20:46 GMT+02:00 M.-A. Lemburg <mal at egenix.com>:
>
>> On 20.04.2015 20:16, Andrew Svetlov wrote:
>> > Marc-Andre, I see your need but we should write C Accelerators for
>> > asyncio first.
>> > The task is not related to the PEP directly, while I like to working on
>> it soon.
>> > The problem is: accelerator should be backward compatibility and
>> > optional (no problem with it).
>> > But making C API requires much more: we need to implement the most
>> > critical parts of asyncio in C and then invent C API for that.
>> > Anyway, thank you for inspiration for desired C Async API.
>>
>> Understood, and thanks for pushing all this. Looking forward to
>> some future PEP which outlines the C API for async/await :-)
>>
>> BTW: If you need some financial backup for these things, please
>> consider writing a grant application to the PSF:
>>
>> https://www.python.org/psf/grants/
>>
>> Cheers,
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source (#1, Apr 20 2015)
>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
>> ________________________________________________________________________
>>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>> Registered at Amtsgericht Duesseldorf: HRB 46611
>> http://www.egenix.com/company/contact/
>>
>> > On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> >> On 20.04.2015 17:17, Yury Selivanov wrote:
>> >>>
>> >>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote:
>> >>>> On 20.04.2015 15:40, Yury Selivanov wrote:
>> >>>>> Hi Marc-Andre,
>> >>>>>
>> >>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:
>> >>>>>> Hi Yury,
>> >>>>>>
>> >>>>>> do you have any plans of also exposing an easy to use C API
>> >>>>>> for async and await ?
>> >>>>>>
>> >>>>>> I think this would be useful to e.g. implement asynchronous
>> >>>>>> database access. Most of those adapters are written in C and
>> >>>>>> need so a C API would help a lot with this.
>> >>>>> I think one way to have a convenient C API is to implement
>> >>>>> a Future-like object in C -- an object with __await__ method
>> >>>>> that should return an iterator, which should also be
>> >>>>> implemented in C and provide a way to attach callback
>> >>>>> or to *pipeline* C functions.
>> >>>>>
>> >>>>> This way C code should seamlessly integrate with asyncio.
>> >>>>>
>> >>>>> I doubt that we have time to include this kind of API in
>> >>>>> the PEP, but it can be developed as a separate library.
>> >>>>>
>> >>>>> Your thoughts?
>> >>>> I'm just sketching here, since I don't have much experience
>> >>>> with writing async code (I'm using threads and multiple
>> >>>> processes instead) and it probably shows ;-)
>> >>>>
>> >>>> From the C side of things, I think using callbacks
>> >>>> and a way to register/unregister polling functions
>> >>>> would be great to have.
>> >>>>
>> >>>> The C code could then run like this in polling mode:
>> >>>>
>> >>>> long_running_query(...) {
>> >>>> /* Start long running operation... creating a handle for it */
>> >>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);
>> >>>> PyAsync_ReturnAwait();
>> >>>> }
>> >>>>
>> >>>> mypolling_fct(int handle) {
>> >>>> /* Check completion of operation */
>> >>>> if (completed) {
>> >>>> /* Fetch results into value */
>> >>>> PyAsync_UnregisterPoll(mypolling_fct, handle);
>> >>>> PyAsync_ReturnReady(value);
>> >>>> }
>> >>>> else
>> >>>> PyAsync_ReturnContinue();
>> >>>> }
>> >>>>
>> >>>> and like this in callback mode:
>> >>>>
>> >>>> long_running_query(...) {
>> >>>> /* Create a handle for a long running operation... */
>> >>>> PyAsync_RegisterCallback(myquery_done, handle, timeout);
>> >>>> /* Start long running operation... */
>> >>>> PyAsync_ReturnAwait();
>> >>>> }
>> >>>>
>> >>>> myquery_done(int handle, PyObject *value) {
>> >>>> PyAsync_UnregisterCallback(myquery_done, handle);
>> >>>> PyAsync_ReturnReady(value);
>> >>>> }
>> >>>>
>> >>>> Both mechanisms will likely have to move the long running
>> >>>> operation into a separate (blocking) thread, unless there's also
>> >>>> a way to register an event loop selector somewhere and the C lib
>> >>>> your interfacing to provides this mechanism as well.
>> >>>>
>> >>>
>> >>> I see what you want to do and why now.
>> >>>
>> >>> However, I can't imagine how this could be implemented
>> >>> without asyncio event loop being implemented in C.
>> >>> Coroutines from PEP 492 aren't an indepenedant concept,
>> >>> they require a library/framework with an event loop
>> >>> to make them work.
>> >>>
>> >>> The only way to go right now is to get 'get_event_loop()'
>> >>> and 'loop.call_soon()' working in C using Python C API.
>> >>>
>> >>> From that point you can implement Future object in C,
>> >>> see 'asyncio/futures.py'; Future._schedule_callbacks,
>> >>> Future.set_result and Future.__iter__ methods in
>> >>> particular.
>> >>>
>> >>> With all that, you can easily return your future object
>> >>> from C methods and everything should work just fine.
>> >>>
>> >>> I know this isn't ideal, and we actually should start a
>> >>> separate discussion on how we can improve things on
>> >>> our asyncio C side.
>> >>
>> >> I'm not expecting this to be ready anytime soon,
>> >> just wanted to bring this up, since it may be something
>> >> that has to be considered in the async/await API design
>> >> early rather than later to not make it too complex to
>> >> add later on.
>> >>
>> >> I'm not sure, but going through the Python API from C
>> >> can potentially result in the code to reenter which sounds
>> >> like it could cause some unwanted race conditions. A C API
>> >> would avoid this.
>> >>
>> >> Thanks,
>> >> --
>> >> Marc-Andre Lemburg
>> >> eGenix.com
>> >>
>> >> Professional Python Services directly from the Source (#1, Apr 20
>> 2015)
>> >>>>> Python Projects, Coaching and Consulting ...
>> http://www.egenix.com/
>> >>>>> mxODBC Plone/Zope Database Adapter ...
>> http://zope.egenix.com/
>> >>>>> mxODBC, mxDateTime, mxTextTools ...
>> http://python.egenix.com/
>> >>
>> ________________________________________________________________________
>> >>
>> >> ::::: Try our mxODBC.Connect Python Database Interface for free !
>> ::::::
>> >>
>> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
>> >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>> >> Registered at Amtsgericht Duesseldorf: HRB 46611
>> >> http://www.egenix.com/company/contact/
>> >> _______________________________________________
>> >> Python-ideas mailing list
>> >> Python-ideas at python.org
>> >> https://mail.python.org/mailman/listinfo/python-ideas
>> >> Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>> >
>> >
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
INADA Naoki <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150421/1a24d8e2/attachment-0001.html>
More information about the Python-ideas
mailing list