<div dir="ltr">I'm +1 for C API.<br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;display:inline"></div><font size="4">Writing server or client in C</font><br><br>To write fast networking application, low-level I/O should be done without any Python call.<br><br>For example, fast HTTP server should accept, read and parse incoming request in C.<br>To achieve it, minimum part of eventloop should provide Capsule for watching fds<br><a href="https://docs.python.org/3.4/library/asyncio-eventloop.html#watch-file-descriptors">https://docs.python.org/3.4/library/asyncio-eventloop.html#watch-file-descriptors</a><br><br>I think we can prototyping in third party event loop like <a href="https://github.com/saghul/aiouv">aiouv</a>.<br><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;display:inline"></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;display:inline"></div><font size="4">Accelerating asyncio application written in Python</font><br><br>I've found bottleneck of aiohttp is stream.<br><br>aiohttp has own FlowControlStreamReader.<br>It extends aiohttp.streams.StreamReader and aiohttp.streams.StreamReader extends asyncio.StreamReader.<br>It makes reading from stream slow.<br><br>To make it faster, imports some feature from aiohttp's stream to asyncio stream, and rewrite stream in C like<br>io module was rewritten.<br><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">But this is hard job because stream provides/uses coroutines.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Before rewriting asyncio.stream, Python should provide way to use and provide coroutine in C.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">And aiohttp may be able to use Cython to prototype faster stream.</div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 21, 2015 at 4:39 AM, Ludovic Gasc <span dir="ltr"><<a href="mailto:gmludo@gmail.com" target="_blank">gmludo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">+1 for C Accelerators for AsyncIO.</div><div class="gmail_quote">I don't have the level to implement that, nevertheless I can contribute freely with two other manners: benchmarks and launch that on several daemons to track bugs.</div><div class="gmail_quote"><br clear="all"><div><div><div dir="ltr">--<br><div>Ludovic Gasc (GMLudo)</div><div><a href="http://www.gmludo.eu/" target="_blank">http://www.gmludo.eu/</a></div></div></div></div></div><div><div class="h5"><div class="gmail_quote"><br></div><div class="gmail_quote">2015-04-20 20:46 GMT+02:00 M.-A. Lemburg <span dir="ltr"><<a href="mailto:mal@egenix.com" target="_blank">mal@egenix.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On 20.04.2015 20:16, Andrew Svetlov wrote:<br>
> Marc-Andre, I see your need but we should write C Accelerators for<br>
> asyncio first.<br>
> The task is not related to the PEP directly, while I like to working on it soon.<br>
> The problem is: accelerator should be backward compatibility and<br>
> optional (no problem with it).<br>
> But making C API requires much more: we need to implement the most<br>
> critical parts of asyncio in C and then invent C API for that.<br>
> Anyway, thank you for inspiration for desired C Async API.<br>
<br>
</span>Understood, and thanks for pushing all this. Looking forward to<br>
some future PEP which outlines the C API for async/await :-)<br>
<br>
BTW: If you need some financial backup for these things, please<br>
consider writing a grant application to the PSF:<br>
<br>
<a href="https://www.python.org/psf/grants/" target="_blank">https://www.python.org/psf/grants/</a><br>
<br>
Cheers,<br>
<div><div>--<br>
Marc-Andre Lemburg<br>
eGenix.com<br>
<br>
Professional Python Services directly from the Source (#1, Apr 20 2015)<br>
>>> Python Projects, Coaching and Consulting ... <a href="http://www.egenix.com/" target="_blank">http://www.egenix.com/</a><br>
>>> mxODBC Plone/Zope Database Adapter ... <a href="http://zope.egenix.com/" target="_blank">http://zope.egenix.com/</a><br>
>>> mxODBC, mxDateTime, mxTextTools ... <a href="http://python.egenix.com/" target="_blank">http://python.egenix.com/</a><br>
________________________________________________________________________<br>
<br>
::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::<br>
<br>
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48<br>
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg<br>
Registered at Amtsgericht Duesseldorf: HRB 46611<br>
<a href="http://www.egenix.com/company/contact/" target="_blank">http://www.egenix.com/company/contact/</a><br>
<br>
> On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg <<a href="mailto:mal@egenix.com" target="_blank">mal@egenix.com</a>> wrote:<br>
>> On 20.04.2015 17:17, Yury Selivanov wrote:<br>
>>><br>
>>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote:<br>
>>>> On 20.04.2015 15:40, Yury Selivanov wrote:<br>
>>>>> Hi Marc-Andre,<br>
>>>>><br>
>>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:<br>
>>>>>> Hi Yury,<br>
>>>>>><br>
>>>>>> do you have any plans of also exposing an easy to use C API<br>
>>>>>> for async and await ?<br>
>>>>>><br>
>>>>>> I think this would be useful to e.g. implement asynchronous<br>
>>>>>> database access. Most of those adapters are written in C and<br>
>>>>>> need so a C API would help a lot with this.<br>
>>>>> I think one way to have a convenient C API is to implement<br>
>>>>> a Future-like object in C -- an object with __await__ method<br>
>>>>> that should return an iterator, which should also be<br>
>>>>> implemented in C and provide a way to attach callback<br>
>>>>> or to *pipeline* C functions.<br>
>>>>><br>
>>>>> This way C code should seamlessly integrate with asyncio.<br>
>>>>><br>
>>>>> I doubt that we have time to include this kind of API in<br>
>>>>> the PEP, but it can be developed as a separate library.<br>
>>>>><br>
>>>>> Your thoughts?<br>
>>>> I'm just sketching here, since I don't have much experience<br>
>>>> with writing async code (I'm using threads and multiple<br>
>>>> processes instead) and it probably shows ;-)<br>
>>>><br>
>>>> From the C side of things, I think using callbacks<br>
>>>> and a way to register/unregister polling functions<br>
>>>> would be great to have.<br>
>>>><br>
>>>> The C code could then run like this in polling mode:<br>
>>>><br>
>>>> long_running_query(...) {<br>
>>>> /* Start long running operation... creating a handle for it */<br>
>>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);<br>
>>>> PyAsync_ReturnAwait();<br>
>>>> }<br>
>>>><br>
>>>> mypolling_fct(int handle) {<br>
>>>> /* Check completion of operation */<br>
>>>> if (completed) {<br>
>>>> /* Fetch results into value */<br>
>>>> PyAsync_UnregisterPoll(mypolling_fct, handle);<br>
>>>> PyAsync_ReturnReady(value);<br>
>>>> }<br>
>>>> else<br>
>>>> PyAsync_ReturnContinue();<br>
>>>> }<br>
>>>><br>
>>>> and like this in callback mode:<br>
>>>><br>
>>>> long_running_query(...) {<br>
>>>> /* Create a handle for a long running operation... */<br>
>>>> PyAsync_RegisterCallback(myquery_done, handle, timeout);<br>
>>>> /* Start long running operation... */<br>
>>>> PyAsync_ReturnAwait();<br>
>>>> }<br>
>>>><br>
>>>> myquery_done(int handle, PyObject *value) {<br>
>>>> PyAsync_UnregisterCallback(myquery_done, handle);<br>
>>>> PyAsync_ReturnReady(value);<br>
>>>> }<br>
>>>><br>
>>>> Both mechanisms will likely have to move the long running<br>
>>>> operation into a separate (blocking) thread, unless there's also<br>
>>>> a way to register an event loop selector somewhere and the C lib<br>
>>>> your interfacing to provides this mechanism as well.<br>
>>>><br>
>>><br>
>>> I see what you want to do and why now.<br>
>>><br>
>>> However, I can't imagine how this could be implemented<br>
>>> without asyncio event loop being implemented in C.<br>
>>> Coroutines from PEP 492 aren't an indepenedant concept,<br>
>>> they require a library/framework with an event loop<br>
>>> to make them work.<br>
>>><br>
>>> The only way to go right now is to get 'get_event_loop()'<br>
>>> and 'loop.call_soon()' working in C using Python C API.<br>
>>><br>
>>> From that point you can implement Future object in C,<br>
>>> see 'asyncio/futures.py'; Future._schedule_callbacks,<br>
>>> Future.set_result and Future.__iter__ methods in<br>
>>> particular.<br>
>>><br>
>>> With all that, you can easily return your future object<br>
>>> from C methods and everything should work just fine.<br>
>>><br>
>>> I know this isn't ideal, and we actually should start a<br>
>>> separate discussion on how we can improve things on<br>
>>> our asyncio C side.<br>
>><br>
>> I'm not expecting this to be ready anytime soon,<br>
>> just wanted to bring this up, since it may be something<br>
>> that has to be considered in the async/await API design<br>
>> early rather than later to not make it too complex to<br>
>> add later on.<br>
>><br>
>> I'm not sure, but going through the Python API from C<br>
>> can potentially result in the code to reenter which sounds<br>
>> like it could cause some unwanted race conditions. A C API<br>
>> would avoid this.<br>
>><br>
>> Thanks,<br>
>> --<br>
>> Marc-Andre Lemburg<br>
>> eGenix.com<br>
>><br>
>> Professional Python Services directly from the Source (#1, Apr 20 2015)<br>
>>>>> Python Projects, Coaching and Consulting ... <a href="http://www.egenix.com/" target="_blank">http://www.egenix.com/</a><br>
>>>>> mxODBC Plone/Zope Database Adapter ... <a href="http://zope.egenix.com/" target="_blank">http://zope.egenix.com/</a><br>
>>>>> mxODBC, mxDateTime, mxTextTools ... <a href="http://python.egenix.com/" target="_blank">http://python.egenix.com/</a><br>
>> ________________________________________________________________________<br>
>><br>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::<br>
>><br>
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48<br>
>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg<br>
>> Registered at Amtsgericht Duesseldorf: HRB 46611<br>
>> <a href="http://www.egenix.com/company/contact/" target="_blank">http://www.egenix.com/company/contact/</a><br>
>> _______________________________________________<br>
>> Python-ideas mailing list<br>
>> <a href="mailto:Python-ideas@python.org" target="_blank">Python-ideas@python.org</a><br>
>> <a href="https://mail.python.org/mailman/listinfo/python-ideas" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
>> Code of Conduct: <a href="http://python.org/psf/codeofconduct/" target="_blank">http://python.org/psf/codeofconduct/</a><br>
><br>
><br>
><br>
_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org" target="_blank">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" target="_blank">http://python.org/psf/codeofconduct/</a><br>
</div></div></blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" target="_blank">http://python.org/psf/codeofconduct/</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">INADA Naoki <<a href="mailto:songofacandy@gmail.com" target="_blank">songofacandy@gmail.com</a>></div>
</div>