[DB-SIG] asyncio support

[M.-A. Lemburg]

On 12.11.2013 12:03, INADA Naoki wrote:
> On Tue, Nov 12, 2013 at 7:41 PM, M.-A. Lemburg <mal at python.org> wrote:
> 
>> On 11.11.2013 10:43, INADA Naoki wrote:
>>> I think having definition about what method returns Future is good.
>>>
>>> For example:
>>>
>>> con = yield from pymysql.connect_async(...)  # connection may cause
>>> cur = con.cursor()  # cursor creation should done without asyncio
>>> yield from cur.execute("...")  # execute may use asyncio
>>> row = yield from cur.fetchone()  # fetch also uses asyncio
>>>
>>> for row in cur:  # raises NotImplementedError
>>
>> This can already be implemented using e.g. gevent, provided the
>> client side is implemented in Python (and gevent can thus patch
>> the socket module).
>>
> 
> I don't talk about "How to use connection with asyncio?"
> 
> I'm one of committer of PyMySQL and I want to support asyncio.
> 
> And I think standard api for supporting asyncio is nice.

With asyncio you mean the new tulip support in Python 3.4 ?

We could add something as standard extension of the DB-API,
but not (yet) to the standard itself, since the DB-API still
has to support Python 2.x.

However, before coming up with a standard extension, I'd suggest
to first get some idea of what the asyncio best practices are.

Ideally, it should also be possible to implement asynchronous
.execute() and .fetch...() without using the asyncio module
and in Python versions prior to Python 3.4.

>> For native C client libs, this will not easily be possible, since
>> the Python wrapper will typically not get access to the underlying
>> socket logic, so you have to rely on native async support in the
>> library.
>>
>> E.g. ODBC has limited support for this, but uses an active
>> polling mechanism, so callbacks won't work and I'm not
>> sure Futures would directly. You would have to hook up the
>> polling with the async event loop.
>>
> 
> Yes. So async API should be optional.
> 
> 
>>
>> What always works is using threads as a way to let other
>> code continue to run. .execute() and .fetch...() usually release
>> the GIL for this purpose.
>>
> 
> I want to make PyMySQL works with asyncio in single thread.
> 
> Does any developers of DB connectors concern about asyncio?

I'm not sure I understand the question.

I've looked into adding async support to our native mxODBC
interface, but the polling mechanism used by the ODBC API
has made this somewhat complicated. It is also not clear
how expensive such a polling call would be and whether
the ODBC drivers support for async I/O is mature enough
yet (it took around 5-10 years for them to properly support
Unicode, so I don't have great expectations).

Other client libs may implement callbacks for async processing
or not support it at all.

If we come up with a standard, it would have to support
asyncio (ie. having methods return Futures), gevent,
callbacks and polling, I guess.

For mxODBC Connect, we've implemented gevent compatibility,
so you can use async processing without having to change
your code - which IMHO is the nicest way to do async
processing :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 12 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2013-11-19: Python Meeting Duesseldorf ...                  7 days to go

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/