[DB-SIG] PEP 249
Mike Bayer
mike_mp at zzzcomputing.com
Sun Oct 21 09:02:39 EDT 2018
On Sun, Oct 21, 2018 at 5:53 AM M.-A. Lemburg <mal at egenix.com> wrote:
>
> On 20.10.2018 19:13, Mike Bayer wrote:
> >
> >
> > On Sat, Oct 20, 2018, 5:02 AM M.-A. Lemburg <mal at egenix.com
> > <mailto:mal at egenix.com>> wrote:
> >
> > On 18.10.2018 16:07, Mike Bayer wrote:
> > > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg <mal at egenix.com
> > <mailto:mal at egenix.com>> wrote:
> > >>
> > >> On 16.10.2018 16:37, Mike Bayer wrote:
> > >>> AFAIK , Postgresql is the only major relational database that has an
> > >>> actual non-blocking protocol. The other databases all require
> > that
> > >>> "async" be simulated in the context of blocking IO.
> > >>
> > >> ODBC does have an async I/O API as well, but I don't know
> > >> how wide spread the implementation of this API is among ODBC
> > >> drivers.
> > >>
> > >> Python modules talking wire protocol and thus having full
> > >> control over sockets could certainly support an async variant
> > >> of the DB-API as well.
> > >
> > > right, unusual though that given async's intense popularity in Python
> > > right now, I haven't seen any drivers doing this with say, MySQL,
> > > which you would think would be the most obvious target, considering
> > > there are multiple pure-Python drivers already.
> > >
> > >>
> > >> E.g. we have added async support to mxODBC Connect (which uses
> > >> its own wire protocol) via gevent a long time ago and it works
> > >> well in that context. We did not introduce any API changes
> > >> for this, only a config option to enable support:
> > >
> > > Well, "gevent" unfortunately does not count with the "async" crowd
> > > these days, I agree it's the easiest way to get at non-blocking
> > > behaviors but Python's asyncio is all the rage right now and it
> > > requires very dramatic API changes.
> >
> > There are certainly multiple ways to add async operations to
> > database interactions. In ODBC, you can kick off an operation
> > and then poll for completion. This has been in place for several
> > years. More recent versions also have a notification model.
> > I just brought up gevent, because it's certainly the easiest
> > way to add async behavior to an existing application.
> >
> > It's also possible to integrate gevent into Python's
> > asyncio, but I'm not sure how far that has been followed
> > through... https://github.com/gevent/gevent/issues/982
> >
> > The native Python support requires deep changes in the code.
> > In particular, you have to make sure that all methods down
> > to the socket calls are async.
> >
> > Problem is that many database modules eventually interface to
> > a database C lib and those don't necessarily give you back
> > control in case an operation takes longer, so it's not
> > always possible to implement this, unless you work with threads
> > to mimic async behavior - which is kind of besides the point :-)
> >
> >
> >
> > That was my original point :) if the DBAPI is pure Python then it can
> > do asyncio / gevent without background threads, but if not, unless it is
> > using Posgresql's async API or this thing that ODBC seems to have, using
> > background threads will often be a factor, which IMO eliminates any even
> > theoretical advantage of using asyncio. I spend time regularly trying
> > to show the world that non-blocking IO (even using the phrase
> > "non-blocking IO", because a lot of the people so crazy over asyncio
> > don't even understand what non-blocking IO even means) gets us next to
> > nothing in the real world and especially not for the vast majority of
> > database use. My original blog post on this still holds:
> > http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and
> > I have a side-hobby of looking for asyncio/greenlet related threads on
> > reddit and elsewhere, finding the posts that say "Async is faster!" and
> > proving them wrong, example: https://stackoverflow.com/a/51932442/34549
> > . Most people aren't running hundreds of concurrent multi-second
> > queries at once against their database. Therefore they don't need
> > non-blocking IO.
>
> I think the main advantage of using asyncio is saturating a single
> core as much as possible given the GIL.
I've yet to see a single demonstration showing this to be more
efficient than an equivalent threaded application. This is the
theory everyone cites but misses the fact that the context switching
in gevent, asyncio is actually more expensive than that of threads
even though it happens much less frequently. asyncio does not save
you from the GIL in any way, you still have the GIL just as much as if
you didn't, and since asyncio applications use only one thread anyway,
the point is moot, unless you're saying the GIL can be disabled
altogether (is that possible?) and save on the time spent locking.
I've written multiple test suites trying to show this is the case for
gevent and asyncio without success. if someone can show me a workload
that completes more quickly with asyncio than threads then I'd finally
have something to work with, but after all of my loud outspokenness,
which I thought would attract people to prove me wrong and teach me
something, nobody has done so as of yet.
>
> It probably also helps when using multiple threads, where asyncio can
> help making most use of the cores in each thread, but it complicates
> things since you need to carefully manage the event loop - probably
> even having the asyncio loop managing them as David Beazley does in
> his thredo package (https://www.youtube.com/watch?v=xOyJiN3yGfU).
>
> Now, the GIL is rarely a problem in Python database programming
> using C database client libs, since most of times, the GIL is released
> when running queries, so not blocking other threads. It also has
> to be noted that database libs typically use background threads
> already, so might as well use them in Python as well.
>
> Anyway, we're drifting off topic I think :-)
the topic is how useful is it to have an asyncio API for pep-249. I
think the only usefulness is just that people want it due to all the
attention asyncio is getting, and that it would make integration with
asyncio applications more straightforward from a programming point of
view. That's a valid reason. Just IMO a sad one :)
>
> >
> > >>
> > >>
> > https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695
> > >>
> > >>
> > >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg <mal at egenix.com
> > <mailto:mal at egenix.com>> wrote:
> > >>>>
> > >>>> Hi Nadar,
> > >>>>
> > >>>> we do have a set of things scheduled for a version 3.0 of
> > >>>> the DB API (see the python.org <http://python.org> wiki), but
> > nothing related to
> > >>>> async I/O yet.
> > >>>>
> > >>>> FWIW: I think the existing APIs can be used in an async form
> > >>>> as well, without having to specify new ones, by simply
> > >>>> providing an async entry point at the module level or
> > >>>> even making the DB module itself be async only.
> > >>>>
> > >>>> Thanks,
> > >>>> --
> > >>>> Marc-Andre Lemburg
> > >>>> eGenix.com
> > >>>>
> > >>>> Professional Python Services directly from the Experts
> > >>>>>>> Python Projects, Coaching and Consulting ...
> > http://www.egenix.com/
> > >>>>>>> Python Database Interfaces ...
> > http://products.egenix.com/
> > >>>>>>> Plone/Zope Database Interfaces ...
> > http://zope.egenix.com/
> > >>>>
> > ________________________________________________________________________
> > >>>>
> > >>>> ::: We implement business ideas - efficiently in both time and
> > costs :::
> > >>>>
> > >>>> eGenix.com Software, Skills and Services GmbH
> > Pastor-Loeh-Str.48
> > >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> > >>>> Registered at Amtsgericht Duesseldorf: HRB 46611
> > >>>> http://www.egenix.com/company/contact/
> > >>>> http://www.malemburg.com/
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 11.10.2018 21:48, Nader Hamady wrote:
> > >>>>> Hello.
> > >>>>>
> > >>>>> Reaching out to find out if there have been any discusdions
> > around a 3.0
> > >>>>> Database API Specification - Or something broader around async
> > >>>>> specifications for dabatase transactions.
> > >>>>>
> > >>>>> Thanks in advance,
> > >>>>>
> > >>>>> -N
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Hello.
> > >>>>>
> > >>>>> Reaching out to find out if there have been any discusdions
> > around a 3.0
> > >>>>> Database API Specification - Or something broader around async
> > >>>>> specifications for dabatase transactions.
> > >>>>>
> > >>>>> Thanks in advance,
> > >>>>>
> > >>>>> -N
> > >>>>>
> > >>>>> --
> > >>>>> Nader B. Hamady
> > >>>>>
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> > >>>>> https://mail.python.org/mailman/listinfo/db-sig
> > >>>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> > >>>> https://mail.python.org/mailman/listinfo/db-sig
> > >>
> > >> --
> > >> Marc-Andre Lemburg
> > >> eGenix.com
> > >>
> > >> Professional Python Services directly from the Experts
> > >>>>> Python Projects, Coaching and Consulting ...
> > http://www.egenix.com/
> > >>>>> Python Database Interfaces ...
> > http://products.egenix.com/
> > >>>>> Plone/Zope Database Interfaces ...
> > http://zope.egenix.com/
> > >>
> > ________________________________________________________________________
> > >>
> > >> ::: We implement business ideas - efficiently in both time and
> > costs :::
> > >>
> > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> > >> Registered at Amtsgericht Duesseldorf: HRB 46611
> > >> http://www.egenix.com/company/contact/
> > >> http://www.malemburg.com/
> > >>
> > > _______________________________________________
> > > DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> > > https://mail.python.org/mailman/listinfo/db-sig
> > >
> >
> > --
> > Marc-Andre Lemburg
> > eGenix.com
> >
> > Professional Python Services directly from the Experts
> > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
> > >>> Python Database Interfaces ... http://products.egenix.com/
> > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
> > ________________________________________________________________________
> >
> > ::: We implement business ideas - efficiently in both time and costs :::
> >
> > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> > Registered at Amtsgericht Duesseldorf: HRB 46611
> > http://www.egenix.com/company/contact/
> > http://www.malemburg.com/
> >
> >
> >
> > _______________________________________________
> > DB-SIG maillist - DB-SIG at python.org
> > https://mail.python.org/mailman/listinfo/db-sig
> >
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts
> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
> >>> Python Database Interfaces ... http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/
> http://www.malemburg.com/
>
More information about the DB-SIG
mailing list