Question about asyncio and blocking operations
Frank Millman
frank at chagford.com
Wed Jan 27 09:40:29 EST 2016
"Ian Kelly" wrote in message
news:CALwzidk-RBkB-vi6CgcEeoFHQrsoTFvqX9MqzDD=rnY5bOCRUg at mail.gmail.com...
> On Tue, Jan 26, 2016 at 7:15 AM, Frank Millman <frank at chagford.com> wrote:
> >
> > If I return the cursor, I can iterate over it, but isn't this a blocking
> > operation? As far as I know, the DB adaptor will only actually retrieve
> > the
> > row when requested.
> >
> > If I am right, I should call fetchall() while inside get_rows(), and
> > return
> > all the rows as a list.
> >
>
> You probably want an asynchronous iterator here. If the cursor doesn't
> provide that, then you can wrap it in one. In fact, this is basically
> one of the examples in the PEP:
> https://www.python.org/dev/peps/pep-0492/#example-1
>
Thanks, Ian. I had a look, and it does seem to fit the bill, but I could not
get it to work, and I am running out of time.
Specifically, I tried to get it working with the sqlite3 cursor. I am no
expert, but after some googling I tried this -
import sqlite3
conn = sqlite3.connect('/sqlite_db')
cur = conn.cursor()
async def __aiter__(self):
return self
async def __anext__(self):
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.__next__)
import types
cur.__aiter__ = types.MethodType( __aiter__, cur )
cur.__anext__ = types.MethodType( __anext__, cur )
It failed with this exception -
AttributeError: 'sqlite3.Cursor' object has no attribute '__aiter__'
I think this is what happens if a class uses 'slots' to define its
attributes - it will not permit the creation of a new one.
Anyway, moving on, I decided to change tack. Up to now I have been trying to
isolate the function where I actually communicate with the database, and
wrap that in a Future with 'run_in_executor'.
In practice, the vast majority of my interactions with the database consist
of very small CRUD commands, and will have minimal impact on response times
even if they block. So I decided to focus on a couple of functions which are
larger, and try to wrap the entire function in a Future with
'run_in_executor'.
It seems to be working, but it looks a bit odd, so I will show what I am
doing and ask for feedback.
Assume a slow function -
async def slow_function(arg1, arg2):
[do stuff]
It now looks like this -
async def slow_function(arg1, arg2):
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, slow_function_1, arg1, arg2)
def slow_function_1(self, arg1, arg2):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(slow_function_2(arg1, arg2))
async slow_function_2(arg1, arg2):
[do stuff]
Does this look right?
Frank
More information about the Python-list
mailing list