Question about asyncio and blocking operations
Ian Kelly
ian.g.kelly at gmail.com
Thu Jan 28 11:53:51 EST 2016
On Thu, Jan 28, 2016 at 9:40 AM, Frank Millman <frank at chagford.com> wrote:
> I have hit a snag. It feels like a bug in 'await q.get()', though I am sure
> it is just me misunderstanding how it works.
>
> I can post some working code if necessary, but here is a short description.
>
> Here is the database handler - 'request_queue' is a queue.Queue -
>
> while not request_queue.empty():
> return_queue, sql = request_queue.get()
> cur.execute(sql)
> for row in cur:
> return_queue.put_nowait(row)
> return_queue.put_nowait(None)
> request_queue.task_done()
As I commented in my previous message, asyncio.Queue is not
thread-safe, so it's very important that the put calls here be done on
the event loop thread using event_loop.call_soon_threadsafe. This
could be the cause of the strange behavior you're seeing in getting
the results.
> The caller requests some data from the database like this.
>
> return_queue = asyncio.Queue()
> sql = 'SELECT ...'
> request_queue.put((return_queue, sql))
Note that since this is a queue.Queue, the put call has the potential
to block your entire event loop.
More information about the Python-list
mailing list