[New-bugs-announce] [issue23812] asyncio.Queue.put_nowait(), followed get() task cancellation leads to item being lost

[Gustavo J. A. M. Carneiro]

New submission from Gustavo J. A. M. Carneiro:

I have a pattern where I read from a queue with a timeout, generally like this:

while True:
  reader = asyncio.async(wait_for(queue.get(), 0.1))
  try:
    item = (yield from reader)
  except asyncio.TimeoutError:
    reader.cancel()
    continue

This is to have a loop where we try to get items from the queue with a timeout.  Prior to Python 3.5, wait_for() doesn't automatically cancel the reading task, so I have to do it explicitly above.

The code has a strange race condition where, if a taks calls put_nowait on a reader task that is just about to be cancelled due to timeout, then the item that wast put onto the queue gets lost.

In the tests framework, the minimal test case I can come up with is mainly this code:


  q = asyncio.Queue(loop=loop)
  reader = loop.create_task(q.get())
  loop.run_until_complete(asyncio.sleep(0.01, loop=loop))

  q.put_nowait(1)
  q.put_nowait(2)
  reader.cancel()

When the reader gets cancelled, the item `1` is lost into the ether, and when you create another reader for the same queue, it only gets the second item `2`.  I would expect that, if a reader task is cancelled, the item at the head of the queue doesn't get lost.  Either the reader succeeds and the item is removed, or it doesn't succeed and the item is left alone.

I attach a patch to the tulip tests that reproduces the problem in both Python 3.4.2 and tulip hg.

----------
components: asyncio
files: bug-test.diff
keywords: patch
messages: 239611
nosy: gustavo, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: asyncio.Queue.put_nowait(), followed get() task cancellation leads to item being lost
type: behavior
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file38740/bug-test.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23812>
_______________________________________