[New-bugs-announce] [issue26259] Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.

[Jonas Brunsgaard]

New submission from Jonas Brunsgaard:

When making repeated calls to queue.get, memory is building up and is not freed until queue.push is called.

I wrote this little program to show my findings. The program will perform a lot of calls to queue.get and once every 60 seconds a queue.push is performed. Every 15 seconds the memory usage of dictionaries is printet to the console. You can find the output below the program

```
import asyncio
from pympler import muppy
from pympler import summary


q = asyncio.Queue()
loop = asyncio.get_event_loop()
closing = False


async def get_with_timeout():
    while not closing:
        try:
            task = asyncio.ensure_future(q.get())
            await asyncio.wait_for(task, 0.2)
        except asyncio.TimeoutError:
            pass

def mem_profiling():
    if not closing:
        types_ = muppy.filter(muppy.get_objects(), Type=dict)
        summary.print_(summary.summarize(types_))
        loop.call_later(15, mem_profiling)

def put():
    q.put_nowait(None)
    loop.call_later(60, put)

put()
tasks = [asyncio.ensure_future(get_with_timeout()) for _ in range(10000)]
mem_profiling()

try:
    loop.run_forever()
except KeyboardInterrupt:
    closing = True
    loop.run_until_complete(
        asyncio.ensure_future(asyncio.wait(tasks)))
finally:
    loop.close()
```

Output:

                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |       11886 |      3.20 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      224835 |     62.44 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      401487 |    111.00 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      576621 |    158.86 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      219745 |     58.99 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      390061 |    105.72 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      560623 |    152.65 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      220372 |     59.11 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============

What we see is that memory builds up ~3mb/s, and when the push method is called the memory usage returns to normal.

Is this the expected behavior or is this a bug? If it is expected I think we should update the documentation, to let people know about this behavior.

--
Jonas Brunsgaard

----------
components: asyncio
files: poc.py
messages: 259327
nosy: Jonas Brunsgaard, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.
type: resource usage
versions: Python 3.5
Added file: http://bugs.python.org/file41772/poc.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26259>
_______________________________________