[New-bugs-announce] [issue33000] IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / character stored

[John Brearley]

New submission from John Brearley <brearley at bell.net>:

While running a tensorflow script in the IDLEX GUI that runs for 8 million steps and produce 2 lines stdout per step, my PC used all 16GB RAM and crashed the python process, not to mention messed up other apps, like Firefox & Norton AntiVirus. While the RAM was recovered, Firefox started responding, but Norton Antivirus didn’t, so the PC had to be rebooted. 
 
The issue is easily reproduced with the short print loop that dumps 20K lines of stdout, at 171 characters / line on the IDLEX GUI window. When the script is run in the IDLEX GUI, the Windows Task Manager shows the python process start at 19MB RAM consumption, then grows to 569MB RAM consumption. If I run the script a second time in the same IDLEX GUI window, it grows to 1.1GB RAM consumption. 
 
So 20K lines off output at 171 characters / line (“i: nnnnn” prefix + 2 * 80 byte string + newline) = 3.4M total characters stored in the scrollback buffer. The delta memory consumed was 569MB – 19MB = 550MB. The RAM consumed / character is 550MB / 3.4M = 161 bytes / character. This seems excessively inefficient.
 
I now understand how the tensorflow script would stop after 550K iterations and the 550K lines of stdout in the IDLEX GUI would consume all 16GB RAM on my PC.
 
BTW, when I run the same test script in the WinPython command prompt window, it only consumes 4MB RAM while it runs. However the scrollback buffer is limited to 10K lines, wrapped at the 80 character mark, so much less data saved.
 
I haven’t found any options in IDLEX GUI to limit the scrollback buffer size.
 
My request is to review the scrollback memory storage algorithms. If nothing can be done to improve them, then please add a circular buffer to limit the memory consumption.
 
# Print loop to test memory consumption in Python IDLEX GUI.
s1 = "0123456789"
s2 = s1+s1+s1+s1+s1+s1+s1+s1
for i in range(20000):
   print("i:", i, s2, s2)

I am using Python 3.6.4 on Windows 7 PC, Intel i7-4770S, 3.1GHz, 16GB RAM.

----------
assignee: terry.reedy
components: IDLE
messages: 313263
nosy: jbrearley, terry.reedy
priority: normal
severity: normal
status: open
title: IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / character stored
type: resource usage
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33000>
_______________________________________