[issue22526] file iteration SystemError for huge lines (2GiB+)

Mon Oct 6 14:04:18 CEST 2014

STINNER Victor added the comment:

On Fedora 20/x86_64, running test_file2k takes up to 4.8 GB (5114316 kB) of RSS memory (VmPeak in /proc/pid/status).

It looks like readahead_get_line_skip() has an efficient code to handle buffer. It uses recursive calls:
---
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.0 MB
readahead: allocate 0.1 MB
readahead: allocate 0.1 MB
readahead: allocate 0.1 MB
readahead: allocate 0.1 MB
readahead: allocate 0.1 MB
readahead: allocate 0.2 MB
readahead: allocate 0.2 MB
readahead: allocate 0.3 MB
readahead: allocate 0.3 MB
readahead: allocate 0.4 MB
readahead: allocate 0.5 MB
readahead: allocate 0.7 MB
readahead: allocate 0.8 MB
readahead: allocate 1.1 MB
readahead: allocate 1.3 MB
readahead: allocate 1.7 MB
readahead: allocate 2.1 MB
readahead: allocate 2.6 MB
readahead: allocate 3.2 MB
readahead: allocate 4.0 MB
readahead: allocate 5.0 MB
readahead: allocate 6.3 MB
readahead: allocate 7.9 MB
readahead: allocate 9.9 MB
readahead: allocate 12.3 MB
readahead: allocate 15.4 MB
readahead: allocate 19.3 MB
readahead: allocate 24.1 MB
readahead: allocate 30.1 MB
readahead: allocate 37.6 MB
readahead: allocate 47.0 MB
readahead: allocate 58.8 MB
readahead: allocate 73.5 MB
readahead: allocate 91.8 MB
readahead: allocate 114.8 MB
readahead: allocate 143.5 MB
readahead: allocate 179.4 MB
readahead: allocate 224.2 MB
readahead: allocate 280.2 MB
readahead: allocate 350.3 MB
readahead: allocate 437.9 MB
readahead: allocate 547.3 MB

Breakpoint 2, PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792
792	    if (nbytes > PY_SSIZE_T_MAX)
(gdb) where
#0  PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792
#1  0x0000000000464af1 in _PyObject_DebugMallocApi (id=111 'o', nbytes=2147483701) at Objects/obmalloc.c:1474
#2  0x0000000000464a48 in _PyObject_DebugMalloc (nbytes=2147483701) at Objects/obmalloc.c:1441
#3  0x000000000046efdc in PyString_FromStringAndSize (str=0x0, size=2147483648) at Objects/stringobject.c:88
#4  0x0000000000436c30 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2147483648, bufsize=573933340) at Objects/fileobject.c:2291
#5  0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1836553986, bufsize=459146672) at Objects/fileobject.c:2311
#6  0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1469236648, bufsize=367317338) at Objects/fileobject.c:2311
#7  0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1175382777, bufsize=293853871) at Objects/fileobject.c:2311
#8  0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=940299680, bufsize=235083097) at Objects/fileobject.c:2311
#9  0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=752233202, bufsize=188066478) at Objects/fileobject.c:2311
#10 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=601780019, bufsize=150453183) at Objects/fileobject.c:2311
#11 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=481417472, bufsize=120362547) at Objects/fileobject.c:2311
#12 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=385127434, bufsize=96290038) at Objects/fileobject.c:2311
#13 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=308095403, bufsize=77032031) at Objects/fileobject.c:2311
#14 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=246469778, bufsize=61625625) at Objects/fileobject.c:2311
#15 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=197169278, bufsize=49300500) at Objects/fileobject.c:2311
#16 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=157728878, bufsize=39440400) at Objects/fileobject.c:2311
#17 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=126176558, bufsize=31552320) at Objects/fileobject.c:2311
#18 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=100934702, bufsize=25241856) at Objects/fileobject.c:2311
#19 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=80741217, bufsize=20193485) at Objects/fileobject.c:2311
#20 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=64586429, bufsize=16154788) at Objects/fileobject.c:2311
#21 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=51662598, bufsize=12923831) at Objects/fileobject.c:2311
#22 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=41323533, bufsize=10339065) at Objects/fileobject.c:2311
#23 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=33052281, bufsize=8271252) at Objects/fileobject.c:2311
#24 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=26435279, bufsize=6617002) at Objects/fileobject.c:2311
#25 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=21141677, bufsize=5293602) at Objects/fileobject.c:2311
#26 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=16906795, bufsize=4234882) at Objects/fileobject.c:2311
#27 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=13518889, bufsize=3387906) at Objects/fileobject.c:2311
#28 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=10808564, bufsize=2710325) at Objects/fileobject.c:2311
#29 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=8640304, bufsize=2168260) at Objects/fileobject.c:2311
#30 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=6905696, bufsize=1734608) at Objects/fileobject.c:2311
#31 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=5518009, bufsize=1387687) at Objects/fileobject.c:2311
#32 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=4407859, bufsize=1110150) at Objects/fileobject.c:2311
#33 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=3519739, bufsize=888120) at Objects/fileobject.c:2311
#34 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2809243, bufsize=710496) at Objects/fileobject.c:2311
#35 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2240846, bufsize=568397) at Objects/fileobject.c:2311
#36 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1786128, bufsize=454718) at Objects/fileobject.c:2311
#37 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1422353, bufsize=363775) at Objects/fileobject.c:2311
#38 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1131333, bufsize=291020) at Objects/fileobject.c:2311
#39 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=898517, bufsize=232816) at Objects/fileobject.c:2311
#40 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=712264, bufsize=186253) at Objects/fileobject.c:2311
#41 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=563261, bufsize=149003) at Objects/fileobject.c:2311
#42 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=444058, bufsize=119203) at Objects/fileobject.c:2311
#43 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=348695, bufsize=95363) at Objects/fileobject.c:2311
#44 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=272404, bufsize=76291) at Objects/fileobject.c:2311
#45 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=211371, bufsize=61033) at Objects/fileobject.c:2311
#46 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=162544, bufsize=48827) at Objects/fileobject.c:2311
#47 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=123482, bufsize=39062) at Objects/fileobject.c:2311
#48 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=92232, bufsize=31250) at Objects/fileobject.c:2311
#49 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=67232, bufsize=25000) at Objects/fileobject.c:2311
#50 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=47232, bufsize=20000) at Objects/fileobject.c:2311
#51 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=31232, bufsize=16000) at Objects/fileobject.c:2311
#52 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=18432, bufsize=12800) at Objects/fileobject.c:2311
#53 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=8192, bufsize=10240) at Objects/fileobject.c:2311
#54 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=0, bufsize=8192) at Objects/fileobject.c:2311
#55 0x0000000000436e52 in file_iternext (f=0x7fffeea2bf40) at Objects/fileobject.c:2335
---

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22526>
_______________________________________