[issue20501] fileinput module will read whole file into memory when using fileinput.hook_encoded

[Gunnar Aastrand Grimnes]

New submission from Gunnar Aastrand Grimnes:

When reading large files with fileinput, it will work as expected and only process a line at a time when used normally, but if you add an hook_encoded openhook it will read the whole file into memory before returning the first line. 

Verify by running this program on a large text file: 

import fileinput

for l in fileinput.input(openhook=fileinput.hook_encoded('iso-8859-1')):
    raw_input()

and check how much memory it uses. Remove the openhook and memory usage goes down to nothing.

----------
components: IO
messages: 210130
nosy: gromgull
priority: normal
severity: normal
status: open
title: fileinput module will read whole file into memory when using fileinput.hook_encoded
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20501>
_______________________________________