[Python-ideas] TextIOWrapper callable encoding parameter

[Rurpy]

As a followup, here are some timing data that seem to confirm
a modest increase in speed as a result of implementing the
callable encoding parameter I proposed (although that would 
not be the main reason for wanting to do it.)  These are just
for illustration.  (Among many other reasons, _pyio benchmarks
are not very useful.)

I read four short test files using four methods for determining 
the test file's encoding.  The test files are a simplified model 
of a python coding declaration (always on first line in our case 
with no BOM present [*1]) followed by mixed english and japanese 
text.

Method 0 (reopen0): 
Use the encoding callable I am proposing.

   def reopen0 (fname):
        def hook (data,buf):
            return get_encoding (data)
        t = io.open (fname, encoding=hook)

Method 1 (reopen1):
Open in binary to determine encoding, then rewrap in a 
TextIOWrapper with the correct encoding.

    def reopen1 (fname):
        b = io.open (fname, 'rb')
        line = b.readline()
        enc = get_encoding (line)
        b.seek (0)
        t = io.TextIOWrapper (b, enc, line_buffering=True)
        t.mode = 'r'

Method 2 (reopen2):
Open in binary to determine encoding, then reopen in text mode
with correct encoding.

    def reopen2 (fname):
        b = io.open (fname, 'rb')
        line = b.readline()
        enc = get_encoding (line)
        t = io.open (fname, encoding=enc)

Method 3 (reopen3):
Open in text mode (latin1) to determine encoding, then reopen
in text mode with correct encoding.

    def reopen3 (fname):
        f = io.open (fname, encoding='latin1')
        line = f.readline()
        enc = get_encoding (line)
        t = io.open (fname, encoding=enc)

The same get_encoding() function is used in all methods [*1].

The input test data are all small files (because we want
to measure encoding detection, not how fast read() runs.)
Each has a python/emacs coding declaration in the first line.

test.utf8 -- Tiny python program with coding declaration 
  and single print statement in main() function that prints
  a short word (literal) in Japanese.  Encoding is utf-8
  (122 bytes).
test.sjis -- Identical to test.utf8 but sjis encoding
  (111 bytes).
test2.utf8 -- A python coding declaration followed by 
  approximately 50 long lines with mixed English and
 Japanese (4274 bytes).
test2.sjis -- Identical to test2.utf8 but sjis encoding
 (3401 bytes).

Results:
---------------------------------------------------------
$ python3 bm.py test.utf8
test.utf8 / reopen0: total time (10000 reps) was 1.188323
test.utf8 / reopen1: total time (10000 reps) was 1.490757
test.utf8 / reopen2: total time (10000 reps) was 1.766081
test.utf8 / reopen3: total time (10000 reps) was 2.141996
$ python3 bm.py test.sjis
test.sjis / reopen0: total time (10000 reps) was 1.175914
test.sjis / reopen1: total time (10000 reps) was 1.471780
test.sjis / reopen2: total time (10000 reps) was 1.764444
test.sjis / reopen3: total time (10000 reps) was 2.122550
$ python3 bm.py test2.utf8
test2.utf8 / reopen0: total time (10000 reps) was 1.690255
test2.utf8 / reopen1: total time (10000 reps) was 1.996235
test2.utf8 / reopen2: total time (10000 reps) was 2.278798
test2.utf8 / reopen3: total time (10000 reps) was 2.727867
$ python3 bm.py test2.sjis
test2.sjis / reopen0: total time (10000 reps) was 1.841388
test2.sjis / reopen1: total time (10000 reps) was 2.147142
test2.sjis / reopen2: total time (10000 reps) was 2.426701
test2.sjis / reopen3: total time (10000 reps) was 2.873278
----------------------------------------------------------

Here is what happen when a test data file is piped 
into a program using the four methods above:

  $ cat test.utf8 | python3 stdin.py reopen0
  read 102 characters

  $ cat test.utf8 | python3 stdin.py reopen1
  got exception: [Errno 29] Illegal seek

  $ cat test.utf8 | python3 stdin.py reopen2
  read 0 characters

  $ cat test.utf8 | python3 stdin.py reopen3
  read 0 characters

----
[*1] Here is the get_encoding function used above.  It is 
a toy simplified python source encoding line reader.  Toy,
in that is looks at only one line, doesn't consider a BOM,
etc.  It purpose was to allow me to sanity check the benefits
of having a callable encoding parameter.

    def get_encoding (line):
        if isinstance (line, bytes):
            nlpos = line.index(b'\n')
            mo = ENC_PATTERN_B.search (line, 0, nlpos)
            if not mo: return None
            enc = mo.group(1).decode ('latin1')
        else:
            nlpos = line.index('\n')
            mo = ENC_PATTERN_S.search (line, 0, nlpos)
            if not mo: return None
            enc = mo.group(1)
        return enc