[Python-Dev] Recent changes to TextIOWrapper and its tests

[Jeff Allen]

On 19/03/2013 08:03, Serhiy Storchaka wrote:
> On 18.03.13 22:26, Jeff Allen wrote:
>> The puzzle is that it requires t.read() to succeed.
>>
>> When I insert a check for bytes type in all the places it seems
>> necessary in my code, I pass the first two conditions, but since
>> t.read() also raises TypeError, the overall test fails. Is reading the
>> stream with read() intended to succeed? Why is this desired?
>
> An alternative option is to change C implementation of 
> TextIOWrapper.read() to raise an exception in this case. However I 
> worry that it can break backward compatibility.
Thanks for this and the previous note. It is good to get it from the 
horse's mouth. I was surprised that the Python 3 version of the test was 
different here. I'd looked at the source of textio.c and found no test 
for bytes type in the n<0 branch of textiowrapper_read. Having tested it 
just now, I see that the TypeError is raised by the decoder in Py3k, 
because the input (when it is a unicode string) does not bear the buffer 
API, and not by a type test in TextIOWrapper.read() at all.

For Jython, I shall make TextIOWrapper raise TypeError and our version 
of test_io will check for it. Any incompatibility relates only to 
whether a particular mistake sometimes goes undetected, so I feel pretty 
compatible. Added to which, this is the behaviour of Python 3 and we 
feel safe anticipating Py3k in small ways.

>   Are there other tests (in other test files) which fail with a new 
> Jython TextIOWrapper?

I don't think there is anything else specific to TextIOWrapper, but if 
there is I will first treat that as a fault in our implementation. This 
is the general approach, to emulate the CPython implementation unless a 
test is clearly specific to arbitrary implementation choices. (There's a 
general exclusion for garbage collection.) In this case the test 
appeared to reflect an accident of implementation, but might just have 
been deliberate.

Parts of the Jython implementation of io not yet implemented in Java are 
supplied by a Python module _jyio. This is essentially a copy of the 
corresponding parts of _pyio, except that it has to pass the C* tests, 
not the Py* tests. In places _jyio is therefore closer to _io than is 
_pyio. For example, it makes the type tests just discussed, and passes 
CTextIOWrapperTest.test_illegal_decoder and test_initialization. 
_jyio.StringIO has getstate and setstate methods lacking in _pyio 
counterparts to pass pickling tests in test_memoryio. This might be of 
interest to CPython for _pyio.

Jeff Allen