detecting newline character

Thomas 'PointedEars' Lahn PointedEars at
Sat Apr 23 15:33:33 EDT 2011

Chris Rebert wrote:

> On Sat, Apr 23, 2011 at 11:09 AM, Daniel Geržo <danger at> wrote:
>> I need to detect the newline characters used in the file I am reading.
>> For this purpose I am using the following code:
>> def _read_lines(self):
>>     with contextlib.closing(, "rU")) as fobj:
>>     fobj.readlines()
>>     if isinstance(fobj.newlines, tuple):
>>         self.newline = fobj.newlines[0]
>>     else:
>>         self.newline = fobj.newlines
>> This works fine, if I call without encoding argument; I am
>> testing with an ASCII enghlish text file, and in such case the
>> fobj.newlines is correctly detected being as '\r\n'. However, when I call
>> with encoding='ascii' argument, the fobj.newlines is None
>> and I can't figure out why that is the case. Reading the PEP at
>> I don't see any reason why would
>> I end up with newlines being None after I call readlines().
>> Anyone has an idea?
> I would hypothesize that it's an interaction bug between universal
> newlines and
> […]
> I would speculate that the upshot of this is that ends
> up calling built-in open() with a nonsense `mode` of "rUb" or similar,
> resulting in strange behavior.
> If this explanation is correct, then there are 2 bugs:
> 1. Built-in open() should treat "b" and "U" as mutually exclusive and
> reject mode strings which involve both.
> 2. should either reject modes involving "U", or be fixed
> so that they work as expected.

You might be correct that it is a bug (already fixed in versions newer than 
2.5), since from my Python 2.6 reads as follows:

def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
    if encoding is not None:
        if 'U' in mode:
            # No automatic conversion of '\n' is done on reading and writing
            mode = mode.strip().replace('U', '')
            if mode[:1] not in set('rwa'):
                mode = 'r' + mode
        if 'b' not in mode:
            # Force opening of the file in binary mode
            mode = mode + 'b'
    file =, mode, buffering)
    if encoding is None:
        return file
    info = lookup(encoding)
    srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, 
    # Add attributes to simplify introspection
    srw.encoding = encoding
    return srw


More information about the Python-list mailing list