[issue10841] binary stdio

[Glenn Linderman]

Glenn Linderman <v+python at g.nevcal.com> added the comment:

Found it.

The file browser doesn't tell what line number it is, but in _io/Fileio.c function fileio_init, there is code like

#ifdef O_BINARY
    flags |= O_BINARY;
#endif

#ifdef O_APPEND
    if (append)
        flags |= O_APPEND;
#endif

    if (fd >= 0) {
        if (check_fd(fd))
            goto error;
        self->fd = fd;
        self->closefd = closefd;
    }


Note that if O_BINARY is defined, it is set into the default flags for opening files by name.  But if "opening" a file by fd, the fd is copied, regardless of whether it has O_BINARY set or not.  The rest of the IO code no doubt assumes the file was opened in O_BINARY mode.  But that isn't true of MSC std* handles by default.

How -u masks or overcomes this problem is not obvious, as yet, but the root bug seems to be the assumption in the above code.  A setmode of O_BINARY should be done, probably #ifdef O_BINARY, when attaching a MS C fd to a Python IO stack.  Otherwise it is going to have \r\r\n problems, it would seem.

Alternately, in the location where the Python IO stacks are attached to std* handles, those specific std* handles should have the setmode done there... other handles, if opened by Python, likely already have it done.

Documentation for open should mention, in the description of the file parameter, that on Windows, it is important to only attach Python IO stack to O_BINARY files, or beware the consequences of two independent newline handling algorithms being applied to the data stream... or to document that setmode O_BINARY will be performed on the handles passed to open.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10841>
_______________________________________