[New-bugs-announce] [issue43421] os.device_encoding(fd) should support any console fd in Windows

[Eryk Sun]

New submission from Eryk Sun <eryksun at gmail.com>:

In Windows, os.device_encoding() is hard coded to map file descriptor 0 and descriptors 1 and 2 respectively to the console's input and output code page if isatty(fd) is true. But isatty() is true for any character device, such as "NUL". Also any fd might be a console, by way of dup(), dup2() -- or open() with the device names "CON", "CONIN$", and "CONOUT$".

The correct device encoding of a console file is needed for use with os.read() and os.write(). It's also necessary for io.TextIOWrapper() if PYTHONLEGACYWINDOWSSTDIO is set.

_Py_device_encoding() in Python/fileutils.c should use _get_osfhandle() to get the OS handle, and, if it's a character-device file, determine the code page to return, if any, depending on whether it's an input or output console file. For example:

PyObject *
_Py_device_encoding(int fd)
{
#if defined(MS_WINDOWS)
    HANDLE handle;
    DWORD temp;
    UINT cp = 0;

    _Py_BEGIN_SUPPRESS_IPH
        handle = (HANDLE)_get_osfhandle(fd);
    _Py_END_SUPPRESS_IPH
    if (handle == INVALID_HANDLE_VALUE ||
            GetFileType(handle) != FILE_TYPE_CHAR)
        Py_RETURN_NONE;

    Py_BEGIN_ALLOW_THREADS
    /* GetConsoleMode requires a console handle. */
    if (!GetConsoleMode(handle, &temp)) {
        /* Assume access denied implies output. */
        if (GetLastError() == ERROR_ACCESS_DENIED)
            cp = GetConsoleOutputCP();
    } else {
        if (GetNumberOfConsoleInputEvents(handle, &temp)) {
            cp = GetConsoleCP();
        } else {
            cp = GetConsoleOutputCP();
        }
    }
    Py_END_ALLOW_THREADS

    if (cp == CP_UTF8) {
        return PyUnicode_FromString("UTF-8");
    } else if (cp != 0) {
        return PyUnicode_FromFormat("cp%u", (unsigned int)cp);
    } else {
        Py_RETURN_NONE;
    }
#else
    if (isatty(fd)) {
        return _Py_GetLocaleEncodingObject();
    } else {
        Py_RETURN_NONE;
    }
#endif  /* defined(MS_WINDOWS) */
}

----------
components: Extension Modules, IO, Interpreter Core, Windows
messages: 388201
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: os.device_encoding(fd) should support any console fd in Windows
type: behavior
versions: Python 3.10, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43421>
_______________________________________