Mailman 3 February 2021 - Python-Dev

bpo-42819 PR review request
by Dustin Rodrigues Feb. 9, 2021

Feb. 9, 2021

Hello, I submitted https://bugs.python.org/issue42819 a little over a month ago with an accompanying PR: https://github.com/python/cpython/pull/24108. Would it be possible to get feedback on it? This addresses a bug that occurs when Python is compiled with the most recent release of GNU Readline and affects both Linux and macOS. Thanks, Dustin

1 0

Change windows installation program name
by Barry Scott Feb. 8, 2021

Feb. 8, 2021

I raise https://bugs.python.org/issue43156 that suggests that new users on windows would be less confused if the name of the installation program did not seem to imply it was the python program. In a nutshell the proposal is to change from this: python-3.10.0a5.exe python-3.10.0a5-amd64.exe to this (thanks to Matthew Barnett for the improved names: python-3.10.0a5-win32-setup.exe python-3.10.0a5-win64-setup.exe Barry

3 2

Comments on PEP 558
by Mark Shannon Feb. 8, 2021

Feb. 8, 2021

Hi Nick, Regarding `f_locals` PEP 558 states: """ Instead of being a direct reference to the internal dynamic snapshot used to populate the independent snapshots returned by locals(), frame.f_locals will be updated to instead return a dedicated proxy type (implemented as a private subclass of the existing types.MappingProxyType) that has two internal attributes not exposed as part of the Python runtime API: * mapping: an implicitly updated snapshot of the function local variables … [View More]

3 13

What's up with assignment expression and tuples?
by Paul Sokolovsky Feb. 8, 2021

Feb. 8, 2021

Hello, Everyone knows how hard to find a compelling usecase for the assignment expression operator (":=", colloquially "walrus operator"). https://www.python.org/dev/peps/pep-0572/ examples never felt compelling and we all remember the split it caused. I finally found a usecase where *not* using assignment expression is *much* worse than using it. I'm working on SSA (Static Single Assignment, https://en.wikipedia.org/wiki/Static_single_assignment_form) conversion of Python programs, where … [View More]

5 10

Dusting off PEP 533?
by Paul Bryan Feb. 8, 2021

Feb. 8, 2021

I've now experienced an issue I believe that PEP 533 was intended to address: When an asynchronous context manager is created within an asynchronous generator, if the generator is not iterated fully, the context manager will not exit until the event loop cancels the task by raising a CancelledError, long after the context manager is assumed to be out of scope. Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that … [View More]

4 4

Concerns about PEP 634
by Mark Shannon Feb. 7, 2021

Feb. 7, 2021

Hi, Since a decision on PEP 634 is imminent, I'd like to reiterate some concerns that I voiced last year. I am worried about the semantics and implementation of PEP 634. I don't want to comment on the merits of pattern matching in general, or the proposed syntax in PEP 634 (or PEP 640 or PEP 642). Semantics --------- 1. PEP 634 eschews the object model, in favour of adhoc instance checks, length checks and attribute accesses. This is in contrast to almost all of the the rest of the … [View More]language, which uses special (dunder) methods: All operators, subscripting, attribute lookup, iteration, calls, tests, object creation, conversions, and the with statement AFAICT, no justification is given for this. Why can't pattern matching use the object model? PEP 343 (the "with" statement) added the __enter__ and __exit__ methods to the object model, and that works very well. 2. PEP 634 deliberately introduces a large area of undefined behaviour into Python. https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavi… Python has, in general, been strict about not having undefined behaviour. Having defined semantics means that you can reason about programs, even non-idiomatic ones. [This is not unique to Python, it is really only C and C++ that have areas of undefined behaviour] I can see no good reason for adding undefined behaviour. It doesn't help anyone. The lack of precise semantics makes programs harder to understand, and it makes the language harder to implement. If the semantics aren't specified, then the implementation becomes the specification. This bakes bugs into the language and makes it harder to maintain, as bug-for-bug compatibility must be maintained. 3. Performance PEP 634 says that each pattern must be checked in turn. That means that multiple redundant checks must be performed on (for example) a sequence if there are several mapping patterns. This is unnecessarily slow. Implementation -------------- My main concern with the implementation is that it does too much work into the interpreter. Much of that work can and should be done in the compiler. For example, deep in the implementation of the MATCH_CLASS instruction is the following comment: https://github.com/brandtbucher/cpython/blob/patma/Python/ceval.c#L981 Such complex control flow should be handled during compilation, rather than in the interpreter. Giant instructions like MATCH_CLASS are likely to have odd corner cases, and may well have a negative impact on the performance of the rest of the language. It is a lot easier to reason about a sequence of simple bytecodes, than one giant one with context-dependent behaviour. We have spent quite a lot of effort over the last few years streamlining the interpreter. Adding these extremely complex instructions would be a big backward step. Cheers, Mark. [View Less]

12 29

PEP 637 - Support for indexing with keyword arguments: request for feedback for SC submission
by Stefano Borini Feb. 6, 2021

Feb. 6, 2021

Hi all, I would like to request feedback by python-dev on the current implementation of PEP 637 - Support for indexing with keyword arguments. https://www.python.org/dev/peps/pep-0637/ The PEP is ready for SC submission and it has a prototype implementation ready, available here (note, not reviewed, but apparently fully functional) https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-im… (note: not sure if there's a preference for the link to be to the diff or to the … [View More]

8 14

PR review request: shutil permission errors
by Winson Luk Feb. 6, 2021

Feb. 6, 2021

Hi folks, I've had an open PR for a few weeks that still needs a review: https://github.com/python/cpython/pull/24001 This PR fixes a shutil.move() bug when permission to a directory is denied. Thanks, Winson

1 1

Summary of Python tracker Issues
by Python tracker Feb. 5, 2021

Feb. 5, 2021

ACTIVITY SUMMARY (2021-01-29 - 2021-02-05) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7379 (-209) closed 47495 (+284) total 54874 (+75) Open issues with patches: 2926 Issues opened (55) ================== #16202: sys.path[0] security issues https://bugs.python.org/issue16202 reopened by christian.heimes #24275: lookdict_* give up too soon … [View More]

1 0

PEP 624: Remove Py_UNICODE encoder APIs
by Inada Naoki Feb. 4, 2021

Feb. 4, 2021

Hi, folks. Since the previous discussion was suspended without consensus, I wrote a new PEP for it. (Thank you Victor for reviewing it!) This PEP looks very similar to PEP 623 "Remove wstr from Unicode", but for encoder APIs, not for Unicode object APIs. URL (not available yet): https://www.python.org/dev/peps/pep-0624/ --- PEP: 624 Title: Remove Py_UNICODE encoder APIs Author: Inada Naoki <songofacandy(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created:… [View More] 06-Jul-2020 Python-Version: 3.11 Abstract ======== This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in Python 3.11: * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` .. note:: `PEP 623 <https://www.python.org/dev/peps/pep-0623/>`_ propose to remove Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP is not relating to Unicode object. These PEPs are split because they have different motivation and need different discussion. Motivation ========== In general, reducing the number of APIs that have been deprecated for a long time and have few users is a good idea for not only it improves the maintainability of CPython, but it also helps API users and other Python implementations. Rationale ========= Deprecated since Python 3.3 --------------------------- ``Py_UNICODE`` and APIs using it are deprecated since Python 3.3. Inefficient ----------- All of these APIs are implemented using ``PyUnicode_FromWideChar``. So these APIs are inefficient when user want to encode Unicode object. Not used widely --------------- When searching from top 4000 PyPI packages [1]_, only pyodbc use these APIs. * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` pyodbc uses these APIs to encode Unicode object into bytes object. So it is easy to fix it. [2]_ Alternative APIs ================ There are alternative APIs to accept ``PyObject *unicode`` instead of ``Py_UNICODE *``. Users can migrate to them. ========================================= ========================================== Deprecated API Alternative APIs ========================================= ========================================== ``PyUnicode_Encode()`` ``PyUnicode_AsEncodedString()`` ``PyUnicode_EncodeASCII()`` ``PyUnicode_AsASCIIString()`` \(1) ``PyUnicode_EncodeLatin1()`` ``PyUnicode_AsLatin1String()`` \(1) ``PyUnicode_EncodeUTF7()`` \(2) ``PyUnicode_EncodeUTF8()`` ``PyUnicode_AsUTF8String()`` \(1) ``PyUnicode_EncodeUTF16()`` ``PyUnicode_AsUTF16String()`` \(3) ``PyUnicode_EncodeUTF32()`` ``PyUnicode_AsUTF32String()`` \(3) ``PyUnicode_EncodeUnicodeEscape()`` ``PyUnicode_AsUnicodeEscapeString()`` ``PyUnicode_EncodeRawUnicodeEscape()`` ``PyUnicode_AsRawUnicodeEscapeString()`` ``PyUnicode_EncodeCharmap()`` ``PyUnicode_AsCharmapString()`` \(1) ``PyUnicode_TranslateCharmap()`` ``PyUnicode_Translate()`` ``PyUnicode_EncodeDecimal()`` \(4) ``PyUnicode_TransformDecimalToASCII()`` \(4) ========================================= ========================================== Notes: (1) ``const char *errors`` parameter is missing. (2) There is no public alternative API. But user can use generic ``PyUnicode_AsEncodedString()`` instead. (3) ``const char *errors, int byteorder`` parameters are missing. (4) There is no direct replacement. But ``Py_UNICODE_TODECIMAL`` can be used instead. CPython uses ``_PyUnicode_TransformDecimalAndSpaceToASCII`` for converting from Unicode to numbers instead. Plan ==== Python 3.9 ---------- Add ``Py_DEPRECATED(3.3)`` to following APIs. This change is committed already [3]_. All other APIs have been marked ``Py_DEPRECATED(3.3)`` already. * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()``. Document all APIs as "will be removed in version 3.11". Python 3.11 ----------- These APIs are removed. * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` Alternative ideas ================= Instead of just removing deprecated APIs, we may be able to use thier names with different signature. Make some private APIs public ------------------------------ ``PyUnicode_EncodeUTF7()`` doesn't have public alternative APIs. Some APIs have alternative public APIs. But they are missing ``const char *errors`` or ``int byteorder`` parameters. We can rename some private APIs and make them public to cover missing APIs and parameters. ============================= ================================ Rename to Rename from ============================= ================================ ``PyUnicode_EncodeASCII()`` ``_PyUnicode_AsASCIIString()`` ``PyUnicode_EncodeLatin1()`` ``_PyUnicode_AsLatin1String()`` ``PyUnicode_EncodeUTF7()`` ``_PyUnicode_EncodeUTF7()`` ``PyUnicode_EncodeUTF8()`` ``_PyUnicode_AsUTF8String()`` ``PyUnicode_EncodeUTF16()`` ``_PyUnicode_EncodeUTF16()`` ``PyUnicode_EncodeUTF32()`` ``_PyUnicode_EncodeUTF32()`` ============================= ================================ Pros: * We have more consistent API set. Cons: * We have more public APIs to maintain. * Existing public APIs are enough for most use cases, and ``PyUnicode_AsEncodedString()`` can be used in other cases. Replace ``Py_UNICODE*`` with ``Py_UCS4*`` ----------------------------------------- We can replace ``Py_UNICODE`` (typedef of ``wchar_t``) with ``Py_UCS4``. Since builtin codecs support UCS-4, we don't need to convert ``Py_UCS4*`` string to Unicode object. Pros: * We have more consistent API set. * User can encode UCS-4 string in C without creating Unicode object. Cons: * We have more public APIs to maintain. * Applications which uses UTF-8 or UTF-32 can not use these APIs anyway. * Other Python implementations may not have builtin codec for UCS-4. * If we change the Unicode internal representation to UTF-8, we need to keep UCS-4 support only for these APIs. Replace ``Py_UNICODE*`` with ``wchar_t*`` ----------------------------------------- We can replace ``Py_UNICODE`` to ``wchar_t``. Pros: * We have more consistent API set. * Backward compatible. Cons: * We have more public APIs to maintain. * They are inefficient on platforms ``wchar_t*`` is UTF-16. It is because built-in codecs supports only UCS-1, UCS-2, and UCS-4 input. Rejected ideas ============== Using runtime warning --------------------- These APIs doesn't release GIL for now. Emitting a warning from such APIs is not safe. See this example. .. code-block:: PyObject *u = PyList_GET_ITEM(list, i); // u is borrowed reference. PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u), PyUnicode_GET_SIZE(u), NULL); // Assumes u is still living reference. PyObject *t = PyTuple_Pack(2, u, b); Py_DECREF(b); return t; If we emit Python warning from ``PyUnicode_EncodeUTF8()``, warning filters and other threads may change the ``list`` and ``u`` can be a dangling reference after ``PyUnicode_EncodeUTF8()`` returned. Additionally, since we are not changing behavior but removing C APIs, runtime ``DeprecationWarning`` might not helpful for Python developers. We should warn to extension developers instead. Discussions =========== * `Plan to remove Py_UNICODE APis except PEP 623 <https://mail.python.org/archives/list/python-dev@python.org/thread/S7KW2U6I…>`_ * `bpo-41123: Remove Py_UNICODE APIs except PEP 623: <https://bugs.python.org/issue41123>`_ References ========== .. [1] Source package list chosen from top 4000 PyPI packages. (https://github.com/methane/notes/blob/master/2020/wchar-cache/package_list.…) .. [2] pyodbc -- Don't use PyUnicode_Encode API #792 (https://github.com/mkleehammer/pyodbc/pull/792) .. [3] Uncomment Py_DEPRECATED for Py_UNICODE APIs (GH-21318) (https://github.com/python/cpython/commit/9c3840870814493fed62e140cfa43c2883…) Copyright ========= This document has been placed in the public domain. -- Inada Naoki <songofacandy(a)gmail.com> [View Less]

9 27