Mailman 3 January 2021 - Python-Dev

Dusting off PEP 533?
by Paul Bryan 08 Feb '21

08 Feb '21

I've now experienced an issue I believe that PEP 533 was intended to address: When an asynchronous context manager is created within an asynchronous generator, if the generator is not iterated fully, the context manager will not exit until the event loop cancels the task by raising a CancelledError, long after the context manager is assumed to be out of scope. Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary. Any appetite for putting PEP 533 back on the table to address this kind of issue?

4 4

PR review request: shutil permission errors
by Winson Luk 06 Feb '21

06 Feb '21

Hi folks, I've had an open PR for a few weeks that still needs a review: https://github.com/python/cpython/pull/24001 This PR fixes a shutil.move() bug when permission to a directory is denied. Thanks, Winson

1 1

PEP 624: Remove Py_UNICODE encoder APIs
by Inada Naoki 04 Feb '21

04 Feb '21

Hi, folks. Since the previous discussion was suspended without consensus, I wrote a new PEP for it. (Thank you Victor for reviewing it!) This PEP looks very similar to PEP 623 "Remove wstr from Unicode", but for encoder APIs, not for Unicode object APIs. URL (not available yet): https://www.python.org/dev/peps/pep-0624/ --- PEP: 624 Title: Remove Py_UNICODE encoder APIs Author: Inada Naoki <songofacandy(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 06-Jul-2020 Python-Version: 3.11 Abstract ======== This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in Python 3.11: * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` .. note:: `PEP 623 <https://www.python.org/dev/peps/pep-0623/>`_ propose to remove Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP is not relating to Unicode object. These PEPs are split because they have different motivation and need different discussion. Motivation ========== In general, reducing the number of APIs that have been deprecated for a long time and have few users is a good idea for not only it improves the maintainability of CPython, but it also helps API users and other Python implementations. Rationale ========= Deprecated since Python 3.3 --------------------------- ``Py_UNICODE`` and APIs using it are deprecated since Python 3.3. Inefficient ----------- All of these APIs are implemented using ``PyUnicode_FromWideChar``. So these APIs are inefficient when user want to encode Unicode object. Not used widely --------------- When searching from top 4000 PyPI packages [1]_, only pyodbc use these APIs. * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` pyodbc uses these APIs to encode Unicode object into bytes object. So it is easy to fix it. [2]_ Alternative APIs ================ There are alternative APIs to accept ``PyObject *unicode`` instead of ``Py_UNICODE *``. Users can migrate to them. ========================================= ========================================== Deprecated API Alternative APIs ========================================= ========================================== ``PyUnicode_Encode()`` ``PyUnicode_AsEncodedString()`` ``PyUnicode_EncodeASCII()`` ``PyUnicode_AsASCIIString()`` \(1) ``PyUnicode_EncodeLatin1()`` ``PyUnicode_AsLatin1String()`` \(1) ``PyUnicode_EncodeUTF7()`` \(2) ``PyUnicode_EncodeUTF8()`` ``PyUnicode_AsUTF8String()`` \(1) ``PyUnicode_EncodeUTF16()`` ``PyUnicode_AsUTF16String()`` \(3) ``PyUnicode_EncodeUTF32()`` ``PyUnicode_AsUTF32String()`` \(3) ``PyUnicode_EncodeUnicodeEscape()`` ``PyUnicode_AsUnicodeEscapeString()`` ``PyUnicode_EncodeRawUnicodeEscape()`` ``PyUnicode_AsRawUnicodeEscapeString()`` ``PyUnicode_EncodeCharmap()`` ``PyUnicode_AsCharmapString()`` \(1) ``PyUnicode_TranslateCharmap()`` ``PyUnicode_Translate()`` ``PyUnicode_EncodeDecimal()`` \(4) ``PyUnicode_TransformDecimalToASCII()`` \(4) ========================================= ========================================== Notes: (1) ``const char *errors`` parameter is missing. (2) There is no public alternative API. But user can use generic ``PyUnicode_AsEncodedString()`` instead. (3) ``const char *errors, int byteorder`` parameters are missing. (4) There is no direct replacement. But ``Py_UNICODE_TODECIMAL`` can be used instead. CPython uses ``_PyUnicode_TransformDecimalAndSpaceToASCII`` for converting from Unicode to numbers instead. Plan ==== Python 3.9 ---------- Add ``Py_DEPRECATED(3.3)`` to following APIs. This change is committed already [3]_. All other APIs have been marked ``Py_DEPRECATED(3.3)`` already. * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()``. Document all APIs as "will be removed in version 3.11". Python 3.11 ----------- These APIs are removed. * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` Alternative ideas ================= Instead of just removing deprecated APIs, we may be able to use thier names with different signature. Make some private APIs public ------------------------------ ``PyUnicode_EncodeUTF7()`` doesn't have public alternative APIs. Some APIs have alternative public APIs. But they are missing ``const char *errors`` or ``int byteorder`` parameters. We can rename some private APIs and make them public to cover missing APIs and parameters. ============================= ================================ Rename to Rename from ============================= ================================ ``PyUnicode_EncodeASCII()`` ``_PyUnicode_AsASCIIString()`` ``PyUnicode_EncodeLatin1()`` ``_PyUnicode_AsLatin1String()`` ``PyUnicode_EncodeUTF7()`` ``_PyUnicode_EncodeUTF7()`` ``PyUnicode_EncodeUTF8()`` ``_PyUnicode_AsUTF8String()`` ``PyUnicode_EncodeUTF16()`` ``_PyUnicode_EncodeUTF16()`` ``PyUnicode_EncodeUTF32()`` ``_PyUnicode_EncodeUTF32()`` ============================= ================================ Pros: * We have more consistent API set. Cons: * We have more public APIs to maintain. * Existing public APIs are enough for most use cases, and ``PyUnicode_AsEncodedString()`` can be used in other cases. Replace ``Py_UNICODE*`` with ``Py_UCS4*`` ----------------------------------------- We can replace ``Py_UNICODE`` (typedef of ``wchar_t``) with ``Py_UCS4``. Since builtin codecs support UCS-4, we don't need to convert ``Py_UCS4*`` string to Unicode object. Pros: * We have more consistent API set. * User can encode UCS-4 string in C without creating Unicode object. Cons: * We have more public APIs to maintain. * Applications which uses UTF-8 or UTF-32 can not use these APIs anyway. * Other Python implementations may not have builtin codec for UCS-4. * If we change the Unicode internal representation to UTF-8, we need to keep UCS-4 support only for these APIs. Replace ``Py_UNICODE*`` with ``wchar_t*`` ----------------------------------------- We can replace ``Py_UNICODE`` to ``wchar_t``. Pros: * We have more consistent API set. * Backward compatible. Cons: * We have more public APIs to maintain. * They are inefficient on platforms ``wchar_t*`` is UTF-16. It is because built-in codecs supports only UCS-1, UCS-2, and UCS-4 input. Rejected ideas ============== Using runtime warning --------------------- These APIs doesn't release GIL for now. Emitting a warning from such APIs is not safe. See this example. .. code-block:: PyObject *u = PyList_GET_ITEM(list, i); // u is borrowed reference. PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u), PyUnicode_GET_SIZE(u), NULL); // Assumes u is still living reference. PyObject *t = PyTuple_Pack(2, u, b); Py_DECREF(b); return t; If we emit Python warning from ``PyUnicode_EncodeUTF8()``, warning filters and other threads may change the ``list`` and ``u`` can be a dangling reference after ``PyUnicode_EncodeUTF8()`` returned. Additionally, since we are not changing behavior but removing C APIs, runtime ``DeprecationWarning`` might not helpful for Python developers. We should warn to extension developers instead. Discussions =========== * `Plan to remove Py_UNICODE APis except PEP 623 <https://mail.python.org/archives/list/python-dev@python.org/thread/S7KW2U6I…>`_ * `bpo-41123: Remove Py_UNICODE APIs except PEP 623: <https://bugs.python.org/issue41123>`_ References ========== .. [1] Source package list chosen from top 4000 PyPI packages. (https://github.com/methane/notes/blob/master/2020/wchar-cache/package_list.…) .. [2] pyodbc -- Don't use PyUnicode_Encode API #792 (https://github.com/mkleehammer/pyodbc/pull/792) .. [3] Uncomment Py_DEPRECATED for Py_UNICODE APIs (GH-21318) (https://github.com/python/cpython/commit/9c3840870814493fed62e140cfa43c2883…) Copyright ========= This document has been placed in the public domain. -- Inada Naoki <songofacandy(a)gmail.com>

9 27

Upcoming 3.7.10 and 3.6.13 Security Releases
by Ned Deily 03 Feb '21

03 Feb '21

We are planning to produce the next security-fix rollup releases for Python 3.7.x and 3.6.x on 2021-01-15. The most recent releases for these versions were on 2020-08-17. There has not been a lot of activity for either branch since then. Core developers: if you know of any additional security issues that should be addressed in these releases, please mark the relevant bpo issues as "release blocker" and, if possible, submit PRs for review prior to the end of 2021-01-14 AOE. Thanks! -- Ned Deily nad(a)python.org -- []

5 6

Why aren't we allowing the use of C11?
by Mark Shannon 01 Feb '21

01 Feb '21

Hi everyone, PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts). C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features. Is there a good reason not to start using C11 now? Cheers, Mark.

14 17

More comments on PEP 558
by Mark Shannon 30 Jan '21

30 Jan '21

Hi Nick, A couple more issues with PEP 558, as I see it. Lack of specification --------------------- There is no specification section in PEP 558. Much of PEP 558 describes the differences between the current behavior and what PEP 558 proposes. These differences are often intermingled with a discussion of the flaws with the current behavior, which makes it difficult to understand what the proposed behavior is. I could not implement PEP 558 from the PEP alone. Cycles and the cost of maintaining the f_locals cache ----------------------------------------------------- PEP 558 proposes that f_locals acts as a proxy mapping to the locals in a frame *and* that the frame caches the proxy. This obviously creates a cycle, which needs to be broken by the cycle GC. Although the cycle only exists if f_locals is actually used, there is still some overhead for *every* call and return, as the f_locals field needs to be initialized on call and checked for non-NULL on return. Cheers, Mark.

2 1

What is the pyclbr public API?
by Terry Reedy 30 Jan '21

30 Jan '21

pyclbr is the stdlib module browser (once just class browser, hence the name). The doc https://docs.python.org/3/library/pyclbr.html#module-pyclbr, which I revised in 2017, documents readline and readline_ex as the public call interface. The functions return a hierarchical structure that includes Function and Class instances. (Both subclass pyclbr._Object.) The doc I revised already omitted signatures for these classes and just listed the attributes of instances. Just before that listing, I added "Users are not expected to create instances of these classes." https://bugs.python.org/issue38307 and https://github.com/python/cpython/pull/24348 propose to add an end_lineno attribute. Since pyclbr is now, since last November, based on the ast tree and since the relevant ast nodes have an end_line attribute, the proposal amounts to copying that attribute, along with others, instead of ignoring it. The patch proposes to add 'end_lineno' after existing start 'lineno' in the _Object, Function, and Class signatures. I prefer doing this, rather than adding the new parameter at the end of the list, because a) being the the logical place would make the code easier to read, and b) new names as the end of the signature, follow optional arguments, would have to be optional, whereas My question is whether the omission of signatures and my added sentence sufficiently defines the signatures of these classes (in particular, the argument order) as private to approve the patch as is? -- Terry Jan Reedy

2 7

Improve CPython tracing performance
by Fabio Zadrozny 30 Jan '21

30 Jan '21

Hi all, Right now, when a debugger is active, the number of local variables can affect the tracing speed quite a lot. For instance, having tracing setup in a program such as the one below takes 4.64 seconds to run, yet, changing all the variables to have the same name -- i.e.: change all assignments to `a = 1` (such that there's only a single variable in the namespace), it takes 1.47 seconds (in my machine)... the higher the number of variables, the slower the tracing becomes. ``` import time t = time.time() def call(): a = 1 b = 1 c = 1 d = 1 e = 1 f = 1 def noop(frame, event, arg): return noop import sys sys.settrace(noop) for i in range(1_000_000): call() print('%.2fs' % (time.time() - t,)) ``` This happens because `PyFrame_FastToLocalsWithError` and `PyFrame_LocalsToFast` are called inside the `call_trampoline` ( https://github.com/python/cpython/blob/master/Python/sysmodule.c#L946). So, I'd like to simply remove those calls. Debuggers can call `PyFrame_LocalsToFast` when needed -- otherwise mutating non-current frames doesn't work anyways. As a note, pydevd already has such a call: https://github.com/fabioz/PyDev.Debugger/blob/0d4d210f01a1c0a8647178b2e665b… and PyPy also has a counterpart. As for `PyFrame_FastToLocalsWithError`, I don't really see any reason to call it at all. i.e.: something as the code below prints the `a` variable from the `main()` frame regardless of that and I checked all pydevd tests and nothing seems to be affected (it seems that accessing f_locals already does this: https://github.com/python/cpython/blob/cb9879b948a19c9434316f8ab6aba9c4601a…, so, I don't see much reason to call it at all). ``` def call(): import sys frame = sys._getframe() print(frame.f_back.f_locals) def main(): a = 1 call() if __name__ == '__main__': main() ``` Does anyone see any issue with this? If it's non controversial, is a PEP needed or just an issue to track it would be enough to remove those 2 lines? Thanks, Fabio

3 9

Summary of Python tracker Issues
by Python tracker 29 Jan '21

29 Jan '21

ACTIVITY SUMMARY (2021-01-22 - 2021-01-29) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7588 (+10) closed 47211 (+50) total 54799 (+60) Open issues with patches: 3032 Issues opened (40) ================== #33387: Simplify bytecodes for try-finally, try-except and with blocks https://bugs.python.org/issue33387 reopened by iritkatriel #36675: Doctest directives and comments missing from code samples https://bugs.python.org/issue36675 reopened by mdk #43006: Changed behaviour of inspect.signature() in Python 3.10 https://bugs.python.org/issue43006 opened by Zac Hatfield-Dodds #43007: set_wakeup_fd() only works in main thread https://bugs.python.org/issue43007 opened by MeneMeneTekel #43009: Port curses capi pointer array to a struct https://bugs.python.org/issue43009 opened by shihai1991 #43010: @functools.wraps and abc.abstractmethod interoperability https://bugs.python.org/issue43010 opened by erezinman #43012: Remove pathlib accessors https://bugs.python.org/issue43012 opened by barneygale #43013: IDLE: update code, mostly by cleanups of 2.x or 2to3 artifact https://bugs.python.org/issue43013 opened by terry.reedy #43015: Add str.replaceall? https://bugs.python.org/issue43015 opened by Nathaniel Manista #43016: Improve tests for curses https://bugs.python.org/issue43016 opened by serhiy.storchaka #43017: Improve error message in the parser when using un-parenthesise https://bugs.python.org/issue43017 opened by pablogsal #43019: wait_for(to_thread)) does not work as expected. Extra document https://bugs.python.org/issue43019 opened by synchronizing #43022: Unable to dynamically load functions from python3.dll https://bugs.python.org/issue43022 opened by paul.moore #43024: improve signature (in help, etc) for functions taking sentinel https://bugs.python.org/issue43024 opened by iritkatriel #43026: Missing words renders meaning unclear in fcntl.html https://bugs.python.org/issue43026 opened by EzraBC #43027: Calling _PyBytes_Resize() on 1-byte bytes may raise error https://bugs.python.org/issue43027 opened by malin #43029: unittest: Add assertUniqeIn https://bugs.python.org/issue43029 opened by rousseldenis #43030: signed/unsigned mismatch in Py_UNICODE_ISSPACE macro https://bugs.python.org/issue43030 opened by doko #43034: Python tutorial misleads users about floor division behavior https://bugs.python.org/issue43034 opened by jessevsilverman #43035: FileNotFoundError in distutils\file_util.py copy_tree https://bugs.python.org/issue43035 opened by knzivid #43036: TOS-behaviour documentation is inconsistent https://bugs.python.org/issue43036 opened by xmorel #43039: tempfile.TemporaryDirectory() name string is incorrect https://bugs.python.org/issue43039 opened by eosborne #43041: copying WeakValueDictionary is not iteration safe https://bugs.python.org/issue43041 opened by djromberg #43042: tutorial ambiguous about creation of local symbol table for re https://bugs.python.org/issue43042 opened by jessevsilverman #43043: Python tutorial could make rules for default argument definiti https://bugs.python.org/issue43043 opened by jessevsilverman #43045: Instructions for installing pip on Ubuntu/WSL2 is incomplete https://bugs.python.org/issue43045 opened by lennart.borgman #43046: argparse: capturing actions https://bugs.python.org/issue43046 opened by monkeyman79 #43047: logging.config formatters documentation is out of sync with co https://bugs.python.org/issue43047 opened by iwienand #43048: Printing RecursionError results in RecursionError https://bugs.python.org/issue43048 opened by vlad2 #43049: Use io.IncrementalNewlineDecoder for doctest newline conversio https://bugs.python.org/issue43049 opened by pdonis #43050: threading timer memory leak https://bugs.python.org/issue43050 opened by fengjiang #43052: _dyld_shared_cache_contains_path needs SYSTEM_VERSION_COMPAT=0 https://bugs.python.org/issue43052 opened by isuruf #43053: Speed up math.isqrt, again https://bugs.python.org/issue43053 opened by juraj.sukop #43054: What does the existence of a struct in a header file imply abo https://bugs.python.org/issue43054 opened by Mark.Shannon #43056: Use of dicts in sections 4.2 and 4.7 of Python tutorial a bit https://bugs.python.org/issue43056 opened by jessevsilverman #43058: setting a logging Handler name https://bugs.python.org/issue43058 opened by bcohen #43059: sqlite3: Externally developed? https://bugs.python.org/issue43059 opened by user1347091 #43060: Convert _decimal C API from pointer array to struct https://bugs.python.org/issue43060 opened by erlendaasland #43062: Non-integer values in collections.Counter https://bugs.python.org/issue43062 opened by HuangFuSL #43063: zipfile.Path / importlib.resources raises KeyError if a file w https://bugs.python.org/issue43063 opened by The Compiler Most recent 15 issues with no replies (15) ========================================== #43062: Non-integer values in collections.Counter https://bugs.python.org/issue43062 #43060: Convert _decimal C API from pointer array to struct https://bugs.python.org/issue43060 #43058: setting a logging Handler name https://bugs.python.org/issue43058 #43056: Use of dicts in sections 4.2 and 4.7 of Python tutorial a bit https://bugs.python.org/issue43056 #43052: _dyld_shared_cache_contains_path needs SYSTEM_VERSION_COMPAT=0 https://bugs.python.org/issue43052 #43049: Use io.IncrementalNewlineDecoder for doctest newline conversio https://bugs.python.org/issue43049 #43047: logging.config formatters documentation is out of sync with co https://bugs.python.org/issue43047 #43045: Instructions for installing pip on Ubuntu/WSL2 is incomplete https://bugs.python.org/issue43045 #43043: Python tutorial could make rules for default argument definiti https://bugs.python.org/issue43043 #43042: tutorial ambiguous about creation of local symbol table for re https://bugs.python.org/issue43042 #43036: TOS-behaviour documentation is inconsistent https://bugs.python.org/issue43036 #43027: Calling _PyBytes_Resize() on 1-byte bytes may raise error https://bugs.python.org/issue43027 #43026: Missing words renders meaning unclear in fcntl.html https://bugs.python.org/issue43026 #43019: wait_for(to_thread)) does not work as expected. Extra document https://bugs.python.org/issue43019 #43016: Improve tests for curses https://bugs.python.org/issue43016 Most recent 15 issues waiting for review (15) ============================================= #43059: sqlite3: Externally developed? https://bugs.python.org/issue43059 #43049: Use io.IncrementalNewlineDecoder for doctest newline conversio https://bugs.python.org/issue43049 #43047: logging.config formatters documentation is out of sync with co https://bugs.python.org/issue43047 #43046: argparse: capturing actions https://bugs.python.org/issue43046 #43030: signed/unsigned mismatch in Py_UNICODE_ISSPACE macro https://bugs.python.org/issue43030 #43029: unittest: Add assertUniqeIn https://bugs.python.org/issue43029 #43024: improve signature (in help, etc) for functions taking sentinel https://bugs.python.org/issue43024 #43017: Improve error message in the parser when using un-parenthesise https://bugs.python.org/issue43017 #43016: Improve tests for curses https://bugs.python.org/issue43016 #43013: IDLE: update code, mostly by cleanups of 2.x or 2to3 artifact https://bugs.python.org/issue43013 #43009: Port curses capi pointer array to a struct https://bugs.python.org/issue43009 #42999: `pathlib.Path.link_to()` documentation is misleading https://bugs.python.org/issue42999 #42997: Improve error message for missing : before suites https://bugs.python.org/issue42997 #42995: Add PurePath.with_suffix_appended() https://bugs.python.org/issue42995 #42994: Missing MIME types for opus, AAC, 3gpp and 3gpp2 https://bugs.python.org/issue42994 Top 10 most discussed issues (10) ================================= #42973: argparse: mixing optional and positional arguments... not agai https://bugs.python.org/issue42973 17 msgs #3353: make built-in tokenizer available via Python C API https://bugs.python.org/issue3353 15 msgs #43013: IDLE: update code, mostly by cleanups of 2.x or 2to3 artifact https://bugs.python.org/issue43013 13 msgs #43022: Unable to dynamically load functions from python3.dll https://bugs.python.org/issue43022 12 msgs #42967: [security] urllib.parse.parse_qsl(): Web cache poisoning - `; https://bugs.python.org/issue42967 11 msgs #43046: argparse: capturing actions https://bugs.python.org/issue43046 7 msgs #43053: Speed up math.isqrt, again https://bugs.python.org/issue43053 7 msgs #43006: Changed behaviour of inspect.signature() in Python 3.10 https://bugs.python.org/issue43006 6 msgs #43059: sqlite3: Externally developed? https://bugs.python.org/issue43059 6 msgs #38307: Add .end_lineno attribute to pyclbr _Objects https://bugs.python.org/issue38307 5 msgs Issues closed (49) ================== #2636: Adding a new regex module (compatible with re) https://bugs.python.org/issue2636 closed by vstinner #23544: IDLE hangs when selecting Stack View with debug active https://bugs.python.org/issue23544 closed by terry.reedy #27772: Refer to actual format string when creating “zero padding” https://bugs.python.org/issue27772 closed by serhiy.storchaka #29076: Mac installer shell updater script silently fails if default s https://bugs.python.org/issue29076 closed by ned.deily #33289: tkinter askcolor returning floats for r, g, b values instead o https://bugs.python.org/issue33289 closed by serhiy.storchaka #36086: Split IDLE into separate feature in Windows installer https://bugs.python.org/issue36086 closed by rhettinger #36379: nb_inplace_pow is always called with an invalid argument https://bugs.python.org/issue36379 closed by gregory.p.smith #38250: enum.Flag should be more set-like https://bugs.python.org/issue38250 closed by ethan.furman #40745: Typo in library/typing https://bugs.python.org/issue40745 closed by iritkatriel #41459: pickle.load raises SystemError on malformed input https://bugs.python.org/issue41459 closed by eric.smith #41798: [C API] Revisit usage of the PyCapsule C API with multi-phase https://bugs.python.org/issue41798 closed by shihai1991 #42300: Typo in translation to portuguese https://bugs.python.org/issue42300 closed by iritkatriel #42383: Pdb does not correclty restart the target if it changes the cu https://bugs.python.org/issue42383 closed by gvanrossum #42384: Inconsistent sys.path between python and pdb https://bugs.python.org/issue42384 closed by gvanrossum #42843: What min_sphinx for Python 3.10 https://bugs.python.org/issue42843 closed by mdk #42869: pydoc does not append .html to documentation https://bugs.python.org/issue42869 closed by mdk #42901: [Enum] move member creation to __set_name__ in order to suppor https://bugs.python.org/issue42901 closed by ethan.furman #42915: enum.Flag ~ bitwise negation is very slow and can't be defined https://bugs.python.org/issue42915 closed by ethan.furman #42948: bytearray.copy is undocumented https://bugs.python.org/issue42948 closed by wim.glenn #42955: Add sys.stdlib_module_names: list of stdlib module names (Pyth https://bugs.python.org/issue42955 closed by vstinner #42957: os.readlink produces wrong result on windows https://bugs.python.org/issue42957 closed by steve.dower #42977: Tkinter Optionmenu Too Narrow on Mac https://bugs.python.org/issue42977 closed by serhiy.storchaka #42979: _zoneinfo: zoneinfomodule_exec() doesn't check for PyDateTime_ https://bugs.python.org/issue42979 closed by vstinner #42985: AMD64 Arch Linux Asan 3.x fails: command timed out: 1200 secon https://bugs.python.org/issue42985 closed by vstinner #42992: Tkinter bbox coordinates incorrectly drawn https://bugs.python.org/issue42992 closed by terry.reedy #43002: Exception chaining accepts exception classes https://bugs.python.org/issue43002 closed by steven.daprano #43003: Parts of the API will be removed in Python 4.0 ? https://bugs.python.org/issue43003 closed by terry.reedy #43004: No type variables left in collections.abc.Callable https://bugs.python.org/issue43004 closed by tyler.yep #43005: Package Relative Imports - double dot doc example not working https://bugs.python.org/issue43005 closed by gvanrossum #43008: IDLE ignores sys.excepthook in normal, subprocess mode https://bugs.python.org/issue43008 closed by terry.reedy #43011: DeprecationWarnings in test_ctypes https://bugs.python.org/issue43011 closed by benjamin.peterson #43014: tokenize spends a lot of time in `re.compile(...)` https://bugs.python.org/issue43014 closed by BTaskaya #43018: unwanted label showing up in ttk.LabeledScale https://bugs.python.org/issue43018 closed by serhiy.storchaka #43020: str.lower method with "İ" character https://bugs.python.org/issue43020 closed by steven.daprano #43021: Unpacking tuple argument in combination with inline if stateme https://bugs.python.org/issue43021 closed by steven.daprano #43023: Remove a redundant check in _PyBytes_Resize() https://bugs.python.org/issue43023 closed by malin #43025: Use normal 'i' character to denote imaginary part of complex n https://bugs.python.org/issue43025 closed by eric.smith #43028: seeking past the end of a file unexpected behavior https://bugs.python.org/issue43028 closed by cotton.seed #43031: aarch64 RHEL7 LTO + PGO 3.7: "make" hangs when running test_as https://bugs.python.org/issue43031 closed by vstinner #43032: Size of pie chart in matplotlib (frame affects it) https://bugs.python.org/issue43032 closed by mark.dickinson #43033: Incorrect handling of PyObject_SetAttrString() in the _zoneinf https://bugs.python.org/issue43033 closed by serhiy.storchaka #43037: Importlib reload by module name (String) https://bugs.python.org/issue43037 closed by brett.cannon #43038: ensurepip: tries to use setup.py/setup.cfg https://bugs.python.org/issue43038 closed by ned.deily #43040: random.py randrange() is very slow if the range is a power of https://bugs.python.org/issue43040 closed by rhettinger #43044: Python 2.7 documentation links to 404 pages when the library w https://bugs.python.org/issue43044 closed by vstinner #43051: Pip Permissions Error overwriting pip directory https://bugs.python.org/issue43051 closed by steve.dower #43055: Inconsistent behaviour when using walrus operator with 'and'/' https://bugs.python.org/issue43055 closed by eric.smith #43057: Timezone 'Etc/GMT-5' functions as Timezone 'Etc/GMT+5' https://bugs.python.org/issue43057 closed by mark.dickinson #43061: subprocess: feature request: Get only the stdout of the last s https://bugs.python.org/issue43061 closed by r.david.murray

1 0

New sys.module_names attribute in Python 3.10: list of all stdlib modules
by Victor Stinner 29 Jan '21

29 Jan '21

Hi, I just added a new sys.module_names attribute, list (technically a frozenset) of all stdlib module names: https://bugs.python.org/issue42955 There are multiple use cases: * Group stdlib imports when reformatting a Python file, * Exclude stdlib imports when computing dependencies. * Exclude stdlib modules when listing extension modules on crash or fatal error, only list 3rd party extension (already implemented in master, see bpo-42923 ;-)). * Exclude stdlib modules when tracing the execution of a program using the trace module. * Detect typo and suggest a fix: ImportError("No module named maths. Did you mean 'math'?",) (test the nice friendly-traceback project!). Example: >>> 'asyncio' in sys.module_names True >>> 'numpy' in sys.module_names False >>> len(sys.module_names) 312 >>> type(sys.module_names) <class 'frozenset'> >>> sorted(sys.module_names)[:10] ['__future__', '_abc', '_aix_support', '_ast', '_asyncio', '_bisect', '_blake2', '_bootsubprocess', '_bz2', '_codecs'] >>> sorted(sys.module_names)[-10:] ['xml.dom', 'xml.etree', 'xml.parsers', 'xml.sax', 'xmlrpc', 'zipapp', 'zipfile', 'zipimport', 'zlib', 'zoneinfo'] The list is opinionated and defined by its documentation: A frozenset of strings containing the names of standard library modules. It is the same on all platforms. Modules which are not available on some platforms and modules disabled at Python build are also listed. All module kinds are listed: pure Python, built-in, frozen and extension modules. Test modules are excluded. For packages, only sub-packages are listed, not sub-modules. For example, ``concurrent`` package and ``concurrent.futures`` sub-package are listed, but not ``concurrent.futures.base`` sub-module. See also the :attr:`sys.builtin_module_names` list. The design (especially, the fact of having the same list on all platforms) comes from the use cases list above. For example, running isort should produce the same output on any platform, and not depend if the Python stdlib was splitted into multiple packages on Linux (which is done by most popular Linux distributions). The list is generated by the Tools/scripts/generate_module_names.py script: https://github.com/python/cpython/blob/master/Tools/scripts/generate_module… When you add a new module, you must run "make regen-module-names, otherwise a pre-commit check will fail on your PR ;-) The list of Windows extensions is currently hardcoded in the script (contributions are welcomed to discover them, since the list is short and evolves rarely, I didn't feel the need to spend time that on that). Currently (Python 3.10.0a4+), there are 312 names in sys.module_names, stored in Python/module_names.h: https://github.com/python/cpython/blob/master/Python/module_names.h It was decided to include "helper" modules like "_aix_support" which is used by sysconfig. But test modules like _testcapi are excluded to make the list shorter (it's rare to run the CPython test suite outside Python). There are 83 private modules, name starting with an underscore (exclude _abc but also __future__): >>> len([name for name in sys.module_names if not name.startswith('_')]) 229 This new attribute may help to define "what is the Python stdlib" ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

13 34