Mailman 3 May 2022 - Python-Dev

PEP 689 – Semi-stable C API tier
by Petr Viktorin 01 Jun '22

01 Jun '22

Hello, Victor and others are organizing the C-API to make it clearer what's public and what's private (see Victor's file-based stats: https://pythoncapi.readthedocs.io/stats.html) It became clear that we need a tier between public and private C-API. PEP 689 proposes such API, with an opt-in macro and transition period. Please discuss. And if you can think of a better name, that would be great :) The PEP is at: https://peps.python.org/pep-0689/ Thread where this started: https://mail.python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6… Implementation notes: https://github.com/python/cpython/issues/91744 Here's the current version for easy quoting: ---------- PEP: 689 Title: Semi-stable C API tier Author: Petr Viktorin <encukou(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 523 Created: 22-Apr-2022 Python-Version: 3.11 Abstract ======== Some functions and types of the C-API are designated *semi-stable*, meaning that they will not change in patch (bugfix/security) releases, but may change between minor releases (e.g. between 3.11 and 3.12) without deprecation warnings. Motivation & Rationale ====================== The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__: - Limited API, with high compatibility expectations - Public API, which follows the :pep:`backwards compatibility policy <387>`, and requires deprecation warnings before changes - Private (internal) API, which can change at any time. We need a tier between Public and Private to accommodate tools like advanced debuggers and JIT compilers, which need access to CPython internals but assume the C-API they use does not change in patch releases. Setting Stability Expectations ------------------------------ Currently, there are no guarantees for the internal API -- that is, anything that requires ``Py_BUILD_CORE`` or is named with a leading underscore. This API can change without warning at any time, and code that uses it is pinned to a specific build of Python. However, in practice, even the internal API usually happens to be stable in patch releases: - Some CPython core developers take this as an an unwritten rule. - Patch releases only contain bugfixes, which are unlikely to change the API. Semi-stable API will make the stability expectations more explicit. It will also hopefully encourage existing users of the private API to reach out to python-dev, so we can expose, standardize and test an API for some of their use cases. Reserving underscores for Private API ------------------------------------- :pep:`523` introduced functions for use by debuggers and JIT compilers, which are stable only across minor releases. The functions names have leading underscores to suggest their limited stability. However, leading underscores usually mark *fully private* API. CPython developers familiar with the “underscore means internal” convention are unlikely to check if underscored functions they are changing are documented and used outside CPython itself. This proposal brings us a bit closer to reserving underscores only for truly private, unstable, hands-off API. Warning about API that is changed often --------------------------------------- The ``PyCode_New()`` family is an example of functions that are documented as unstable, and also often change in practice. Moving it to the semi-stable tier will make its status obvious even to people who don't read the docs carefully enough, and will make it hard to use accidentally. Changes during the Beta period ------------------------------ Since the API itself can change continuously up until Beta 1 (feature freeze) of a minor version, major users of this API are unlikely to test Alpha releases and provide feedback. It is very difficult to determine what needs to be exposed as semi-stable. Additions to the semi-stable tier will count as *stabilization*, and will be allowed up to Release Candidate 1. Specification ============= Several functions and types (“APIs”) will be moved to a new *semi-stable* tier. They will be expected to stay stable across patch releases, but may change or be removed without warning in minor releases (3.x.0), including Alpha and Beta releases of 3.x.0. When they change significantly, code that uses them should no longer compile (e.g. arguments should be added/removed, or a function should be renamed, but the semantic meaning of an argument should not change). Their definitions will be moved to a new directory, ``Include/semistable/``, and will be included from ``Python.h``. From Python 3.12 on, these APIs will only be usable when the ``Py_USING_SEMI_STABLE_API`` macro is defined. CPython will only define the macro for building CPython itself (``Py_BUILD_CORE``). To make transition to semi-stable API easier, in Python 3.11 the APIs will be available without ``Py_USING_SEMI_STABLE_API`` defined. In this case, using them will generate a deprecation warning on compilers that support ``Py_DEPRECATED``. A similar deprecation period will be used when making more APIs semi-stable in the future: - When moving from public API, the deprecation period should follow Python's backwards compatibility policy (currently, it should last at least two releases). - When moving from public API that is documented as unstable, the deprecation period can only last one release. - When moving from private API or adding new API, no deprecation period is necessary. Leading underscores will be removed from the names of the moved APIs. The old underscored name of a renamed API will be available (as an alias using ``#define``) at least until that API changes. The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be documented, and documentation of each semi-stable API will be updated. Adjustments during Beta periods ------------------------------- New APIs can be added to the semi-stable tier, and private APIs can be moved to it, up to the first release candidate of a new minor version. Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period. In the Beta period, no API may be moved to more private tier, e.g. what is public in Beta 1 must stay public until the final release. Initial semi-stable API ----------------------- The following API will initially be semi-stable. The set may be adjusted for 3.11. Code object constructors: - ``PyCode_New()`` - ``PyCode_NewWithPosOnlyArgs()`` Frame evaluation API (PEP 523): - ``_PyFrameEvalFunction`` - ``_PyInterpreterState_GetEvalFrameFunc()`` - ``_PyInterpreterState_SetEvalFrameFunc()`` - ``_PyEval_RequestCodeExtraIndex()`` - ``_PyCode_GetExtra()`` - ``_PyCode_SetExtra()`` - ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct) - ``_PyFrame_GetFrameObject`` - ``PyEval_EvalFrameDefault`` (new function that calls ``_PyEval_EvalFrameDefault``, but takes ``PyFrameObject`` rather than ``_PyInterpreterFrame``) (Leading underscores will be removed as mentioned above.) Backwards Compatibility ======================= The C API backwards compatibility story will be made clearer. How to Teach This ================= The changes affect advanced C programmers, who should consult the updated reference documentation, devguide and/or What's New document·. Reference Implementation ======================== https://github.com/python/cpython/issues/91744 Rejected Ideas ============== It might be good to add a similar tier in the Python (not C) API, e.g. for ``types.CodeType``. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP. Open Issues =========== - “Semi-stable” is not a perfect name. - The exact set of exposed API may change. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

13 24

Is it possible to view tokenizer output?
by Jack 31 May '22

31 May '22

Hi, I'm just getting into the CPython codebase just for fun, and I've just started messing around with the tokenizer and the grammar. I was wondering, is there a way to just print out the results of the tokenizer (as in just the stream of tokens it generates) in a human readable format? It would be really helpful for debugging. Hope the question's not too basic. Cheers.

5 9

Raw strings ending with a backslash
by Steven D'Aprano 30 May '22

30 May '22

Now that we have a new parser for CPython, can we fix the old gotcha that raw strings cannot end in a backslash? Its an FAQ and has come up again on the bug tracker. https://docs.python.org/3/faq/design.html#id26 https://github.com/python/cpython/issues/93314 -- Steve

9 13

spling1975@gmail.com
by seanpling05＠gmail.com 29 May '22

29 May '22

1 0

compiled python3.10 is unable to find _ssl
by Sandeep Gupta 29 May '22

29 May '22

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/shared/Builds/Python-3.10.0/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' I can't find _ssl.so in build or install directory. Thanks -S

6 12

Re: [OT] Re: Raw strings ending with a backslash
by Jonathan Goble 28 May '22

28 May '22

Trying again after I was mysteriously moderated. Thanks Ethan for fixing that. On Sat, May 28, 2022 at 6:59 PM Jonathan Goble <jcgoble3(a)gmail.com> wrote: > On Sat, May 28, 2022, 4:25 PM Gregory P. Smith <greg(a)krypto.org> wrote: > >> >> On Sat, May 28, 2022 at 12:55 PM Guido van Rossum <guido(a)python.org> >> wrote: >> >>> >>> On Sat, May 28, 2022 at 12:11 MRAB >>> >>> Names in Python are case-sensitive, yet the string prefixes are >>>> case-/insensitive/. >>>> >>>> Why? >>> >>> >>> IIRC we copied this from C for numeric suffixes (0l and 0L are the >>> same; also hex digits and presumably 0XA == 0xa) and then copied that for >>> string prefixes without thinking about it much. I guess it’s too late to >>> change. >>> >> >> Given that 99.99% of code uses lower case string prefixes we *could* >> change it, it'd just take a longer deprecation cycle - you'd probably want >> a few releases where the upper case prefixes become an error in files >> without a `from __future__ import case_sensitive_quote_prefixes` rather >> than jumping straight from parse time DeprecationWarning to repurposing the >> uppercase to have a new meaning. The inertia behind doing that over the >> course of 5+ years is high. Implying that we'd need a compelling reason to >> orchestrate it. None has sprung up. >> > > There already is a semantic meaning in one case, though not in Python > proper. Some syntax highlighters, including the one used in VSCode, treat r > and R differently: the former is syntax highlighted as a regex and the > latter is syntax highlighted as an generic string. I have seen > project-specific style guides advising to use r/R accordingly. > > So there is meaningful use of capital R in real-world code, and any future > change to require lowercase would need to at least consider the impact on > that use case. > >>

1 0

Summary of Python tracker Issues
by Python tracker 27 May '22

27 May '22

ACTIVITY SUMMARY (2022-05-20 - 2022-05-27) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/issue47258 #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47253: LOAD_GLOBAL instruction with wrong source position https://bugs.python.org/issue47253 #47252: socket.makefile documentation is missing data regarding the 'b https://bugs.python.org/issue47252 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47244: email.utils.formataddr does not respect double spaces https://bugs.python.org/issue47244 #47242: Annoying white bar in IDLE (line 457 in sidebar.py) https://bugs.python.org/issue47242 #47241: [C API] Move the PyCodeObject structure to the internal C API https://bugs.python.org/issue47241 #47238: Python threading.Event().wait() depends on the system time https://bugs.python.org/issue47238 #47236: Document types.CodeType.replace() changes about co_exceptionta https://bugs.python.org/issue47236 #47228: Document that naïve datetime objects represent local time https://bugs.python.org/issue47228 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47219: asyncio with two interpreter instances https://bugs.python.org/issue47219 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 Most recent 15 issues waiting for review (15) ============================================= #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47255: Many broken :meth: roles in the docs https://bugs.python.org/issue47255 #47254: enhanced dir? https://bugs.python.org/issue47254 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47243: Duplicate entry in 'Objects/unicodetype_db.h' https://bugs.python.org/issue47243 #47233: show_caches option affects code positions reported by dis.get_ https://bugs.python.org/issue47233 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 #47216: adding mtime option to gzip open() https://bugs.python.org/issue47216 #47215: Add "unstable" frame stack api https://bugs.python.org/issue47215 #47208: Support libffi implementations that cannot support invocations https://bugs.python.org/issue47208 #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo https://bugs.python.org/issue47205 #47200: Add ZipInfo.mode property https://bugs.python.org/issue47200 #47199: multiprocessing: micro-optimize Connection.send_bytes() method https://bugs.python.org/issue47199

1 0

Advice/proceedures for setting up cPython on WSL with cLion
by ward.davidj＠gmail.com 27 May '22

27 May '22

Hello, I can build cPython with PCbuild/build.bat and also in WSL/Debian `make` works fine but if I try to use the WSL toolchain/build configuration I get `g++: fatal error: no input files`. Clion is better geared for cmake projects and there is the `https://github.com/python-cmake-buildsystem/python-cmake-buildsystem` project. Has anyone made a tutorial/recipe for getting cLion and cpython to play nice? If I can't get cLion to work directly with cpython, I am curious what other Window's centric people use: Visual Studio or just the PCbuild.bat script? Thanks, DevDave/David

1 0

Re: compiled python3.10 is unable to find _ssl
by Lou King 25 May '22

25 May '22

I should have mentioned, earlier I made these changes to Setup 2022-05-22 14:54 vim /home/lking/python/Python-3.10.4/Modules/Setup 207c207 < #_socket socketmodule.c --- > _socket socketmodule.c 211,217c211,217 < # OPENSSL=/path/to/openssl/directory < # _ssl _ssl.c \ < # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ < # -lssl -lcrypto < #_hashlib _hashopenssl.c \ < # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ < # -lcrypto --- > OPENSSL=/usr/local/custom-openssl > _ssl _ssl.c \ > -I$(OPENSSL)/include -L$(OPENSSL)/lib \ > -lssl -lcrypto > _hashlib _hashopenssl.c \ > -I$(OPENSSL)/include -L$(OPENSSL)/lib \ > -lcrypto

1 0

Riddle: Funny decorator chaining behaviour
by Jesus Cea 25 May '22

25 May '22

Chatting in a Spanish python podcast about how convenient would be for abstract classes to raise exceptions intermediately instead of at instantiation time (how could you create an abstract class in that case?), somebody produced this code: import abc class a(abc.ABC): @classmethod @property @abc.abstractmethod def x(cls): print("Hello world") class b(a): def x(*args): pass (NOTHING happens, the abstract method/property is overriden) class c(a): pass "HELLO WORLD" is printed, creating the class (not instanciating an object of that type but creating the class itself), because the abstract method/property is not overriden. Note that the method is called at class creation time. if you replace the body of "x" method with a "raise NotImplementedError" you can produce that exception at import time, if some class remains abstract. This is an interesting interaction between decorators trying to be clever supporting both regular and abstract methods/properties. Trying to figure out how it worked took some time and python source code diving. I think this is interesting enough to share. I don't think this is a bug to solve, just some brain teasing. I let other to decide what to do with this... collateral effect. Let me know what do you think after trying to understand how it works. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - https://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

2 1