Mailman 3 October 2021 - Python-Dev

PEP 638: Syntactic macros
by Mark Shannon 05 Feb '23

05 Feb '23

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com 19 Oct '22

19 Oct '22

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin 21 Sep '22

21 Sep '22

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final pronouncement one way or the other. I personally believe that the PEP will result in a significant reduction in boilerplate code, and it is substantially similar to the same operators now found in a number of other languages, especially C# and JavaScript (https://wikipedia.org/wiki/Null_coalescing_operator and https://wikipedia.org/wiki/Safe_navigation_operator). I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding (and that adding additional support for it to the language will increase the use of None in this way), but I also believe there are many use cases in programming where it is by far the simplest way to express various semantics, and the fact exists that None is already used extensively in large quantities of code, and further that there is already a great deal of code written to constantly test against None and break out of a statement without throwing an error. I also understand the argument that especially the maybe-dot (?.) and maybe-subscript (?[) operators can decrease readability of code and also believe these are valid arguments against it. While I believe the existence and use of these operators in other languages definitely helps the case that these can be used and understood successfully, I think it is entirely valid to either consider other syntax (though I prefer the chosen syntax of PEP 505), or even to reduce PEP 505 to having only the coalesce operator (??) and the maybe-assign operator (??=). Separately, I have implemented a pure-Python solution for PEP505 (which is definitely rather beta) which might help test the waters for a final implementation in CPython (though the CPython implementation would of course be much more efficient). It can be found at https://pypi.org/project/pep505/ Thanks, Doug

22 53

PEP 563 and Python 3.10.
by Thomas Wouters 22 Jul '22

22 Jul '22

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!

6 5

compiled python3.10 is unable to find _ssl
by Sandeep Gupta 29 May '22

29 May '22

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/shared/Builds/Python-3.10.0/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' I can't find _ssl.so in build or install directory. Thanks -S

6 12

Python multithreading without the GIL
by Sam Gross 25 Apr '22

25 Apr '22

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

23 47

PEP 651, Robust Stack Overflow Handling, Rejection notice
by Python Steering Council 31 Jan '22

31 Jan '22

Hi Mark, Thank you for submitting PEP 651. The Steering Council has spent the past two weeks reviewing PEP 651. After careful consideration, we have decided to reject the PEP. The following were the key points that led us to this decision: * The benefits are not compelling enough. Deep recursion is not a common tool in Python, and even with PEP 651 it would not be efficient enough to make it a common tool. * The benefit of PEP 651 is negated as soon as a non-Python function is involved in the recursion, making the likelihood of it being useful even smaller. It also creates easy pitfalls for users who do end up relying on recursion. * We believe the PEP understates the disruption created by the technical solution of multiple Python stack frames per C call. Although this may be solvable, it will certainly cause substantial disruption to existing debuggers, tracers, and state inspection tools as they need to adapt to this change (which may not be trivial). * As the way to approach this will be platform-specific (as some parts of the proposal are not portable), this can cause generic Python code to behave differently on different platforms, making this kind of code less portable and less predictable. In the end, the benefit of the PEP does not outweigh the cost of the potential breakage, confusion, and unpredictability of the feature. With our appreciation, The Python Steering Council

6 8

PEP 646 (Variadic Generics): final call for comments
by Matthew Rahtz 18 Jan '22

18 Jan '22

Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.) Thanks! Matthew and Pradeep

9 19

PEP 670: Convert macros to functions in the Python C API
by Victor Stinner 24 Nov '21

24 Nov '21

Hi, Erlend and me wrote a PEP to move away from macros in the Python C API. We are now waiting for feedback :-) Read the PEP online: https://www.python.org/dev/peps/pep-0670/ There is a copy of the PEP below for inline replies. Victor --- PEP: 670 Title: Convert macros to functions in the Python C API Author: Erlend Egeberg Aasland <erlend.aasland(a)protonmail.com>, Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-Oct-2021 Python-Version: 3.11 Abstract ======== Convert macros to static inline functions or regular functions. Remove the return value of macros having a return value, whereas they should not, to aid detecting bugs in C extensions when the C API is misused. Some function arguments are still cast to ``PyObject*`` to prevent emitting new compiler warnings. Rationale ========= The use of macros may have unintended adverse effects that are hard to avoid, even for experienced C developers. Some issues have been known for years, while others have been discovered recently in Python. Working around macro pitfalls makes the macro coder harder to read and to maintain. Converting macros to functions has multiple advantages: * By design, functions don't have macro pitfalls. * Arguments type and return type are well defined. * Debuggers and profilers can retrieve the name of inlined functions. * Debuggers can put breakpoints on inlined functions. * Variables have a well defined scope. * Code is usually easier to read and to maintain than similar macro code. Functions don't need the following workarounds for macro pitfalls: * Add parentheses around arguments. * Use line continuation characters if the function is written on multiple lines. * Add commas to execute multiple expressions. * Use ``do { ... } while (0)`` to write multiple statements. Converting macros and static inline functions to regular functions makes these regular functions accessible to projects which use Python but cannot use macros and static inline functions. Macro Pitfalls ============== The `GCC documentation <https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_ lists several common macro pitfalls: - Misnesting - Operator precedence problems - Swallowing the semicolon - Duplication of side effects - Self-referential macros - Argument prescan - Newlines in arguments Performance and inlining ======================== Static inline functions is a feature added to the C99 standard. Modern C compilers have efficient heuristics to decide if a function should be inlined or not. When a C compiler decides to not inline, there is likely a good reason. For example, inlining would reuse a register which require to save/restore the register value on the stack and so increase the stack memory usage or be less efficient. Debug build ----------- When Python is built in debug mode, most compiler optimizations are disabled. For example, Visual Studio disables inlining. Benchmarks must not be run on a Python debug build, only on release build: using LTO and PGO is recommended for reliable benchmarks. PGO helps the compiler to decide if function should be inlined or not. Force inlining -------------- The ``Py_ALWAYS_INLINE`` macro can be used to force inlining. This macro uses ``__attribute__((always_inline))`` with GCC and Clang, and ``__forceinline`` with MSC. So far, previous attempts to use ``Py_ALWAYS_INLINE`` didn't show any benefit and were abandoned. See for example: `bpo-45094 <https://bugs.python.org/issue45094>`_: "Consider using ``__forceinline`` and ``__attribute__((always_inline))`` on static inline functions (``Py_INCREF``, ``Py_TYPE``) for debug build". When the ``Py_INCREF()`` macro was converted to a static inline functions in 2018 (`commit <https://github.com/python/cpython/commit/2aaf0c12041bcaadd7f2cc5a54450eefd7…>`__), it was decided not to force inlining. The machine code was analyzed with multiple C compilers and compiler options: ``Py_INCREF()`` was always inlined without having to force inlining. The only case where it was not inlined was the debug build. See discussion in the `bpo-35059 <https://bugs.python.org/issue35059>`_: "Convert ``Py_INCREF()`` and ``PyObject_INIT()`` to inlined functions". Disable inlining ---------------- On the other side, the ``Py_NO_INLINE`` macro can be used to disable inlining. It is useful to reduce the stack memory usage. It is especially useful on a LTO+PGO build which is more aggressive to inline code: see `bpo-33720 <https://bugs.python.org/issue33720>`_. The ``Py_NO_INLINE`` macro uses ``__attribute__ ((noinline))`` with GCC and Clang, and ``__declspec(noinline)`` with MSC. Specification ============= Convert macros to static inline functions ----------------------------------------- Most macros should be converted to static inline functions to prevent `macro pitfalls`_. The following macros should not be converted: * Empty macros. Example: ``#define Py_HAVE_CONDVAR``. * Macros only defining a number, even if a constant with a well defined type can better. Example: ``#define METH_VARARGS 0x0001``. * Compatibility layer for different C compilers, C language extensions, or recent C features. Example: ``#define Py_ALWAYS_INLINE __attribute__((always_inline))``. Convert static inline functions to regular functions ---------------------------------------------------- The performance impact of converting static inline functions to regular functions should be measured with benchmarks. If there is a significant slowdown, there should be a good reason to do the conversion. One reason can be hiding implementation details. Using static inline functions in the internal C API is fine: the internal C API exposes implemenation details by design and should not be used outside Python. Cast to PyObject* ----------------- When a macro is converted to a function and the macro casts its arguments to ``PyObject*``, the new function comes with a new macro which cast arguments to ``PyObject*`` to prevent emitting new compiler warnings. So the converted functions still accept pointers to structures inheriting from ``PyObject`` (ex: ``PyTupleObject``). For example, the ``Py_TYPE(obj)`` macro casts its ``obj`` argument to ``PyObject*``:: #define _PyObject_CAST_CONST(op) ((const PyObject*)(op)) static inline PyTypeObject* _Py_TYPE(const PyObject *ob) { return ob->ob_type; } #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob)) The undocumented private ``_Py_TYPE()`` function must not be called directly. Only the documented public ``Py_TYPE()`` macro must be used. Later, the cast can be removed on a case by case basis, but that is out of scope for this PEP. Remove the return value ----------------------- When a macro is implemented as an expression, it has an implicit return value. In some cases, the macro must not have a return value and can be misused in third party C extensions. See `bpo-30459 <https://bugs.python.org/issue30459>`_ for the example of ``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros. It is not easy to notice this issue while reviewing macro code. These macros are converted to functions using the ``void`` return type to remove their return value. Removing the return value aids detecting bugs in C extensions when the C API is misused. Backwards Compatibility ======================= Removing the return value of macros is an incompatible API change made on purpose: see the `Remove the return value`_ section. Rejected Ideas ============== Keep macros, but fix some macro issues -------------------------------------- Converting macros to functions is not needed to `remove the return value`_: casting a macro return value to ``void`` also fix the issue. For example, the ``PyList_SET_ITEM()`` macro was already fixed like that. Macros are always "inlined" with any C compiler. The duplication of side effects can be worked around in the caller of the macro. People using macros should be considered "consenting adults". People who feel unsafe with macros should simply not use them. Examples of hard to read macros =============================== _Py_NewReference() ------------------ Example showing the usage of an ``#ifdef`` inside a macro. Python 3.7 macro (simplified code):: #ifdef COUNT_ALLOCS # define _Py_INC_TPALLOCS(OP) inc_count(Py_TYPE(OP)) # define _Py_COUNT_ALLOCS_COMMA , #else # define _Py_INC_TPALLOCS(OP) # define _Py_COUNT_ALLOCS_COMMA #endif /* COUNT_ALLOCS */ #define _Py_NewReference(op) ( \ _Py_INC_TPALLOCS(op) _Py_COUNT_ALLOCS_COMMA \ Py_REFCNT(op) = 1) Python 3.8 function (simplified code):: static inline void _Py_NewReference(PyObject *op) { _Py_INC_TPALLOCS(op); Py_REFCNT(op) = 1; } PyObject_INIT() --------------- Example showing the usage of commas in a macro. Python 3.7 macro:: #define PyObject_INIT(op, typeobj) \ ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) ) Python 3.8 function (simplified code):: static inline PyObject* _PyObject_INIT(PyObject *op, PyTypeObject *typeobj) { Py_TYPE(op) = typeobj; _Py_NewReference(op); return op; } #define PyObject_INIT(op, typeobj) \ _PyObject_INIT(_PyObject_CAST(op), (typeobj)) The function doesn't need the line continuation character. It has an explicit ``"return op;"`` rather than a surprising ``", (op)"`` at the end of the macro. It uses one short statement per line, rather than a single long line. Inside the function, the *op* argument has a well defined type: ``PyObject*``. References ========== * `bpo-45490 <https://bugs.python.org/issue45490>`_: [meta][C API] Avoid C macro pitfalls and usage of static inline functions (October 2021). * `What to do with unsafe macros <https://discuss.python.org/t/what-to-do-with-unsafe-macros/7771>`_ (March 2021). * `bpo-43502 <https://bugs.python.org/issue43502>`_: [C-API] Convert obvious unsafe macros to static inline functions (March 2021). Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -- Night gathers, and now my watch begins. It shall not end until my death.

6 19

Oh wow, this is really impressive
by Steven D'Aprano 08 Nov '21

08 Nov '21

I was using Python 3.10 and got this NameError when I mistyped a name: NameError: name 'KeyboardInterrupted' is not defined. Did you mean: 'KeyboardInterrupt'? It even works for attribute errors too. That's fantastic! This is a really amazing useability improvement, thank you everyone who was involved! I literally squeed :-) I just may spend a few days deliberately mistyping names in the REPL just to see this :-) -- Steve

4 3

Python-Dev October 2021