Mailman 3 April 2020 - Python-Dev

The Python 2 death march
by Benjamin Peterson 17 Apr '20

17 Apr '20

Hi all, It's finally time to schedule the last releases in Python 2's life. There will be two more releases of Python 2.7: Python 2.7.17 and Python 2.7.18. Python 2.7.17 release candidate 1 will happen on October 5th followed by the final release on October 19th. I'm going to time Python 2.7.18 to coincide with PyCon 2020 in April, so attendees can enjoy some collective catharsis. We'll still say January 1st is the official EOL date. Thanks to Sumana Harihareswara, there's now a FAQ about the Python 2 sunset on the website: https://www.python.org/doc/sunset-python-2/ Regards, Benjamin

24 38

Re: Improvement to SimpleNamespace
by Raymond Hettinger 16 Apr '20

16 Apr '20

[Serhiy] > As a workaround you can use > > object_hook=lambda x: SimpleNamespace(**x) That doesn't suffice because some valid JSON keys are not valid identifiers. You still need a way to get past those when they arise: catalog.books.fiction['Paradise Lost'].isbn Also, it still leaves you with using setattr(ns, attrname, attrvalue) or tricks with vars() when doing updates. The AttrDict recipe is popular for a reason. Raymond

3 2

PEP: Modify the C API to hide implementation details
by Victor Stinner 15 Apr '20

15 Apr '20

Hi, Here is a first draft a PEP which summarize the research work I'm doing on CPython C API since 2017 and the changes that me and others already made since Python 3.7 towards an "opaque" C API. The PEP is also a collaboration with developers of PyPy, HPy, Rust-CPython and many others! Thanks to everyone who helped me to write it down! Maybe this big document should be reorganized as multiple smaller better defined goals: as multiple PEPs. The PEP is quite long and talks about things which are not directly related. It's a complex topic and I chose to put everything as a single document to have a good starting point to open the discussion. I already proposed some of these ideas in 2017: see the Prior Art section ;-) The PEP can be read on GitHub where it's better formatted: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst If someone wants to work on the PEP itself, the document on GitHub is the current reference. Victor ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PEP xxx: Modify the C API to hide implementation details ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Abstract ======== * Hide implementation details from the C API to be able to `optimize CPython`_ and make PyPy more efficient. * The expectation is that `most C extensions don't rely directly on CPython internals`_ and so will remain compatible. * Continue to support old unmodified C extensions by continuing to provide the fully compatible "regular" CPython runtime. * Provide a `new optimized CPython runtime`_ using the same CPython code base: faster but can only import C extensions which don't use implementation details. Since both CPython runtimes share the same code base, features implemented in CPython will be available in both runtimes. * `Stable ABI`_: Only build a C extension once and use it on multiple Python runtimes and different versions of the same runtime. * Better advertise alternative Python runtimes and better communicate on the differences between the Python language and the Python implementation (especially CPython). Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. Rationale ========= To remain competitive in term of performance with other programming languages like Go or Rust, Python has to become more efficient. Make Python (at least) two times faster --------------------------------------- The C API leaks too many implementation details which prevent optimizing CPython. See `Optimize CPython`_. PyPy's support for Python's C API (pycext) is slow because it has to emulate CPython internals like memory layout and reference counting. The emulation causes memory overhead, memory copies, conversions, etc. See `Inside cpyext: Why emulating CPython C API is so Hard <https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c…>`_ (Sept 2018) by Antonio Cuni. While this PEP may make CPython a little bit slower in the short term, the long-term goal is to make "Python" at least two times faster. This goal is not hypothetical: PyPy is already 4.2x faster than CPython and is fully compatible. C extensions are the bottleneck of PyPy. This PEP proposes a migration plan to move towards opaque C API which would make PyPy faster. Separated the Python language and the CPython runtime (promote alternative runtimes) ------------------------------------------------------------------------------------ The Python language should be better separated from its runtime. It's common to say "Python" when referring to "CPython". Even in this PEP :-) Because the CPython runtime remains the reference implementation, many people believe that the Python language itself has design flaws which prevent it from being efficient. PyPy proved that this is a false assumption: on average, PyPy runs Python code 4.2 times faster than CPython. One solution for separating the language from the implementation is to promote the usage of alternative runtimes: not only provide the regular CPython, but also PyPy, optimized CPython which is only compatible with C extensions using the limited C API, CPython compiled in debug mode to ease debugging issues in C extensions, RustPython, etc. To make alternative runtimes viable, they should be competitive in term of features and performance. Currently, C extension modules remain the bottleneck for PyPy. Most C extensions don't rely directly on CPython internals ---------------------------------------------------------- While the C API is still tidely coupled to CPython internals, in practical, most C extensions don't rely directly on CPython internals. The expectation is that these C extensions will remain compatible with an "opaque" C API and only a minority of C extensions will have to be modified. Moreover, more and more C extensions are implemented in Cython or cffi. Updating Cython and cffi to be compatible with the opaque C API will make all these C extensions without having to modify the source code of each extension. Stable ABI ---------- The idea is to build a C extension only once: the built binary will be usable on multiple Python runtimes and different versions of the same runtime (stable ABI). The idea is not new but is an extension of the `PEP 384: Defining a Stable ABI <https://www.python.org/dev/peps/pep-0384/>`__ implemented in CPython 3.4 with its "limited C API". The limited API is not used by default and is not widely used: PyQt is one of the only few known users. The idea here is that the default C API becomes the limited C API and so all C extensions will benefit of advantages of a stable ABI. Flaws of the C API ================== Borrowed references ------------------- A borrowed reference is a pointer which doesn't “hold” a reference. If the object is destroyed, the borrowed reference becomes a dangling pointer, pointing to freed memory which might be reused by a new object. Borrowed references can lead to bugs and crashes when misused. An example of a CPython bug caused by this is `bpo-25750: crash in type_getattro() <https://bugs.python.org/issue25750>`_. Borrowed references are a problem whenever there is no reference to borrow: they assume that a referenced object already exists (and thus has a positive reference count). Tagged pointers are an example of this problem: since there is no concrete ``PyObject*`` to represent the integer, it cannot easily be manipulated. This issue complicates optimizations like PyPy's list strategies: if a list contains only small integers, it is stored as a compact C array of longs. The equivalent of ``PyObject`` is only created when an item is accessed. (Most of the time the object is optimized away by the JIT, but this is another story.) This makes it hard to support the C API function ``PyList_GetItem()``, which should return a reference borrowed from the list, but the list contains no concrete ``PyObject`` that it could lend a reference to! PyPy's current solution is very bad: the first time ``PyList_GetItem()`` is called, the whole list is de-optimized (converted to a list of ``PyObject*``). See ``cpyext`` ``get_list_storage()``. See also the Specialized list use case, which is the same optimization applied to CPython. Like in PyPy, this optimization is incompatible with borrowed references since the runtime cannot guess when the temporary object should be destroyed. If ``PyList_GetItem()`` returned a strong reference, the ``PyObject*`` could just be allocated on the fly and destroyed when the user decrements its reference count. Basically, by putting borrowed references in the API, we are making it impossible to change the underlying data structure. Functions stealing strong references ------------------------------------ There are functions which steal strong references, for example ``PyModule_AddObject()`` and ``PySet_Discard()``. Stealing references is an issue similar to borrowed references. PyObject** ---------- Some functions of the C API return a pointer to an array of ``PyObject*``: * ``PySequence_Fast_ITEMS()`` * ``PyTuple_GET_ITEM()`` is sometimes abused to get an array of all of the tuple's contents: ``PyObject **items = &PyTuple_GET_ITEM(0);`` In effect, these functions return an array of borrowed references: like with ``PyList_GetItem()``, all callers of ``PySequence_Fast_ITEMS()`` assume the sequence holds references to its elements. Leaking structure members ------------------------- ``PyObject``, ``PyTypeObject``, ``PyThreadState``, etc. structures are currently public: C extensions can directly read and modify the structure members. For example, the ``Py_INCREF()`` macro directly increases ``PyObject.ob_refcnt``, without any abstraction. Hopefully, ``Py_INCREF()`` implementation can be modified without affecting the API. Change the C API ================ This PEP doesn't define an exhaustive list of all C API changes, but define some guidelines of bad patterns which should be avoided in the C API to prevent leaking implementation details. Separate header files of limited and internal C API --------------------------------------------------- In Python 3.6, all headers (.h files) were directly in the ``Include/`` directory. In Python 3.7, work started to move the internal C API into a new subdirectory, ``Include/internal/``. The work continued in Python 3.8 and 3.9. The internal C API is only partially exported: some functions are only declared with ``extern`` and so cannot be used outside CPython (with compilers supporting ``-fvisibility=hidden``, see above), whereas some functions are exported with ``PyAPI_FUNC()`` to make them usable in C extensions. Debuggers and profilers are typical users of the internal C API to inspect Python internals without calling functions (to inspect a coredump for example). Python 3.9 is now built with ``-fvisibility=hidden`` (supported by GCC and clang): symbols which are not declared with ``PyAPI_FUNC()`` or ``PyAPI_DATA()`` are no longer exported by the dynamical library (libpython). Another change is to separate the limited C API from the "CPython" C API: Python 3.8 has a new ``Include/cpython/`` sub-directory. It should not be used directly, but it is used automatically from the public headers when the ``Py_LIMITED_API`` macro is not defined. **Backward compatibility:** fully backward compatible. **Status:** basically completed in Python 3.9. Changes without API changes and with minor performance overhead --------------------------------------------------------------- * Replace macros with static inline functions. Work started in 3.8 and made good progress in Python 3.9. * Modify macros to avoid directly accessing structures fields. For example, the `Hide implementation detail of trashcan macros <https://github.com/python/cpython/commit/38965ec5411da60d312b59be281f3510d5…>`_ commit modifies ``Py_TRASHCAN_BEGIN_CONDITION()`` macro to call a new ``_PyTrash_begin()`` function rather than accessing directly ``PyThreadState.trash_delete_nesting`` field. **Backward compatibility:** fully backward compatible. **Status:** good progress in Python 3.9. Changes without API changes but with performance overhead --------------------------------------------------------- Replace macros or inline functions with regular functions. Work started in 3.9 on a limited set of functions. Converting macros to function calls can have a small overhead on performances. For example, ``Py_INCREF()`` macro modifies directly ``PyObject.ob_refcnt``: this macro would become an alias to the opaque ``Py_IncRef()`` function. It is possible that the regular CPython runtime keeps the ``Py_INCREF()`` macro which modifies directly ``PyObject.ob_refcnt`` to avoid any performance overhead. A tradeoff should be defined to limit differences between the regular and the new optimized CPython runtimes, without hurting too much performances of the regular CPython runtime. **Backward compatibility:** fully backward compatible. **Status:** not started. The performance overhead must be measured with benchmarks and this PEP should be accepted. API and ABI incompatible changes -------------------------------- * Make structures opaque: move them to the internal C API. * Remove functions from the public C API which are tied to CPython internals. Maybe begin by marking these functions as private (rename ``PyXXX`` to ``_PyXXX``) or move them to the internal C API. * Ban statically allocated types (by making ``PyTypeObject`` opaque): enforce usage of ``PyType_FromSpec()``. Examples of issues to make structures opaque: * ``PyGC_Head``: https://bugs.python.org/issue40241 * ``PyObject``: https://bugs.python.org/issue39573 * ``PyTypeObject``: https://bugs.python.org/issue40170 * ``PyThreadState``: https://bugs.python.org/issue39573 Another example are ``Py_REFCNT()`` and ``Py_TYPE()`` macros which can currently be used l-value to modify an object reference count or type. Python 3.9 has new ``Py_SET_REFCNT()`` and ``Py_SET_TYPE()`` macros which should be used instead. ``Py_REFCNT()`` and ``Py_TYPE()`` macros should be converted to static inline functions to prevent their usage as l-value. **Backward compatibility:** backward incompatible on purpose. Break the limited C API and the stable ABI, with the assumption that `Most C extensions don't rely directly on CPython internals`_ and so will remain compatible. CPython specific behavior ========================= Some C functions and some Python functions have a behavior which is closely tied to the current CPython implementation. is operator ----------- The "x is y" operator is closed tied to how CPython allocates objects and to ``PyObject*``. For example, CPython uses singletons for numbers in [-5; 256] range:: >>> x=1; (x + 1) is 2 True >>> x=1000; (x + 1) is 1001 False Python 3.8 compiler now emits a ``SyntaxWarning`` when the right operand of the ``is`` and ``is not`` operators is a literal (ex: integer or string), but don't warn if it is ``None``, ``True``, ``False`` or ``Ellipsis`` singleton (`bpo-34850 <https://bugs.python.org/issue34850>`_). Example:: >>> x=1 >>> x is 1 <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="? True CPython PyObject_RichCompareBool -------------------------------- CPython considers that two objects are identical if their memory address are equal: ``x is y`` in Python (``IS_OP`` opcode) is implemented internally in C as ``left == right`` where ``left`` and ``right`` are ``PyObject *`` pointers. The main function to implement comparison in CPython is ``PyObject_RichCompareBool()``. This function considers that two objects are equal if the two ``PyObject*`` pointers are equal (if the two objects are "identical"). For example, ``PyObject_RichCompareBool(obj1, obj2, Py_EQ)`` doesn't call ``obj1.__eq__(obj2)`` if ``obj1 == obj2`` where ``obj1`` and ``obj2`` are ``PyObject*`` pointers. This behavior is an optimization to make Python more efficient. For example, the ``dict`` lookup avoids ``__eq__()`` if two pointers are equal. Another example are Not-a-Number (NaN) floating pointer numbers which are not equal to themselves:: >>> nan = float("nan") >>> nan is nan True >>> nan == nan False The ``list.__contains__(obj)`` and ``list.index(obj)`` methods are implemented with ``PyObject_RichCompareBool()`` and so rely on objects identity:: >>> lst = [9, 7, nan] >>> nan in lst True >>> lst.index(nan) 2 >>> lst[2] == nan False In CPython, ``x == y`` is implemented with ``PyObject_RichCompare()`` which don't make the assumption that identical objects are equal. That's why ``nan == nan`` or ``lst[2] == nan`` return ``False``. Issues for other Python implementations --------------------------------------- The Python language doesn't require to be implemented with ``PyObject`` structure and use ``PyObject*`` pointers. PyPy doesn't use ``PyObject`` nor ``PyObject*``. If CPython is modified to use `Tagged Pointers`_, CPython would have the same issue. Alternative Python implementations have to mimick CPython to reduce incompatibilities. For example, PyPy mimicks CPython behavior for the ``is`` operator with CPython small integer singletons:: >>>> x=1; (x + 1) is 2 True It also mimicks CPython ``PyObject_RichCompareBool()``. Example with the Not-a-Number (NaN) float:: >>>> nan=float("nan") >>>> nan == nan False >>>> lst = [9, 7, nan] >>>> nan in lst True >>>> lst.index(nan) 2 >>>> lst[2] == nan False Better advertise alternative Python runtimes ============================================ Currently, PyPy and other "alternative" Python runtimes are not well advertised on the `Python website <https://www.python.org/>`_. They are only listed as the last choice in the Download menu. Once enough C extensions will be compatible with the limited C API, PyPy and other Python runtimes should be better advertised on the Python website and in the Python documentation, to no longer introduce them as as first-class citizen. Obviously, CPython is likely to remain the most feature-complete implementation in mid-term, since new PEPs are first implemented in CPython. Limitations can be simply documented, and users should be free to make their own choice, depending on their use cases. HPy project =========== The `HPy project <https://github.com/pyhandle/hpy>`__ is a brand new C API written from scratch. It is designed to ease migration from the current C API and to be efficient on PyPy. HPy hides all implementation details: it is based on "handles" so objects cannot be inspected with direct memory access: only opaque function calls are allowed. This abstraction has many benefits: * No more ``PyObject`` emulation needed: smaller memory footprint in PyPy cpyext, no more expensive conversions. * It is possible to have multiple handles pointing to the same object. It helps to better track the object lifetime and makes the PyPy implementation easier. PyPy doesn't use reference counting but a tracing garbage collector. When the PyPy GC moves objects in memory, handles don't change! HPy uses an array mapping handle to objects: only this array has to be updated. It is way more efficient. * The Python runtime is free to modify deep internals compared to CPython. Many optimizations become possible: see `Optimize CPython`_ section. * It is easy to add a debug wrapper to add checks before and after the function calls. For example, ensure that that GIL is held when calling CPython. HPy is developed outside CPython, is implemented on top of the existing Python C API, and so can support old Python versions. By default, binaries compiled in "universal" HPy ABI mode can be used on CPython and PyPy. HPy can also target CPython ABI which has the same performance than native C extensions. See HPy documentation of `Target ABIs documentation <https://github.com/pyhandle/hpy/blob/feature/improve-docs/docs/overview.rst…>`_. The PEP moves the C API towards HPy design and API. New optimized CPython runtime ============================== Backward incompatible changes is such a pain for the whole Python community. To ease the migration (accelerate adoption of the new C API), one option is to provide not only one but two CPython runtimes: * Regular CPython: fully backward compatible, support direct access to structures like ``PyObject``, etc. * New optimized CPython: incompatible, cannot import C extensions which don't use the limited C API, has new optimizations, limited to the C API. Technically, both runtimes would have the same code base, to ease maintenance: CPython. The new optimized CPython would be a ./configure flag to build a different Python. On Windows, it would be a different project of the Visual Studio solution reusing pythoncore project, but define a macro to build enable optimization and change the C API. The new optimized CPython runtime remains compatible with CPython 3.8 `stable ABI`_. CPython code base remains 30 years old. Many technical choices made 30 years ago are no longer relevant today. This PEP should ease the development of new Python implementation which would be even more efficient, like PyPy! Cython and cffi =============== Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. Cython may be modified to add a new build mode where only the "limited C API" is used. Use Cases ========= Optimize CPython ---------------- The new optimized runtime can implement new optimizations since it only supports C extension modules which don't access Python internals. Tagged pointers ............... `Tagged pointer <https://en.wikipedia.org/wiki/Tagged_pointer>`_. Avoid ``PyObject`` for small objects (ex: small integers, short Latin-1 strings, None and True/False singletons): store the content directly in the pointer, with a tag for the object type. Tracing garbage collector ......................... Experiment with a tracing garbage collector inside CPython. Keep reference counting for the C API. Rewriting CPython with a tracing garbage collector is large project which is out of the scope of this PEP. This PEP fix some blockers issues which prevent to start such project today. One of the issue are functions of the C API which return a pointer like ``PyBytes_AsString()``. Python doesn't know when the caller stops using the pointer, and so cannot move the object in memory (for a moving garbage collector). API like ``PyBuffer`` is better since it requires the caller to call ``PyBuffer_Release()`` when it is done. Specialized list ................ Specialize lists of small integers: if a list only contains numbers which fit into a C ``int32_t``, a Python list object could use a more efficient ``int32_t`` array to reduce the memory footprint (avoid ``PyObject`` overhead for these numbers). Temporary ``PyObject`` objects would be created on demand for backward compatibility. This optimization is less interesting if tagged pointers are implemented. PyPy already implements this optimization. O(1) bytearray to bytes conversion .................................. Convert bytearray to bytes without memory copy. Currently, bytearray is used to build a bytes string, but it's usually converted into a bytes object to respect an API. This conversion requires to allocate a new memory block and copy data (O(n) complexity). It is possible to implement O(1) conversion if it would be possible to pass the ownership of the bytearray object to bytes. That requires modifying the ``PyBytesObject`` structure to support multiple storages (support storing content into a separate memory block). Fork and "Copy-on-Read" problem ............................... Solve the "Copy on read" problem with fork: store reference counter outside ``PyObject``. Currently, when a Python object is accessed, its ``ob_refcnt`` member is incremented temporarily to hold a "strong reference" to it (ensure that it cannot be destroyed while we use it). Many operating system implement fork() using copy-on-write ("CoW"). A memory page (ex: 4 KB) is only copied when a process (parent or child) modifies it. After Python is forked, modifying ``ob_refcnt`` copies the memory page, even if the object is only accessed in "read only mode". `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-i…>`_ (Jan 2017) by Instagram Engineering. Instagram contributed `gc.freeze() <https://docs.python.org/dev/library/gc.html#gc.freeze>`_ to Python 3.7 which works around the issue. One solution for that would be to store reference counters outside ``PyObject``. For example, in a separated hash table (pointer to reference counter). Changing ``PyObject`` structures requires that C extensions don't access them directly. Debug runtime and remove debug checks in release mode ..................................................... If the C extensions are no longer tied to CPython internals, it becomes possible to switch to a Python runtime built in debug mode to enable runtime debug checks to ease debugging C extensions. If using such a debug runtime becomes harder, indirectly it means that runtime debug checks can be removed from the release build. CPython code base is still full of runtime checks calling ``PyErr_BadInternalCall()`` on failure. Removing such checks in release mode can make Python more efficient. PyPy ---- ujson is 3x faster on PyPy when using HPy instead of the Python C API. See `HPy kick-off sprint report <https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.html>`_ (December 2019). This PEP should help to make PyPy cpyext more efficient, or at least ease the migration of C extensions to HPy. GraalPython ----------- `GraalPython <https://github.com/graalvm/graalpython>`_ is a Python 3 implementation built on `GraalVM <https://www.graalvm.org/>`_ ("Universal VM for a polyglot world"). It is interested in supporting HPy. See `Leysin 2020 Sprint Report <https://morepypy.blogspot.com/2020/03/leysin-2020-sprint-report.html>`_. It would also benefit of this PEP. RustPython, Rust-CPython and PyO3 --------------------------------- Rust-CPython is interested in supporting HPy. See `Leysin 2020 Sprint Report <https://morepypy.blogspot.com/2020/03/leysin-2020-sprint-report.html>`_. RustPython and PyO3 would also benefit of this PEP. Links: * `PyO3 <https://github.com/PyO3/pyo3>`_: Rust bindings for the Python (CPython) interpreter * `rust-cpython <https://github.com/dgrunwald/rust-cpython>`_: Rust <-> Python (CPython) bindings * `RustPython <https://github.com/RustPython/RustPython>`_: A Python Interpreter written in Rust Rejected Ideas ============== Drop the C API -------------- One proposed alternative to a new better C API is to drop the C API at all. The reasoning is that since existing solutions are already available, complete and reliable, like Cython and cffi. What about the long tail of C extensions on PyPI which still use the C API? Would a Python without these C extensions would remain relevant? Lots of project do not use those solution, and the C API is part of Python success. For example, there would be no numpy without the C API. It doesn't sound like a workable solution. Bet on HPy, leave the C API unchanged ------------------------------------- The HPy project is developed outside CPython and so doesn't cause any backward incompatibility in CPython. HPy API was designed with efficiency in mind. The problem is the long tail of C extensions on PyPI which are written with the C API and will not be converted soon or will never be converted to HPy. The transition from Python 2 to Python 3 showed that migrations are very slow and never fully complete. The PEP also rely on the assumption that `Most C extensions don't rely directly on CPython internals`_ and so will remain compatible with the new opaque C API. The concept of HPy is not new: CPython has a limited C API which provides a stable ABI since Python 3.4, see `PEP 384: Defining a Stable ABI <https://www.python.org/dev/peps/pep-0384/>`_. Since it is an opt-in option, most users simply use the **default** C API. Prior Art ========= * `pythoncapi.readthedocs.io <https://pythoncapi.readthedocs.io/>`_: Research project behind this PEP * July 2019: Keynote `Python Performance: Past, Present, Future <https://github.com/vstinner/talks/raw/master/2019-EuroPython/python_perform…>`_ (slides) by Victor Stinner at EuroPython 2019 * [python-dev] `Make the stable API-ABI usable <https://mail.python.org/pipermail/python-dev/2017-November/150607.html>`_ (November 2017) by Victor Stinner * [python-ideas] `PEP: Hide implementation details in the C API <https://mail.python.org/pipermail/python-ideas/2017-July/046399.html>`_ (July 2017) by Victor Stinner. Old PEP draft which proposed to add an option to build C extensions. * `A New C API for CPython <https://vstinner.github.io/new-python-c-api.html>`_ (Sept 2017) article by Victor Stinner * `Python Performance <https://github.com/vstinner/conf/raw/master/2017-PyconUS/summit.pdf>`_ (May 2017 at the Language Summit) by Victor Stinner: early discusssions on reorganizing header files, promoting PyPy, fix the C API, etc. Discussion summarized in `Keeping Python competitive <https://lwn.net/Articles/723949/>`_ article. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -- Night gathers, and now my watch begins. It shall not end until my death.

16 43

Providing fix for modulefinder.py regression on Windows
by Barry Scott 15 Apr '20

15 Apr '20

modulefinder.py does not open source files in "rb" which prevents compile() from applying the encoding rules. This first showed up for me on Windows with Python 3.8. I'd like to provide patches for 3.8 (if any more releases) and 3.9. What is the process I should follow? I have signed the Contributor Agreement. Here is my test case and the results on Windows with 3.7. import modulefinder with open('import_functools.py', 'w') as f: f.write('import functools\n') mf = modulefinder.ModuleFinder() mf.run_script('import_functools.py') print('Test passed') py -3.8-32 modulefinder_test.py Traceback (most recent call last): File "modulefinder_test.py", line 7, in <module> mf.run_script('import_functools.py') File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script self.load_module('__main__', fp, pathname, stuff) File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module self.scan_code(co, m) File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code self._safe_import_hook(name, m, fromlist, level=0) File "C:\Python38.win32\lib\modulefinder.py", line 378, in _safe_import_hook self.import_hook(name, caller, level=level) File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook q, tail = self.find_head_package(parent, name) File "C:\Python38.win32\lib\modulefinder.py", line 233, in find_head_package q = self.import_module(head, qname, parent) File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module m = self.load_module(fqname, fp, pathname, stuff) File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module co = compile(fp.read()+'\n', pathname, 'exec') File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 308: character maps to <undefined> Barry

5 6

Do we need port some extension modules to the modules directory?
by Hai Shi 14 Apr '20

14 Apr '20

Hi, I notice some modules not in modules directory(for example: _warnings、marshal in python directory). Do we need port those modules to modules directory? Sorry for my noisy info. Bests Wishs, Hai Shi

3 4

Python2 as 𝑣 → 𝑒
by Mike Miller 12 Apr '20

12 Apr '20

Unless I've read something wrong, it looks like the final Python 2 release (2.7.18) should approximate the math constant e: >>> import math >>> math.e 2.718281828459045 Aka: Python as 𝑣 → 𝑒 Python ≈ 𝑒 Would be fun to note that in the announcement/docs somehow. -Mike

4 3

Fixing Copy on Writes with Immortal Instances
by Eddie Elizondo 12 Apr '20

12 Apr '20

Copy on writes are a big problem in large Python application that rely on multiple processes sharing the same memory. I just sent out a patch introduces Immortal Instances which allows the user to bypass reference counting on specific objects to avoid CoW on forked processes. This has helped in reducing the memory constraints in some of our applications. The PR has a more in-depth explanation about the change. Looking forward to get your thoughts. Github PR: https://github.com/python/cpython/pull/19474 Bug Report: https://bugs.python.org/issue40255 - Eddie E.

1 0

Summary of Python tracker Issues
by Python tracker 11 Apr '20

11 Apr '20

ACTIVITY SUMMARY (2020-04-03 - 2020-04-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7433 (+32) closed 44563 (+37) total 51996 (+69) Open issues with patches: 2934 Issues opened (50) ================== #39984: Move pending calls from _PyRuntime to PyInterpreterState https://bugs.python.org/issue39984 reopened by vstinner #40178: Convert the remaining os funtions to Argument Clinic https://bugs.python.org/issue40178 opened by serhiy.storchaka #40179: Argument Clinic incorretly translates #elif https://bugs.python.org/issue40179 opened by serhiy.storchaka #40180: isinstance(cls_with_metaclass, non_type) raises KeyError https://bugs.python.org/issue40180 opened by terry.reedy #40181: IDLE: remove positional-only note from calltips https://bugs.python.org/issue40181 opened by terry.reedy #40183: AC_COMPILE_IFELSE doesn't work in all cases https://bugs.python.org/issue40183 opened by jerome.hamm #40186: test_notify_all hangs forever in sparc64 https://bugs.python.org/issue40186 opened by BTaskaya #40188: Azure Pipelines jobs failing randomly with: Unable to connect https://bugs.python.org/issue40188 opened by vstinner #40191: tempfile.mkstemp() | Documentation Error https://bugs.python.org/issue40191 opened by Howard Waterfall #40192: time.thread_time isn't outputting in nanoseconds in AIX https://bugs.python.org/issue40192 opened by BTaskaya #40195: multiprocessing.Queue.put can fail silently due to pickle erro https://bugs.python.org/issue40195 opened by Sander Land #40197: Add nanoseconds to timing table in What's new python 3.8 https://bugs.python.org/issue40197 opened by mchels #40198: macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH d https://bugs.python.org/issue40198 opened by dgelessus #40199: Invalid escape sequence DeprecationWarnings don't trigger by d https://bugs.python.org/issue40199 opened by Numerlor #40202: Misleading grammatically of ValueError Message? https://bugs.python.org/issue40202 opened by Jacob RR #40203: Warn about invalid PYTHONUSERBASE https://bugs.python.org/issue40203 opened by Volker Weißmann #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 opened by xtreak #40205: Profile 'builtins' parameter documentation missing https://bugs.python.org/issue40205 opened by bar.harel #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 opened by BTaskaya #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 opened by BTaskaya #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 opened by hakancelik #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 opened by Nikolai Ehrhardt #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 opened by BTaskaya #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 opened by BTaskaya #40213: contextlib.aclosing() https://bugs.python.org/issue40213 opened by John Belmonte #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 opened by aeros #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 opened by Óscar García Amor #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 opened by vstinner #40218: sys.executable is a non existing path if python is executed fr https://bugs.python.org/issue40218 opened by Volker Weißmann #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 opened by Stephen Bell #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 opened by vstinner #40222: "Zero cost" exception handling https://bugs.python.org/issue40222 opened by Mark.Shannon #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 opened by skrah #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 opened by brendon-zhang(a)hotmail.com #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 opened by iivanyuk #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 opened by Mark.Shannon #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 opened by Steven Lu #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 opened by Henry Carscadden #40231: Fix pending calls in subinterpreters https://bugs.python.org/issue40231 opened by vstinner #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 opened by vstinner #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 opened by eric.snow #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 opened by dholth #40236: datetime.datetime.strptime get day error https://bugs.python.org/issue40236 opened by zhanying #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 opened by vstinner #40238: os.stat() on Windows succeeds for nonexistent paths with trail https://bugs.python.org/issue40238 opened by laloch #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 opened by Eric Cousineau #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 opened by vstinner #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 opened by wmeehan #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 opened by Michael.Felt #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 opened by 0az Most recent 15 issues with no replies (15) ========================================== #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 Most recent 15 issues waiting for review (15) ============================================= #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 Top 10 most discussed issues (10) ================================= #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 16 msgs #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 16 msgs #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 13 msgs #39481: Implement PEP 585 (Type Hinting Generics In Standard Collectio https://bugs.python.org/issue39481 12 msgs #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 12 msgs #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 11 msgs #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 9 msgs #40060: socket.TCP_NOTSENT_LOWAT is missing in official macOS builds https://bugs.python.org/issue40060 8 msgs #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 7 msgs #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 7 msgs Issues closed (36) ================== #5654: Add C hook in PyDict_SetItem for debuggers https://bugs.python.org/issue5654 closed by vstinner #25780: Add support for CAN_RAW_JOIN_FILTERS https://bugs.python.org/issue25780 closed by pablogsal #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 closed by vstinner #37127: Handling pending calls during runtime finalization may cause p https://bugs.python.org/issue37127 closed by vstinner #38501: multiprocessing.Pool hangs atexit (and garbage collection some https://bugs.python.org/issue38501 closed by pablogsal #38689: IDLE crashes when KeyError is raised during calltip generation https://bugs.python.org/issue38689 closed by terry.reedy #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 closed by BTaskaya #39689: struct and memoryview tests rely on undefined behavior (as rev https://bugs.python.org/issue39689 closed by petr.viktorin #40082: trip_signal() gets NULL tstate on Windows on CTRL+C https://bugs.python.org/issue40082 closed by vstinner #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 closed by ammar2 #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 closed by terry.reedy #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 closed by pablogsal #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 closed by SilentGhost #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 closed by steve.dower #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 closed by benjamin.peterson #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 closed by nergall2 #40177: Python Language Reference Documentation https://bugs.python.org/issue40177 closed by mark.dickinson #40182: Remove the _field_types attribute of NamedTuple https://bugs.python.org/issue40182 closed by serhiy.storchaka #40184: Compiler warnings under sparc64 https://bugs.python.org/issue40184 closed by benjamin.peterson #40185: Refactor typing.NamedTuple https://bugs.python.org/issue40185 closed by serhiy.storchaka #40187: Refactor typing.TypedDict https://bugs.python.org/issue40187 closed by serhiy.storchaka #40189: MultiProcessing Subclass - overrides run method issue https://bugs.python.org/issue40189 closed by Python_dev_IL #40190: Add support for _SC_AIX_REALMEM in posix.sysconf https://bugs.python.org/issue40190 closed by vstinner #40193: thread.get_native_id() support for solaris https://bugs.python.org/issue40193 closed by BTaskaya #40194: Document special meanings of a single underscore https://bugs.python.org/issue40194 closed by SilentGhost #40196: symtable.Symbol.is_local() can be True for global symbols https://bugs.python.org/issue40196 closed by pablogsal #40200: count 0 crash https://bugs.python.org/issue40200 closed by BTaskaya #40201: Last digit count error https://bugs.python.org/issue40201 closed by josh.r #40206: Multiplying 4.6*100 will result in 459.99999999999994 https://bugs.python.org/issue40206 closed by eric.smith #40216: Support --runstatedir in configure https://bugs.python.org/issue40216 closed by vstinner #40220: Modules/gcmodule.c:434: update_refs: Assertion "gc_get_refs(gc https://bugs.python.org/issue40220 closed by gregory.p.smith #40224: delete_me https://bugs.python.org/issue40224 closed by ronaldoussoren #40226: Leak in tstate->interp->ceval.pending https://bugs.python.org/issue40226 closed by vstinner #40233: Awkward to set compression on writeable ZipFile.open() https://bugs.python.org/issue40233 closed by dholth #40239: Add a function for merging sorted iterables https://bugs.python.org/issue40239 closed by serhiy.storchaka #40242: zmq mysql core dump https://bugs.python.org/issue40242 closed by eric.smith

1 0

Need help with test_ctypes failing on Windows (test_load_dll_with_flags)
by Guido van Rossum 07 Apr '20

07 Apr '20

I have a large PR (https://github.com/python/cpython/pull/18239, for PEP 585) that's failing in the Azures pipeline on Win32 and Win64 only. My trusty assistant who has a Windows laptop couldn't reproduce the failure. Can I buy a hint from someone? Steve? The relevant failure output is: ====================================================================== ERROR: test_load_dll_with_flags (ctypes.test.test_loading.LoaderTest) [WinDLL('_sqlite3.dll', winmode=0)] ---------------------------------------------------------------------- Traceback (most recent call last): File "d:\a\1\s\lib\ctypes\test\test_loading.py", line 140, in should_pass subprocess.check_output( File "d:\a\1\s\lib\subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "d:\a\1\s\lib\subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['d:\\a\\1\\s\\PCbuild\\win32\\python.exe', '-c', "from ctypes import *; import nt;WinDLL('_sqlite3.dll', winmode=0)"]' returned non-zero exit status 1. ---------------------------------------------------------------------- -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

4 5

Problem with instantiating a C extension class from another class
by Musbur 07 Apr '20

07 Apr '20

Hello, (I've asked this question before on python-list but only got scarce and ultimately unhelpful answers. Although this isn't about Python development itself, I'm assuming that there is more CPython knowledge on this list than on the other, so please bear with me.) I've written a C extension module, each defining a class (called Series and CLHist, respectively). I can import both modules in Python and use the classes. Everything works fine, also according to valgrind. Now I want to return a Series instance directly from a CLHist method, and that immediately crashes Python with a segfault. Here's a bit of code and just the _new, _init, and _finalize methods which I've sprinkled with debugging code. When I instantiate a Series object from Python, I see output from all three methods. When I call pyseries_new() from C, none of the three functions are called and it crashes. At some point I thought maybe I should call PyType_Ready() from the C side before instantiating Series. In a "standard" (system) Python installation, it printed the desired debugging output and crashed later. Under a debugging Python installation I built it outputs nothing except an interesting post-mortem "Fatal Python error: UNREF invalid object" (see below). Here's my C code with the boilerplate stuff left out. typedef struct { PyObject_HEAD struct clh_series *series; } Series; static PyObject *Series_new(PyTypeObject *type, PyObject *args, PyObject *kw) { Series *self; self = (Series *) type->tp_alloc(type, 0); fprintf(stderr, "Series_new(%p)\n", self); self->series = NULL; return (PyObject*)self; } static int Series_init(Series *self, PyObject *args, PyObject *kw) { fprintf(stderr, "Series_init(%p)\n", self); self->series = NULL; return 0; } static void Series_finalize(PyObject *self) { fprintf(stderr, "Series_finalize(%p)\n", self); clh_series_free(((Series*)self)->series); } /* To create a new Series object directly from C */ PyObject *pyseries_new(struct clh_series *series) { Series *pyseries; pyseries = PyObject_New(Series, &series_type); PyObject_Init((PyObject *)pyseries, &series_type); pyseries->series = series; return (PyObject *) pyseries; } Here's the debugging output: * ob object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 * op->_ob_prev->_ob_next object : <_Series object at 0x7f1a4e2ee9d0> type : _Series refcount: 1 address : 0x7f1a4e2ee9d0 * op->_ob_next->_ob_prev object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 Fatal Python error: UNREF invalid object

3 3