Mailman 3 February 2022 - Python-Dev

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for … [View More]

2 1

It's now time to deprecate the stdlib urllib module
by Victor Stinner March 31, 2022

March 31, 2022

Hi, I propose to deprecate the urllib module in Python 3.11. It would emit a DeprecationWarning which warn users, so users should consider better alternatives like urllib3 or httpx: well known modules, better maintained, more secure, support HTTP/2 (httpx), etc. I don't propose to schedule its removal. Let's discuss the removal in 1 or 2 years. -- urllib has many abstraction to support a wide range of protocols with "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP authentication, … [View More]HTTP Cookie, etc. A simple HTTP request using Basic Authentication requires 10-20 lines of code, whereas it should be a single line. Users (me included) don't like urllib API which was too complicated for common tasks. -- Unhappy users created multiple better alternatives to the stdlib urllib module. In 2008, the "urllib3" module was created to provide an API designed to be as simple as possible for the most common HTTP and HTTPS requests. Example: req = http.request('GET', 'http://httpbin.org/robots.txt'). In 2011, the "requests" module based on urllib3 was created. In 2013, the "aiohttp" module based on asyncio was created. In 2015, new "httpx" module was created: req = httpx.get('https://www.example.org/') Not only httpx has a regular "synchronous" API (blocking function calls), but it also has an asynchronous API! Sadly, while HTTP/3 is being developed, it seems like in this list, httpx is the only HTTP client library supporting HTTP/2 currently :-( For HTTP/2, I also found the "httplib2" module. For HTTP/3, I found the "http3" and "aioquic" modules. -- Let's come back to urllib: * It's API is too complicated * It doesn't support HTTP/2 nor HTTP/3 * It's barely maintained: there are 121 open issues including 3 security issues! The 3 open security issues: * bpo-33661 open 2018; * bpo-36338 open in 2019; * bpo-45795 open in 2021. Usually, it's bad when you refer to an open security issue by its creation year :-( The urllib module has long history of security vulnerabilities. List of *fixed* vulnerabilities: * 2011 (bpo-11662): https://python-security.readthedocs.io/vuln/urllib-redirect.html * 2017 (bpo-30119): https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html * 2017 (bpo-30500): https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html * 2019 (bpo-35907): https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html * 2019 (bpo-38826): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html * 2021 (bpo-42967): https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-s… * 2021 (bpo-43075): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html * 2021 (bpo-44022): https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html urllib is a package made of 4 parts: * urllib.request for opening and reading URLs * urllib.error containing the exceptions raised by urllib.request * urllib.parse for parsing URLs * urllib.robotparser for parsing robots.txt files I propose to deprecate all of them. Maybe the deprecation can be different for each sub-module? Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

39 75

A memory map based data persistence and startup speedup approach
by Yichen Yan March 20, 2022

March 20, 2022

Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. Currently, we’ve made it a third-party library and have been working on open-… [View More]sourcing. Our implementation (whose non-official name is “pycds”) mainly contains two parts: importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. After pycds has been installed, complete workflow of our approach includes three parts: Record name of imported packages to heap.lst, `PYCDSMODE=TRACE PYCDSLIST=heap.lst python run.py` Dump memory archive of code objects of imported packages, this step does not involve the python script, `PYCDSMODE=DUMP PYCDSLIST=heap.lst PYCDSARCHIVE=heap.img python` Run other python processes with created archive, `PYCDSMODE=SHARE PYCDSARCHIVE=heap.img python run.py` We could even make use of immortal objects if PEP 683 [2] was accepted, which could give CDS more performance improvements. Currently, any archived object is virtually immortal, we add rc by 1 to who has been copied to the archive to avoid being deallocated. However, without changes to CPython, rc fields of archived objects will still be updated, therefore have extra footprint due to CoW. More background and detailed implementation could be found at [1]. We think it could be an effective way to improve python’s startup performance, and could even do more like sharing large data between python instances. As suggested in python-ideas [3], we posted this here, looking for questions/suggestions to the overall design and workflow, we also welcome code reviews after we get our lawyers happy and can publish the code. Best, Yichen Yan Alibaba Compiler Group [1] “Faster startup -- Share code objects from memory-mapped file”, https://github.com/faster-cpython/ideas/discussions/150 [2] PEP 683: "Immortal Objects, Using a Fixed Refcount" (draft), https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… [3] [Python-ideas] "A memory map based data persistence and startup speedup approach", https://mail.python.org/archives/list/python-ideas@python.org/thread/UKEBNH… [View Less]

2 1

RFC on PEP 655: Required[] and NotRequired[] for TypedDict
by David Foster March 10, 2022

March 10, 2022

This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as … [View More]

4 6

PEP 682: Format Specifier for Signed Zero
by John Belmonte March 6, 2022

March 6, 2022

Just a note that this PEP is being discussed at https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero TLDR; When formatting a number (especially with rounding options), often you don't want negative-signed zero as an output. A format spec extension enables normalizing such values to positive zero. Regards, --John Belmonte

2 1

PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 2)
by Eric Snow Feb. 28, 2022

Feb. 28, 2022

Thanks to all those that provided feedback. I've worked to substantially update the PEP in response. The text is included below. Further feedback is appreciated. -eric ------------------------ PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCX… Status: Draft Type: Standards Track … [View More]Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022 Resolution: Abstract ======== Currently the CPython runtime maintains a `small amount of mutable state <Runtime Object State_>`_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. The solution proposed here provides a way to mark an object as one for which that per-object runtime state should not change. Specifically, if an object's refcount matches a very specific value (defined below) then that object is treated as "immortal". If an object is immortal then its refcount will never be modified by ``Py_INCREF()``, etc. Consequently, the refcount will never reach 0, so that object will never be cleaned up (unless explicitly done, e.g. during runtime finalization). Additionally, all other per-object runtime state for an immortal object will be considered immutable. This approach has some possible negative impact, which is explained below, along with mitigations. A critical requirement for this change is that the performance regression be no more than 2-3%. Anything worse the performance-neutral requires that the other benefits are proportionally large. Aside from specific applications, the fundamental improvement here is that now an object can be truly immutable. (This proposal is meant to be CPython-specific and to affect only internal implementation details. There are some slight exceptions to that which are explained below. See `Backward Compatibility`_, `Public Refcount Details`_, and `scope`_.) Motivation ========== As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation ------------------------------- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. CPython's GIL helps reduce this effect, since only one thread runs at a time, but it doesn't completely eliminate the penalty. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and has led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Anything that uses ``mmap`` relies on copy-on-write, including sharing data from shared objects files between processes. Rationale ========= The proposed solution is obvious enough that both of this proposal's authors came to the same conclusion (and implementation, more or less) independently. The Pyston project `uses a similar approach <Pyston_>`_. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the two examples above stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. Several promising mitigation strategies will be pursued in the effort to bring it closer to performance-neutral. See the `mitigation`_ section below. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ---------------------- This proposal is meant to be completely compatible. It focuses strictly on internal implementation details. It does not involve changes to any public API, other a few minor changes in behavior related to refcounts (but only for immortal objects): * code that inspects the refcount will see a really, really large value * the new noop behavior may break code that: * depends specifically on the refcount to always increment or decrement (or have a specific value from ``Py_SET_REFCNT()``) * relies on any specific refcount value, other than 0 * directly manipulates the refcount to store extra information there Again, those changes in behavior only apply to immortal objects, not most of the objects a user will access. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by `Pyston`_ shouldn't apply here since we do not modify the refcount. See `Public Refcount Details`_ and `scope`_ below for further discussion. Stable ABI ---------- The approach is also compatible with extensions compiled to the stable ABI. Unfortunately, they will modify the refcount and invalidate all the performance benefits of immortal objects. However, the high bit of the refcount `will still match _Py_IMMORTAL_REFCNT <_Py_IMMORTAL_REFCNT_>`_ so we can still identify such objects as immortal. At worst, objects in that situation would feel the effects described in the `Motivation`_ section. Even then the overall impact is unlikely to be significant. Also see `_Py_IMMORTAL_REFCNT`_ below. Accidental Immortality ---------------------- Hypothetically, a regular object could be incref'ed so much that it reaches the magic value needed to be considered immortal. That means it would accidentally never be cleaned up (by going back to 0). While it isn't impossible, this accidental scenario is so unlikely that we need not worry. Even if done deliberately by using ``Py_INCREF()`` in a tight loop and each iteration only took 1 CPU cycle, it would take 2^61 cycles (on a 64-bit processor). At a fast 5 GHz that would still take nearly 500,000,000 seconds (over 5,000 days)! If that CPU were 32-bit then it is (technically) more possible though still highly unlikely. Also note that it is doubly unlikely to be a problem because it wouldn't matter until the refcount got back to 0 and the object was cleaned up. So any object that hit that magic "immortal" refcount value would have to be decref'ed that many times again before the change in behavior would be noticed. Again, the only realistic way that the magic refcount would be reached (and then reversed) is if it were done deliberately. (Of course, the same thing could be done efficiently using ``Py_SET_REFCNT()`` though that would be even less of an accident.) At that point we don't consider it a concern of this proposal. Alternate Python Implementations -------------------------------- This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in `Backward Compatibility`_ above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted). Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Specification ============= The approach involves these fundamental changes: * add `_Py_IMMORTAL_REFCNT`_ (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal GC objects ("containers") * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. (There are other minor, internal changes which are not described here.) In the following sub-sections we dive into the details. First we will cover some conceptual topics, followed by more concrete aspects like specific affected APIs. Public Refcount Details ----------------------- In `Backward Compatibility`_ we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general). Between the names and the docs, we can clearly see answers to the following questions: * what behavior do users expect? * what guarantees do we make? * do we indicate how to interpret the refcount value they receive? * what are the use cases under which a user would set an object's refcount to a specific value? * are users setting the refcount of objects they did not create? As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount value with any meaning is 0. All other values are considered "not 0". This information will be clarified in the `documentation <Documentation_>`_. Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following: * ``Py_INCREF()`` -> ``Py_ACQUIRE_REF()`` (or only support ``Py_NewRef()``) * ``Py_DECREF()`` -> ``Py_RELEASE_REF()`` * ``Py_REFCNT()`` -> ``Py_HAS_REFS()`` * ``Py_SET_REFCNT()`` -> ``Py_RESET_REFS()`` and ``Py_SET_NO_REFS()`` However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change. Constraints ----------- * ensure that otherwise immutable objects can be truly immutable * minimize performance penalty for normal Python use cases * be careful when immortalizing objects that we don't actually expect to persist until runtime finalization. * be careful when immortalizing objects that are not otherwise immutable .. _scope: Scope of Changes ---------------- Object immortality is not meant to be a public feature but rather an internal one. So the proposal does *not* including adding any new public C-API, nor any Python API. However, this does not prevent us from adding (publicly accessible) private API to do things like immortalize an object or tell if one is immortal. The particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only Cpython-specific but are also private implementation details that are expected to change in subsequent versions. Immortal Mutable Objects ------------------------ Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application's lifetime. Marking a mutable object as immortal can make sense in some situations. Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won't actually be modified. On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, ``sys.modules`` is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized. By making it immortal, we no longer incur the extra overhead during incref/decref. We explore this idea further in the `mitigation`_ section below. (Note that we are still investigating the impact on GC of immortalizing containers.) Implicitly Immortal Objects --------------------------- If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object's refcount can never reach 0 until the immortal object releases it. Examples: * containers like ``dict`` and ``list`` * objects that hold references internally like ``PyTypeObject.tp_subclasses`` * an object's type (held in ``ob_type``) Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn't a change in behavior. The only difference is that the immortal object (holding the reference) doesn't ever get cleaned up. We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing. Un-Immortalizing Objects ------------------------ This proposal does not include any mechanism for taking an immortal object and returning it to a "normal" condition. Currently there is no need for such an ability. On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we'd set it to the value that it would have been if it hadn't been made immortal. However, that value has long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so. _Py_IMMORTAL_REFCNT ------------------- We will add two internal constants:: #define _Py_IMMORTAL_BIT (1LL << (8 * sizeof(Py_ssize_t) - 4)) #define _Py_IMMORTAL_REFCNT (_Py_IMMORTAL_BIT + (_Py_IMMORTAL_BIT / 2)) The refcount for immortal objects will be set to ``_Py_IMMORTAL_REFCNT``. However, to check if an object is immortal we will compare its refcount against just the bit:: (op->ob_refcnt & _Py_IMMORTAL_BIT) != 0 The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension). Note that top two bits of the refcount are already reserved for other uses. That's why we are using the third top-most bit. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Immortal Global Objects ----------------------- All objects that we expect to be shared globally (between interpreters) will be made immortal. That includes the following: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) All such objects will be immutable. In the case of the static types, they will be effectively immutable. ``PyTypeObject`` has some mutable start (``tp_dict`` and ``tp_subclasses``), but we can work around this by storing that state on ``PyInterpreterState`` instead of on the respective static type object. Then the ``__dict__``, etc. getter will do a lookup on the current interpreter, if appropriate, instead of using ``tp_dict``. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For GC objects ("containers") we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by ``pylifecycle.c:finalize_modules()`` which cleans up the remaining modules as best as we can. It will change which modules are available during __del__ but that's already defined as undefined behavior by the docs. Optionally, we could do some topological disorder to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything leftover (if any) can be found through the permanent generation gc list which we can clear after finalize_modules(). For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. .. _mitigation: Performance Regression Mitigation --------------------------------- In the interest of clarify, here are some of the ways we are going to try to recover some of the lost `performance <Performance_>`_: * at the end of runtime init, mark all objects as immortal * drop refcount operations in code where we know the object is immortal (e.g. ``Py_RETURN_NONE``) * specialize for immortal objects in the eval loop (see `Pyston`_) Regarding that first point, we can apply the concept from `Immortal Mutable Objects`_ in the pursuit of getting back some of that 4% performance we lose with the naive implementation of immortal objects. At the end of runtime init we can mark *all* objects as immortal and avoid the extra cost in incref/decref. We only need to worry about immutability with objects that we plan on sharing between threads without a GIL. Note that none of this section is part of the proposal. The above is included here for clarity. Possible Changes ---------------- * mark every interned string as immortal * mark the "interned" dict as immortal if shared else share all interned strings * (Larry,MvL) mark all constants unmarshalled for a module as immortal * (Larry,MvL) allocate (immutable) immortal objects in their own memory page(s) Documentation ------------- The immortal objects behavior and API are internal, implementation details and will not be added to the documentation. However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically: * ``Py_INCREF()`` - change "Increment the reference count for object o." to "Acquire a new reference to object o." * ``Py_DECREF()`` - change "Decrement the reference count for object o." to "Release a reference to object o." * similar for ``Py_XINCREF()``, ``Py_XDECREF()``, ``Py_NewRef()``, ``Py_XNewRef()``, ``Py_Clear()``, ``Py_REFCNT()``, and ``Py_SET_REFCNT()`` We *may* also add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add such a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * is there any other impact on GC? * `are the copy-on-write benefits real? <https://mail.python.org/archives/list/python-dev@python.org/message/J53GY7X…>`__ * must the fate of this PEP be tied to acceptance of a per-interpreter GIL PEP? References ========== .. _Pyston: https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… Prior Art --------- * `Pyston`_ Discussions ----------- This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Runtime Object State -------------------- Here is the internal state that the CPython runtime keeps for each Python object: * `PyObject.ob_refcnt`_: the object's `refcount <refcounting_>`_ * `_PyGC_Head <PyGC_Head>`_: (optional) the object's node in a list of `"GC" objects <refcounting_>`_ * `_PyObject_HEAD_EXTRA <PyObject_HEAD_EXTRA>`_: (optional) the object's node in the list of heap objects ``ob_refcnt`` is part of the memory allocated for every object. However, ``_PyObject_HEAD_EXTRA`` is allocated only if CPython was built with ``Py_TRACE_REFS`` defined. ``PyGC_Head`` is allocated only if the object's type has ``Py_TPFLAGS_HAVE_GC`` set. Typically this is only container types (e.g. ``list``). Also note that ``PyObject.ob_refcnt`` and ``_PyObject_HEAD_EXTRA`` are part of ``PyObject_HEAD``. .. _PyObject.ob_refcnt: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyGC_Head: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyObject_HEAD_EXTRA: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _refcounting: Reference Counting, with Cyclic Garbage Collection -------------------------------------------------- Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used. Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object. With CPython, code must explicitly take or release references using the C-API's ``Py_INCREF()`` and ``Py_DECREF()``. These macros happen to directly modify the object's refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it's memory is freed). Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers. CPython has dedicated machinery to deal with reference cycles, which we call the "cyclic garbage collector", or often just "garbage collector" or "GC". Don't let the name confuse you. It only deals with breaking reference cycles. See the docs for a more detailed explanation of refcounting and cyclic garbage collection: * https://docs.python.org/3.11/c-api/intro.html#reference-counts * https://docs.python.org/3.11/c-api/refcounting.html * https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt * https://docs.python.org/3.11/c-api/gcsupport.html Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

9 19

Require a C compiler supporting C99 to build Python 3.11
by Victor Stinner Feb. 28, 2022

Feb. 28, 2022

Hi, I made a change to require C99 <math.h> "NAN" constant and I'm was asked to update the PEP 7 to clarify the C subset is needed to build Python 3.11. Python 3.6 requires a subset of the C99 standard to build defined by the PEP 7: https://www.python.org/dev/peps/pep-0007/ I modified Python 3.11 to require more C99 features of the <math.h> header: * bpo-45440: copysign(), hypot(), isfinite(), isinf(), isnan(), round() * bpo-46640: NAN After my NAN change (bpo-46640), Petr … [View More]

16 45

Need Help
by Prasad, PCRaghavendra Feb. 25, 2022

Feb. 25, 2022

Hi All, we are using the python 3.9.5 version in our application. In 3.9.5 it is using libexpat 2.2.8 version, as part of the Black duck scan, it is showing critical vulnerabilities in libexpat 2.2.8. (CVE-2022-22824 CVE-2022-23990 CVE-2022-23852 CVE-2022-25236 CVE-2022-22823) when there are any issues ( security issues ) in external modules like OpenSSL, bzip2, and zlib we were able to get the latest code and build as it is straightforward, but libexpat is an internal module to the python … [View More]

4 4

Summary of Python tracker Issues
by Python tracker Feb. 25, 2022

Feb. 25, 2022

ACTIVITY SUMMARY (2022-02-18 - 2022-02-25) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7220 ( +2) closed 51363 (+64) total 58583 (+66) Open issues with patches: 2939 Issues opened (46) ================== #46793: expose expat XML billion laughs attack mitigation APIs https://bugs.python.org/issue46793 opened by gregory.p.smith #46794: Please … [View More]

1 0

Embedding multiple Python runtimes in the same process on Windows
by Tony Roberts Feb. 24, 2022

Feb. 24, 2022

Hi all, This is specifically about embedding Python on Windows, and I'm hoping some of the Windows Python devs might have some ideas or be interested in this. I have implemented a partial solution (linked below) and I'm interested to hear what other people think of this. Currently when embedding Python both python3x.dll and python3.dll get loaded, with python3.dll redirecting stable API calls to python3x.dll. If a process embeds two different versions of Python 3, e.g., python39.dll and … [View More]python310.dll they both load their respective python3.dlls. At this point I imagine you are thinking "don't do that", but bear with me... The problem comes when the Python loaded second imports an extension module linked with python3.dll. The python3.dll that it gets linked to will be the first one that was loaded, not the one that relates to the second Python that is actually doing the import. This results in a call into the wrong Python dll and 'bad things' happen. None of this is unexpected and I'm sure that the sensible thing to do is to simply not do this... but I've been working on a way to make this work anyway. In my case I have a plugin to another application that embeds Python into that application. It's perfectly possible (and reasonable) for other plugins to also want to embed Python. This can be dealt with by having both plugins use the same Python environment easily enough in most cases. With Python 2 it used to be possible to have two different Python interpreters embedded at the same time and not have them interfere with each other (although it is possible there would still be issues with DLL versions used by extension modules). With Python 3 if we want to have two different versions of Python 3 embedded at the same time then it will fail because of the reason outlined above. My idea is to redirect all loaded python3.dll dlls to the one we want to be used before loading any extension modules (i.e., just before any call to LoadLibraryEx) and then restore them afterwards. This can be done by manipulating the loader modules list in Windows. I have implemented this as a proof of concept in my own plugin and confirmed that this works and does allow two different versions of Python 3 to be embedded at the same time. This works with an unmodified version of Python by applying the redirect in an import hook. It uses several undocumented Windows structures and APIs in order to safely manipulate the loader table. Here is my proof of concept code that performs the redirect https://gist.github.com/tonyroberts/74888762f0063238d4f7fd7c7d36f0f0 While this works for different versions of Python 3, there is still a problem when trying to embed two different instances of the same version of Python 3. The problem is basically the same but with the added complication that the pyd files are named the same, and the ones loaded first get found by the second Python runtime and you end up again calling across versions. I managed to solve this using a similar method to the code above, but rather than redirecting just python3.dll I look for any other loaded python3x.dll and then remove *all* modules loaded under that Python distribution from the loader table. This ensures that both Python runtimes are effectively isolated from each other as neither see any of the same modules. This gets more complicated once you start thinking about user site-packages folders and venvs, but for simple distributions where everything is under the same root folder this technique works. Anyway, just keen to hear what people think or whether this has been tackled before in another way. Best regards, Tony [View Less]

1 0