Mailman 3 February 2022 - Python-Dev

Version 2 of PEP 670 – Convert macros to functions in the Python C API
by Victor Stinner Feb. 24, 2022

Feb. 24, 2022

Hi, Since Erlend and me posted PEP 670 on python-dev last October, we took all feedback (python-dev and Steering Council) in account to clarify the intent of the PEP and to reduce its scope (remove *any* risk of backward compatibility). PEP 670: https://python.github.io/peps/pep-0670/ Changes since the first version: * Stricter policy on not changing argument types and return type. * Better explain why pointer arguments require a cast to not emit new compiler warnings. * Macros which can be used as l-values are no longer modified by the PEP (I wrote PEP 674 just for that). * Macros having multiple return types are no longer modified by the PEP. * Limited C API version 3.11 no longer casts pointer arguments (move towards a cleaner API, but only if you opt in for that!). * No longer remove return values of macros "which should not have a return value" (this rule was complicated and in fact, all macros have already been fixed). * Add "Macros converted to functions since Python 3.8" section to show that macros already converted didn't introduce performance issue nor new compiler warnings. * Add "Benchmark comparing macros and static inline functions" section to prove that there is no significant impact on performance. Note: we already submitted PEP 670 to the Steering Council at the end of November, and the SC 2022 came with feedback, so we modified the PEP: https://mail.python.org/archives/list/python-dev@python.org/message/IJ3IBVY… Following the SC decision, we already modified PEP 7 to add: "static inline functions should be preferred over macros in new code." https://www.python.org/dev/peps/pep-0007/#c-dialect Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 4

SC reply to PEP 674 -- Disallow using macros as l-values
by Petr Viktorin Feb. 23, 2022

Feb. 23, 2022

Hello, Thank you for submitting PEP 674 -- Disallow using macros as l-values! For Py_TYPE, we respect the previous steering council's decision: https://github.com/python/steering-council/issues/79 The Py_TYPE macro can be changed as proposed in 3.11. However, the old SC explicitly noted that the exception only applies to Py_SET_TYPE, and that in general, there should be a deprecation notice in some way or form. After deliberating the proposed changes, the SC agrees that they will make CPython better. However, we are not convinced that they should be made *right now*. We see no reason to rush. We would like to ask you to rework the PEP to reflect these suggestions. It is not feasible to provide visible deprecation warnings for the proposed changes. We feel that this is not a good reason to skip the deprecation process entirely. Quite the opposite: if we need to skip a step, we should move even more carefully than usual. We suggest the following process: First, add notices to any documentation that using the macro as a l-value is a deprecated, CPython-specific detail. It should be clear that the only reason this usage is allowed is for backwards compatibility, and that alternate implementations of the C API are free to not allow this. Then, wait until all supported versions of CPython include any relevant replacement macros such as Py_SET_SIZE. (This doesn't mean adding "set" variants to all the changed macros -- just to those few where setting can be useful and safe.) With the current schedule, this means waiting until Python 3.14, which is planned for October 2024, when Python 3.8 will reach end-of-life. The SC can give an exception to shorten the wait for specific macros, if there is a real benefit to it. (For example: the nogil branch should include any necessary changes, and they should be approved and merged with nogil itself). Finally, ask the new Steering Council to make the change and finally disallow using the macros as l-values. If there's no negative feedback or unexpected downsides, we hope that the future SC will summarily approve. Hopefully, this strategy will also work for PyDescr_NAME and other macros that were left out of the PEP for now. Additionally, consider making the change in the 3.11 version of the limited API. This version is opt-in, so users are free to do the switch when it is convenient for them. Also, it would allow early testing and review of the changes. Further discussion about this should happen on python-dev, like usual. - Petr, on behalf of the Steering Council

2 3

PEP 683: "Immortal Objects, Using a Fixed Refcount"
by Eric Snow Feb. 23, 2022

Feb. 23, 2022

Eddie and I would appreciate your feedback on this proposal to support treating some objects as "immortal". The fundamental characteristic of the approach is that we would provide stronger guarantees about immutability for some objects. A few things to note: * this is essentially an internal-only change: there are no user-facing changes (aside from affecting any 3rd party code that directly relies on specific refcounts) * the naive implementation shows a 4% slowdown * we have a number of strategies that should reduce that penalty * without immortal objects, the implementation for per-interpreter GIL will require a number of non-trivial workarounds That last one is particularly meaningful to me since it means we would definitely miss the 3.11 feature freeze. With immortal objects, 3.11 would still be in reach. -eric ----------------------- PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: Resolution: Abstract ======== Under this proposal, any object may be marked as immortal. "Immortal" means the object will never be cleaned up (at least until runtime finalization). Specifically, the `refcount`_ for an immortal object is set to a sentinel value, and that refcount is never changed by ``Py_INCREF()``, ``Py_DECREF()``, or ``Py_SET_REFCNT()``. For immortal containers, the ``PyGC_Head`` is never changed by the garbage collector. Avoiding changes to the refcount is an essential part of this proposal. For what we call "immutable" objects, it makes them truly immutable. As described further below, this allows us to avoid performance penalties in scenarios that would otherwise be prohibitive. This proposal is CPython-specific and, effectively, describes internal implementation details. .. _refcount: https://docs.python.org/3.11/c-api/intro.html#reference-counts Motivation ========== Without immortal objects, all objects are effectively mutable. That includes "immutable" objects like ``None`` and ``str`` instances. This is because every object's refcount is frequently modified as it is used during execution. In addition, for containers the runtime may modify the object's ``PyGC_Head``. These runtime-internal state currently prevent full immutability. This has a concrete impact on active projects in the Python community. Below we describe several ways in which refcount modification has a real negative effect on those projects. None of that would happen for objects that are truly immutable. Reducing Cache Invalidation --------------------------- Every modification of a refcount causes the corresponding cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. This is also true even with the GIL, though the impact is smaller. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple threads that concurrently incref or decref. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object, including the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and has led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Rationale ========= The proposed solution is obvious enough that two people came to the same conclusion (and implementation, more or less) independently. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the two examples above stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. Several promising mitigation strategies will be pursued in the effort to bring it closer to performance-neutral. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ----------------------- This proposal is completely compatible. It is internal-only so no API is changing. The approach is also compatible with extensions compiled to the stable ABI. Unfortunately, they will modify the refcount and invalidate all the performance benefits of immortal objects. However, the high bit of the refcount will still match ``_Py_IMMORTAL_REFCNT`` so we can still identify such objects as immortal. No user-facing behavior changes, with the following exceptions: * code that inspects the refcount (e.g. ``sys.getrefcount()`` or directly via ``ob_refcnt``) will see a really, really large value * ``Py_SET_REFCNT()`` will be a no-op for immortal objects Neither should cause a problem. Alternate Python Implementations -------------------------------- This proposal is CPython-specific. Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Non-Obvious Consequences ------------------------ * immortal containers effectively immortalize each contained item * the same is true for objects held internally by other objects (e.g. ``PyTypeObject.tp_subclasses``) * an immortal object's type is effectively immortal * though extremely unlikely (and technically hard), any object could be incref'ed enough to reach ``_Py_IMMORTAL_REFCNT`` and then be treated as immortal Specification ============= The approach involves these fundamental changes: * add ``_Py_IMMORTAL_REFCNT`` (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal containers * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. To be clear, we will likely use the most-significant bit of ``_Py_IMMORTAL_REFCNT`` to tell if an object is immortal, rather than comparing with ``_Py_IMMORTAL_REFCNT`` directly. (There are other minor, internal changes which are not described here.) This is not meant to be a public feature but rather an internal one. So the proposal does *not* including adding any new public C-API, nor any Python API. However, this does not prevent us from adding (publicly accessible) private API to do things like immortalize an object or tell if one is immortal. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Immortal Global Objects ----------------------- The following objects will be made immortal: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) There will likely be others we have not enumerated here. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For container objects we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by pylifecycle.c:finalize_modules which cleans up the remaining modules as best as we can. It will change which modules are available during __del__ but that's already defined as undefined behavior by the docs. Optionally, we could do some topological disorder to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything leftover (if any) can be found through the permanent generation gc list which we can clear after finalize_modules. For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. Documentation ------------- The feature itself is internal and will not be added to the documentation. We *may* add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Rejected Ideas ============== Equate Immortal with Immutable ------------------------------ Making a mutable object immortal isn't particularly helpful. The exception is if you can ensure the object isn't actually modified again. Since we aren't enforcing any immutability for immortal objects it didn't make sense to emphasis that relationship. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * is there any other impact on GC? References ========== This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

11 27

PyCon 2022: Call For Papers!
by Grete Kungla Feb. 22, 2022

Feb. 22, 2022

Hi, I am working with the PyCon Estonia team, helping them organise a conference this year. The last three PyCon events were a great success and after a year of break due to corona, we want to come back to keep on bringing the Python community together in Tallinn. As we are preparing for PyCon Estonia 2020, we have also announced our call for papers. We are looking for talks on various topics. The theme this year is "Connect. Collab. Contribute" and we want the focus of the conference to be around the Python ecosystem. However, we are also welcoming talks on other relevant topics - Python challenges, outstanding packages, security, automation, AI/ML, web development, IoT, mental health for developers, conducive work environment for coding, etc. I would like to invite you to participate in the call for papers and be a part of PyCon Estonia 2022 as a speaker. We are waiting for the proposals before the 1st of April here: https://pyconestonia.typeform.com/to/HSWhN8YW We would have a team from our end review the proposals anonymously and reach out to you if the talk fits with the overall theme. As a speaker, you do not need to purchase the tickets for the event and if you already have, we'll be happy to make a refund or transfer. If you have any questions please do not hesitate to reach out to me. Looking forward to having you with us at PyCon on 25th of August, 2022. Best Wishes, -- Grete Kungla +372 5628 0032 Project Manager PyCon Conference 2022 https://pycon.ee/

1 0

SC Acceptance: PEP 680 -- tomllib
by Petr Viktorin Feb. 22, 2022

Feb. 22, 2022

Hello, The Steering Council discussed PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library, and unanimously decided to accept it. Congratulations! I'll change the PEP status to Accepted. Please open a PR for Python 3.11 at your convenience. - Petr (on behalf of the Python Steering Council)

1 0

Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
by Petr Viktorin Feb. 21, 2022

Feb. 21, 2022

Thank you for submitting PEP 670 (Convert macros to functions in the Python C API)! The steering council is still discussing it. We're not ready to accept the PEP as a whole, but there are parts on which we agree unanimously. To unblock the work, we're making the following pronouncements: *All other things being equal, static inline functions are better than macros.* Specifically, inline static functions should be preferred over function-like macros in new code. Please add a note about this to PEP 7. *The performance impact of changing macros to static inline functions is negligible.* The performance changes are indistinguishable from noise caused by other factors, such as layout changes and specific compiler optimizations. It's possible that this isn't true in all cases. If any are found, they should be kept as macros and added to the benchmark suite to avoid regressions. More generally, if there is a non-obvious reason for a conscious design decision, it should be explained (e.g. in a comment) to help future developers. *When there is no backwards compatibility issue, macros can be changed to static inline functions.* In effect, the SC accepts the first part of the PEP, except cases where: a macro is converted to macro and function ("Cast to PyObject*"), and where the return value is removed. Please do the changes in several smaller pull requests, don't skip reviews, and be prepared to revert if any issues are found. The straightforward macros should be converted first. When they are, we hope that it will become easier to evaluate the remaining ones. - Petr, on behalf of the Steering Council

7 11

Move the pythoncapi_compat project under the GitHub Python or PSF organization?
by Victor Stinner Feb. 21, 2022

Feb. 21, 2022

Hi, Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable? "The pythoncapi_compat project can be used to write a C extension supporting a wide range of Python versions with a single code base. It is made of the pythoncapi_compat.h header file and the upgrade_pythoncapi.py script." Documentation: https://pythoncapi-compat.readthedocs.io/en/latest/ GitHub: https://github.com/pythoncapi/pythoncapi_compat In the past, I managed to move my personal pyperf project under the PSF organization. Now other core developers are contributing and using it. It's better than having it as a personal project. pythoncapi_compat respects the PSF requirements to move a project in the GitHub PSF organization: contributors are required to respect the PSF Code of Conduct and the project has 3 maintainers (Dong-hee Na, Erlend E. AAsland and me). --- Some context. Incompatible C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions. For example, What's New in Python 3.10 suggests adding the following code to your project to get the Py_SET_REFCNT() function on Python 3.9 and older: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0) #endif Python 3.11 requires even more compatibility functions (7) and the implementation of these functions take more lines of code: * Py_SET_TYPE() * Py_SET_SIZE() * PyFrame_GetCode() * PyFrame_GetBack() * PyThreadState_GetFrame() * PyThreadState_EnterTracing() * PyThreadState_LeaveTracing() I added a reference to the pythoncapi_compat project to What's New in Python 3.11: "Or use the pythoncapi_compat project to get these functions on old Python functions." https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes The project provides an upgrade_pythoncapi.py script to automatically add Python 3.10 and 3.11 support without losing support with Python 2.7. Just run the script, maybe copy pythoncapi_compat.h, and you're done! :-) It also provides a "pythoncapi_compat.h" header file which contains all the required compatibility code. Just copy the header file and use >#include "pythoncapi_compat.h"< in your project. You don't have to maintain this compatibilty code yourself anymore (pythoncapi_compat.h is made of 400 lines of C code). pythoncapi_compat supports Python (2.7 to 3.11), PyPy (2.7, 3.6, 3.7) and C++. It is distributed under the MIT license. My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future. --- The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy: https://pythoncapi-compat.readthedocs.io/en/latest/users.html I already made a similar request in June 2021 (move the project under the PSF organization): https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNO… What changed since June 2021? Python 3.11 now requires way more compatibility code than Python 3.10 does: 7 functions instead of 1. On Discord, I was asked to ask the Steering Council. I asked the SC who asked me to ask on python-dev, so here I am :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

7 15

Github Issues Migration is coming soon
by Łukasz Langa Feb. 18, 2022

Feb. 18, 2022

I just published the migration plan and a call to action on Discourse: https://discuss.python.org/t/github-issues-migration-is-coming-soon/13791 <https://discuss.python.org/t/github-issues-migration-is-coming-soon/13791> -- See you there, Łukasz Langa CPython Developer in Residence Python Software Foundation

1 0

Summary of Python tracker Issues
by Python tracker Feb. 18, 2022

Feb. 18, 2022

ACTIVITY SUMMARY (2022-02-11 - 2022-02-18) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7218 (+40) closed 51299 (+27) total 58517 (+67) Open issues with patches: 2932 Issues opened (52) ================== #46725: Unpacking without parentheses is allowed since 3.9 https://bugs.python.org/issue46725 opened by pablogsal #46726: Thread spuriously marked dead after interrupting a join call https://bugs.python.org/issue46726 opened by Kevin Shweh #46727: Should shutil functions support bytes paths? https://bugs.python.org/issue46727 opened by Jelle Zijlstra #46729: Better str() for BaseExceptionGroup https://bugs.python.org/issue46729 opened by iritkatriel #46731: posix._fcopyfile flags addition https://bugs.python.org/issue46731 opened by devnexen #46732: object.__bool__ docstring is wrong https://bugs.python.org/issue46732 opened by Jelle Zijlstra #46733: pathlib.Path methods can raise NotImplementedError https://bugs.python.org/issue46733 opened by barneygale #46734: Add Maildir.get_flags() to access message flags without openin https://bugs.python.org/issue46734 opened by gildea #46735: gettext.translations crashes when locale is unset https://bugs.python.org/issue46735 opened by amazingminecrafter2015 #46736: Generate HTML 5 with SimpleHTTPRequestHandler.list_directory https://bugs.python.org/issue46736 opened by dom1310df #46740: Improve Telnetlib's throughput https://bugs.python.org/issue46740 opened by martin_kirch #46742: Add '-d $fd' option to trace module, akin to bash -x feature https://bugs.python.org/issue46742 opened by PenelopeFudd #46743: Enable usage of object.__orig_class__ in __init__ https://bugs.python.org/issue46743 opened by Gobot1234 #46744: installers on ARM64 suggest wrong folders https://bugs.python.org/issue46744 opened by conio #46746: IDLE: Consistently handle non .py source files https://bugs.python.org/issue46746 opened by terry.reedy #46748: Python.h includes stdbool.h https://bugs.python.org/issue46748 opened by petr.viktorin #46749: Support cross compilation on macOS https://bugs.python.org/issue46749 opened by autoantwort #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 opened by slingamn #46751: Windows-style path is not recognized under cygwin https://bugs.python.org/issue46751 opened by mikekaganski #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 opened by gvanrossum #46753: Statically allocate and initialize the empty tuple. https://bugs.python.org/issue46753 opened by eric.snow #46754: Improve Python Language Reference based on [K��hl 2020] https://bugs.python.org/issue46754 opened by gvanrossum #46755: QueueHandler logs stack_info twice https://bugs.python.org/issue46755 opened by erik.montnemery #46756: Incorrect authorization check in urllib.request https://bugs.python.org/issue46756 opened by serhiy.storchaka #46757: dataclasses should define an empty __post_init__ https://bugs.python.org/issue46757 opened by NeilGirdhar #46758: Incorrect behaviour creating a Structure with ctypes.c_bool bi https://bugs.python.org/issue46758 opened by dudenwatschn #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 opened by cjwatson #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 opened by Mark.Shannon #46761: functools.update_wrapper breaks the signature of functools.par https://bugs.python.org/issue46761 opened by larry #46763: os.path.samefile incorrect results for shadow copies https://bugs.python.org/issue46763 opened by nijave #46764: Wrapping a bound method with a @classmethod no longer works https://bugs.python.org/issue46764 opened by msullivan #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 opened by eric.snow #46767: [Doc] sqlite3 Cursor.execute() return value is unspecified https://bugs.python.org/issue46767 opened by kephas #46769: Improve documentation for `typing.TypeVar` https://bugs.python.org/issue46769 opened by AlexWaygood #46770: ConfigParser(dict_type=) not behaving as expected https://bugs.python.org/issue46770 opened by malonn #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 opened by gvanrossum #46772: Statically Initialize PyArg_Parser in clinic.py https://bugs.python.org/issue46772 opened by eric.snow #46773: Add a Private API for Looking Up Global Objects https://bugs.python.org/issue46773 opened by eric.snow #46774: Importlib.metadata.version picks first distribution not latest https://bugs.python.org/issue46774 opened by kkirsche-github #46775: [Windows] OSError should unconditionally call winerror_to_errn https://bugs.python.org/issue46775 opened by eryksun #46777: Fix incorrect use of directives in asyncio documentation https://bugs.python.org/issue46777 opened by serhiy.storchaka #46779: Add ssl.CERT_REQUIRED_NO_VERIFY as possible value for ssl.SSLC https://bugs.python.org/issue46779 opened by freundTech #46781: Tracing: c_return doesn't report the result https://bugs.python.org/issue46781 opened by smurfix #46782: Docs error for 3.10 https://bugs.python.org/issue46782 opened by usetohandletrush #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 opened by yilei #46785: On Windows, os.stat() can fail if called while another process https://bugs.python.org/issue46785 opened by Antony.Lee #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 opened by jnns #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 opened by themylogin #46788: regrtest fails to start on missing performance counter names https://bugs.python.org/issue46788 opened by jkloth #46789: Restore caching of externals on Windows buildbots https://bugs.python.org/issue46789 opened by jkloth #46790: Normalize handling of negative timeouts in subprocess.py https://bugs.python.org/issue46790 opened by jkloth #46791: Allow os.remove to defer to rmdir https://bugs.python.org/issue46791 opened by bup Most recent 15 issues with no replies (15) ========================================== #46791: Allow os.remove to defer to rmdir https://bugs.python.org/issue46791 #46788: regrtest fails to start on missing performance counter names https://bugs.python.org/issue46788 #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 #46782: Docs error for 3.10 https://bugs.python.org/issue46782 #46779: Add ssl.CERT_REQUIRED_NO_VERIFY as possible value for ssl.SSLC https://bugs.python.org/issue46779 #46775: [Windows] OSError should unconditionally call winerror_to_errn https://bugs.python.org/issue46775 #46774: Importlib.metadata.version picks first distribution not latest https://bugs.python.org/issue46774 #46773: Add a Private API for Looking Up Global Objects https://bugs.python.org/issue46773 #46772: Statically Initialize PyArg_Parser in clinic.py https://bugs.python.org/issue46772 #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 #46761: functools.update_wrapper breaks the signature of functools.par https://bugs.python.org/issue46761 #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 Most recent 15 issues waiting for review (15) ============================================= #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 #46781: Tracing: c_return doesn't report the result https://bugs.python.org/issue46781 #46777: Fix incorrect use of directives in asyncio documentation https://bugs.python.org/issue46777 #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 #46764: Wrapping a bound method with a @classmethod no longer works https://bugs.python.org/issue46764 #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 #46756: Incorrect authorization check in urllib.request https://bugs.python.org/issue46756 #46755: QueueHandler logs stack_info twice https://bugs.python.org/issue46755 #46753: Statically allocate and initialize the empty tuple. https://bugs.python.org/issue46753 #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 Top 10 most discussed issues (10) ================================= #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 38 msgs #46726: Thread spuriously marked dead after interrupting a join call https://bugs.python.org/issue46726 16 msgs #46541: Replace _Py_IDENTIFIER() with statically initialized objects. https://bugs.python.org/issue46541 11 msgs #45447: Make IDLE recognize .pyi stub files as source for open, save, https://bugs.python.org/issue45447 10 msgs #46743: Enable usage of object.__orig_class__ in __init__ https://bugs.python.org/issue46743 10 msgs #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 8 msgs #46751: Windows-style path is not recognized under cygwin https://bugs.python.org/issue46751 7 msgs #46716: regrtest didn't respect the timeout when running test_subproce https://bugs.python.org/issue46716 6 msgs #46733: pathlib.Path methods can raise NotImplementedError https://bugs.python.org/issue46733 6 msgs #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 6 msgs Issues closed (27) ================== #29418: inspect.isroutine does not return True for some bound builtin https://bugs.python.org/issue29418 closed by BTaskaya #38619: [Doc] UUID.hex is lowercase https://bugs.python.org/issue38619 closed by Mariatta #41086: Exception for uninstantiated interpolation (configparser) https://bugs.python.org/issue41086 closed by lukasz.langa #45948: Unexpected instantiation behavior for xml.etree.ElementTree.XM https://bugs.python.org/issue45948 closed by serhiy.storchaka #46066: Deprecate keyword args syntax for TypedDict definition https://bugs.python.org/issue46066 closed by Jelle Zijlstra #46333: ForwardRef.__eq__ does not respect module parameter https://bugs.python.org/issue46333 closed by Jelle Zijlstra #46436: Pass the -d/--directory command-line option to http.server.CGI https://bugs.python.org/issue46436 closed by eric.araujo #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 closed by erlendaasland #46666: IDLE Add indent guide https://bugs.python.org/issue46666 closed by terry.reedy #46700: wrong nomenclature (options vs. arguments) in argparse https://bugs.python.org/issue46700 closed by rhettinger #46704: Parser API not checking for null-terminator https://bugs.python.org/issue46704 closed by pablogsal #46718: Feature: itertools: add batches https://bugs.python.org/issue46718 closed by rhettinger #46728: Docstring of combinations_with_replacement for consistency https://bugs.python.org/issue46728 closed by corona10 #46730: Please consider mentioning property without setter when an att https://bugs.python.org/issue46730 closed by Dennis Sweeney #46737: Default to the standard normal distribution https://bugs.python.org/issue46737 closed by rhettinger #46738: Allow http.server to emit HTML 5 https://bugs.python.org/issue46738 closed by Jelle Zijlstra #46739: dataclasses __eq__ isn't logical https://bugs.python.org/issue46739 closed by Dennis Sweeney #46741: Docstring for asyncio.protocols.BufferedProtocol appears out o https://bugs.python.org/issue46741 closed by asvetlov #46745: Typo in new PositionsIterator https://bugs.python.org/issue46745 closed by sobolevn #46747: bisect.bisect/insort don't document key parameter https://bugs.python.org/issue46747 closed by Dennis Sweeney #46762: assertion failure in f-string parsing Parser/string_parser.c https://bugs.python.org/issue46762 closed by pablogsal #46766: Add a class for file operations so a syntax such as open("file https://bugs.python.org/issue46766 closed by serhiy.storchaka #46768: CVE-201-4160 Vulnerability Is Found in Lib/site-packages/crypt https://bugs.python.org/issue46768 closed by christian.heimes #46776: RecursionError when using property() inside classes https://bugs.python.org/issue46776 closed by steven.daprano #46778: Enable parallel compilation on Windows builds https://bugs.python.org/issue46778 closed by steve.dower #46780: Allow Fractions to return 1/6 for "0.17", "0.167", "0.1667", e https://bugs.python.org/issue46780 closed by mark.dickinson #46783: Add a new feature to enumerate(iterable, start=0) built-in fun https://bugs.python.org/issue46783 closed by rhettinger

1 0

Compiling of ast.Module in Python 3.10 and co_firstlineno behavior
by Fabio Zadrozny Feb. 18, 2022

Feb. 18, 2022

Hi all, I'm stumbling with an issue where the co_firstlineno behavior changed from Python 3.9 to Python 3.10 and I was wondering if this was intentional or not. i.e.: Whenever a code is compiled in Python 3.10, the `code.co_firstlineno` is now always 1, whereas previously it was equal to the first statement. Also, does anyone know if there is any way to restore the old behavior in Python 3.10? I tried setting the `module.lineno` but it didn't really make any difference... As an example, given the code below: import dis source = ''' print(1) print(2) ''' initial_module = compile(source, '<nofilename>', 'exec', PyCF_ONLY_AST, 1) import sys print(sys.version) for i in range(2): module = Module([initial_module.body[i]], []) module_code = compile(module, '<no filename>', 'exec') print(' --> First lineno:', module_code.co_firstlineno) print(' --> Line starts :', list(lineno for offset, lineno in dis.findlinestarts(module_code))) print('---- dis ---') dis.dis(module_code) I have the following outputs for Pyhon 3.9/Python 3.10: 3.9.6 (default, Jul 30 2021, 11:42:22) [MSC v.1916 64 bit (AMD64)] --> First lineno: 2 --> Line starts : [2] ---- dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (1) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First lineno: 4 --> Line starts : [4] ---- dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] --> First lineno: 1 --> Line starts : [2] ---- dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (1) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First lineno: 1 --> Line starts : [4] ---- dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE Thanks, Fabio

3 5