Mailman 3 April 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to … [View More]prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian [View Less]

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem,… [View More]

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com Oct. 19, 2022

Oct. 19, 2022

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin Sept. 21, 2022

Sept. 21, 2022

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final … [View More]

22 53

PEP 563 and Python 3.10.
by Thomas Wouters July 22, 2022

July 22, 2022

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the … [View More]default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread! [View Less]

6 5

Almost accepting PEP 681 – Data Class Transforms
by Petr Viktorin June 6, 2022

June 6, 2022

Hello, As an initial implementation that will be improved in the future, the specification in PEP 681 is fine. Feel free to add the decorator to Python 3.11 at your convenience. However, the PEP includes several worrying recommendations like: - we recommend that the maintainers of attrs move away from the legacy semantics and adopt auto_attribs behaviors by default. - We chose not to support this feature and recommend that attrs users avoid converters. - Attrs users should use the … [View More]

6 7

PEP 689 – Semi-stable C API tier
by Petr Viktorin June 1, 2022

June 1, 2022

Hello, Victor and others are organizing the C-API to make it clearer what's public and what's private (see Victor's file-based stats: https://pythoncapi.readthedocs.io/stats.html) It became clear that we need a tier between public and private C-API. PEP 689 proposes such API, with an opt-in macro and transition period. Please discuss. And if you can think of a better name, that would be great :) The PEP is at: https://peps.python.org/pep-0689/ Thread where this started: https://mail.… [View More]python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6… Implementation notes: https://github.com/python/cpython/issues/91744 Here's the current version for easy quoting: ---------- PEP: 689 Title: Semi-stable C API tier Author: Petr Viktorin <encukou(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 523 Created: 22-Apr-2022 Python-Version: 3.11 Abstract ======== Some functions and types of the C-API are designated *semi-stable*, meaning that they will not change in patch (bugfix/security) releases, but may change between minor releases (e.g. between 3.11 and 3.12) without deprecation warnings. Motivation & Rationale ====================== The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__: - Limited API, with high compatibility expectations - Public API, which follows the :pep:`backwards compatibility policy <387>`, and requires deprecation warnings before changes - Private (internal) API, which can change at any time. We need a tier between Public and Private to accommodate tools like advanced debuggers and JIT compilers, which need access to CPython internals but assume the C-API they use does not change in patch releases. Setting Stability Expectations ------------------------------ Currently, there are no guarantees for the internal API -- that is, anything that requires ``Py_BUILD_CORE`` or is named with a leading underscore. This API can change without warning at any time, and code that uses it is pinned to a specific build of Python. However, in practice, even the internal API usually happens to be stable in patch releases: - Some CPython core developers take this as an an unwritten rule. - Patch releases only contain bugfixes, which are unlikely to change the API. Semi-stable API will make the stability expectations more explicit. It will also hopefully encourage existing users of the private API to reach out to python-dev, so we can expose, standardize and test an API for some of their use cases. Reserving underscores for Private API ------------------------------------- :pep:`523` introduced functions for use by debuggers and JIT compilers, which are stable only across minor releases. The functions names have leading underscores to suggest their limited stability. However, leading underscores usually mark *fully private* API. CPython developers familiar with the “underscore means internal” convention are unlikely to check if underscored functions they are changing are documented and used outside CPython itself. This proposal brings us a bit closer to reserving underscores only for truly private, unstable, hands-off API. Warning about API that is changed often --------------------------------------- The ``PyCode_New()`` family is an example of functions that are documented as unstable, and also often change in practice. Moving it to the semi-stable tier will make its status obvious even to people who don't read the docs carefully enough, and will make it hard to use accidentally. Changes during the Beta period ------------------------------ Since the API itself can change continuously up until Beta 1 (feature freeze) of a minor version, major users of this API are unlikely to test Alpha releases and provide feedback. It is very difficult to determine what needs to be exposed as semi-stable. Additions to the semi-stable tier will count as *stabilization*, and will be allowed up to Release Candidate 1. Specification ============= Several functions and types (“APIs”) will be moved to a new *semi-stable* tier. They will be expected to stay stable across patch releases, but may change or be removed without warning in minor releases (3.x.0), including Alpha and Beta releases of 3.x.0. When they change significantly, code that uses them should no longer compile (e.g. arguments should be added/removed, or a function should be renamed, but the semantic meaning of an argument should not change). Their definitions will be moved to a new directory, ``Include/semistable/``, and will be included from ``Python.h``. From Python 3.12 on, these APIs will only be usable when the ``Py_USING_SEMI_STABLE_API`` macro is defined. CPython will only define the macro for building CPython itself (``Py_BUILD_CORE``). To make transition to semi-stable API easier, in Python 3.11 the APIs will be available without ``Py_USING_SEMI_STABLE_API`` defined. In this case, using them will generate a deprecation warning on compilers that support ``Py_DEPRECATED``. A similar deprecation period will be used when making more APIs semi-stable in the future: - When moving from public API, the deprecation period should follow Python's backwards compatibility policy (currently, it should last at least two releases). - When moving from public API that is documented as unstable, the deprecation period can only last one release. - When moving from private API or adding new API, no deprecation period is necessary. Leading underscores will be removed from the names of the moved APIs. The old underscored name of a renamed API will be available (as an alias using ``#define``) at least until that API changes. The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be documented, and documentation of each semi-stable API will be updated. Adjustments during Beta periods ------------------------------- New APIs can be added to the semi-stable tier, and private APIs can be moved to it, up to the first release candidate of a new minor version. Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period. In the Beta period, no API may be moved to more private tier, e.g. what is public in Beta 1 must stay public until the final release. Initial semi-stable API ----------------------- The following API will initially be semi-stable. The set may be adjusted for 3.11. Code object constructors: - ``PyCode_New()`` - ``PyCode_NewWithPosOnlyArgs()`` Frame evaluation API (PEP 523): - ``_PyFrameEvalFunction`` - ``_PyInterpreterState_GetEvalFrameFunc()`` - ``_PyInterpreterState_SetEvalFrameFunc()`` - ``_PyEval_RequestCodeExtraIndex()`` - ``_PyCode_GetExtra()`` - ``_PyCode_SetExtra()`` - ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct) - ``_PyFrame_GetFrameObject`` - ``PyEval_EvalFrameDefault`` (new function that calls ``_PyEval_EvalFrameDefault``, but takes ``PyFrameObject`` rather than ``_PyInterpreterFrame``) (Leading underscores will be removed as mentioned above.) Backwards Compatibility ======================= The C API backwards compatibility story will be made clearer. How to Teach This ================= The changes affect advanced C programmers, who should consult the updated reference documentation, devguide and/or What's New document·. Reference Implementation ======================== https://github.com/python/cpython/issues/91744 Rejected Ideas ============== It might be good to add a similar tier in the Python (not C) API, e.g. for ``types.CodeType``. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP. Open Issues =========== - “Semi-stable” is not a perfect name. - The exact set of exposed API may change. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

13 24

compiled python3.10 is unable to find _ssl
by Sandeep Gupta May 29, 2022

May 29, 2022

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>",… [View More]

6 12

Add -P command line option to not add sys.path[0]
by Victor Stinner May 5, 2022

May 5, 2022

Hi, There are 4 main ways to run Python: (1) python -m module [...] (2) python script.py [...] (3) python -c code [...] (4) python [...] (1) and (2) insert the directory of the module/script at sys.path[0]. (3) and (4) insert an empty string at sys.path[0]. This behavior is convenient and is maybe part of Python usability success: importing a module in the current directory is as easy as "import other_module" (load other_module.py). But it's also a threat to security: an attacker can … [View More]

9 19

Using the Python C API in C++
by Victor Stinner May 3, 2022

May 3, 2022

Hi, If you got issues with using the Python C API <Python.h> in C++, please speak up! I'm looking for feedback :-) Extending Python by writing C++ code is now easy with the pybind11 project: https://pybind11.readthedocs.io/ It seems like over the last years, building C++ extensions with the Python C API started to emit more C++ compiler warnings. One explanation may be that converting macros to static inline functions (PEP 670) introduce new warnings, even if the old and the new code … [View More]

5 16

Proto-PEP part 4: The wonderful third option
by Larry Hastings May 1, 2022

May 1, 2022

Sorry, folks, but I've been busy the last few days--the Language Summit is Wednesday, and I had to pack and get myself to SLC for PyCon, &c. I'll circle back and read the messages on the existing threads tomorrow. But for now I wanted to post "the wonderful third option" for forward class definitions we've been batting around for a couple of days. The fundamental tension in the proposal: we want to /allocate/ the object at "forward class" time so that everyone can take a … [View More]reference to it, but we don't want to /initialize/ the class (e.g. run the class body) until "continue class" time. However, the class might have a metaclass with a custom __new__, which would be responsible for allocating the object, and that isn't run until after the "class body". How do we allocate the class object early while still supporting custom metaclass.__new__ calls? So here's the wonderful third idea. I'm going to change the syntax and semantics a little, again because we were batting them around quite a bit, so I'm going to just show you our current thinking. The general shape of it is the same. First, we have some sort of forward declaration of the class. I'm going to spell it like this: forward class C just for clarity in the discussion. Note that this spelling is also viable: class C That is, a "class" statement without parentheses or a colon. (This is analogous to how C++ does forward declarations of classes, and it was survivable for them.) Another viable spelling: C = ForwardClass() This spelling is nice because it doesn't add new syntax. But maybe it's less obvious what is going on from a user's perspective. Whichever spelling we use here, the key idea is that C is bound to a "ForwardClass" object. A "ForwardClass" object is /not/ a class, it's a forward declaration of a class. (I suspect ForwardClass is similar to a typing.ForwardRef, though I've never worked with those so I couldn't say for sure.) Anyway, all it really has is a name, and the promise that it might get turned into a class someday. To be explicit about it, "isinstance(C, type)" is False. I'm also going to call instances of ForwardClass "immutable". C won't be immutable forever, but for now you're not permitted to set or change attributes of C. Next we have the "continue" class statement. I'm going to spell it like this: continue class C(BaseClass, ..., metaclass=MyMetaclass): # class body goes here ... I'll mention other possible spellings later. The first change I'll point out here: we've moved the base classes and the metaclass from the "forward" statement to the "continue" statement. Technically we could put them either place if we really cared to. But moving them here seems better, for reasons you'll see in a minute. Other than that, this "continue class" statement is similar to what I (we) proposed before. For example, here C is an expression, not a name. Now comes the one thing that we might call a "trick". The trick: when we allocate the ForwardClass instance C, we make it as big as a class object can ever get. (Mark Shannon assures me this is simply "heap type", and he knows far more about CPython internals than I ever will.) Then, when we get to the "continue class" statement, we convince metaclass.__new__ call to reuse this memory, and preserve the reference count, but to change the type of the object to "type" (or what-have-you). C has now been changed from a "ForwardClass" object into a real type. (Which almost certainly means C is now mutable.) These semantics let us preserve the entire existing class creation mechanism. We can call all the same externally-visible steps in the same externally-visible order. We don't add any new dunder methods, we don't remove any dunder methods, we don't expose a new dunder attribute for users to experiment with. What mechanism do we use to achieve this? metaclass.__new__ always has to do one of these two things to create the class object: either it calls "super().__new__", or what we usually call "three-argument type". In both cases, it passes through the **kwargs that it received into the super().__new__ call or the three-argument type call. So the "continue class C" statement will internally add a new kwarg: "__forward__ = C". If super().__new__ or three-argument type get this kwarg, they won't allocate a new object, they'll reuse C. They'll preserve the current reference count, but otherwise overwrite C with all the juicy vitamins and healthy minerals packed into a Python class object. So, technically, this means we could spell the "continue class" step like so: class C(BaseClass, ..., metaclass=MyMetaClass, __forward__=C): ... Which means that, combined with the "C = ForwardClass()" statement above, we could theoretically implement this idea without changing the syntax of the language. And since we already don't have to change the underlying semantics of Python class creation, the technical debt incurred by adding this to the language becomes much smaller. What could go wrong? My biggest question so far: is there such a thing as a metaclass written in C, besides type itself? Are there metaclasses with a __new__ that /doesn't/ call super().__new__ or three-argument type? If there are are metaclasses that allocate their own class objects out of raw bytes, they'd likely sidestep this entire process. I suspect this is rare, if indeed it has ever been done. Anyway, that'd break this mechanism, so exotic metaclasses like these wouldn't work with "forward-declared classes". But at least they needn't fail silently. We just need to add a guard after the call to metaclass.__new__: if we passed in "__forward__=C" into metaclass.__new__, and metaclass.__new__ didn't return C, we raise an exception. Cheers, //arry/ p.s. When I say "we" above, I generally mean Eric V. Smith, Barry Warsaw, Mark Shannon, and myself. But please assume that any dumb ideas in the proposal are mine, and I was too wrong-headed to listen to the sage advice from these three wise men when I wrote this email. [View Less]

12 20

Summary of Python tracker Issues
by Python tracker April 30, 2022

April 30, 2022

ACTIVITY SUMMARY (2022-04-22 - 2022-04-29) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/… [View More]

2 1

Decreasing refcount for locals before popping frame
by Malthe April 29, 2022

April 29, 2022

Consider this example code: def test(): a = A() test() Currently, the locals (i.e. `a`) are cleared only after the function has returned: If we attach a finalizer to `a` immediately after the declaration then the frame stack available via `sys._getframe()` inside the finalizer function does not include the frame used to evaluate the function (i.e. with the code object of the `test` function). The nearest frame is that of the top-level module (where we make the call to the function). … [View More]

4 9

Re: Add -P command line option to not add sys.path[0]
by Miro Hrončok April 27, 2022

April 27, 2022

On 27. 04. 22 21:34, Steve Holden wrote: > So would PYTHONDONTADDSCRIPTDIR=1 be a better choice? No because for other invocations, it prevents PWD being added, not the scriptdir. Both names are bad becasue they only describe half of what's it doing (or maybe the option should not do both, if it's hard to explain). -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Proto-PEP part 1: Forward declaration of classes
by Larry Hastings April 27, 2022

April 27, 2022

This document is a loose proto-PEP for a new "forward class" / "continue class" syntax. Keep in mind, the formatting is a mess. If I wind up submitting it as a real PEP I'll be sure to clean it up first. /arry -------------------------------------- PEP XXXX: Forward declaration of classes Overview -------- Python currently has one statement to define a class, the `class` statement: ```Python class X(): # class body … [View More]goes here def __init__(self, key): self.key = key ``` This single statement declares the class, including its bases and metaclass, and also defines the contents of the class in the "class body". This PEP proposes an additional syntax for declaring a class which splits this work across two statements: * The first statement is `forward class`, which declares the class and binds the class object. * The second statement is `continue class`, which defines the contents of the class in the "class body". To be clear: `forward class` creates the official, actual class object. Code that wants to take a reference to the class object may take references to the `forward class` declared class, and interact with it as normal. However, a class created by `forward class` can't be *instantiated* until after the matching `continue class` statement finishes. Defining class `X` from the previous example using this new syntax would read as follows: ``` forward class X() continue class X: # class body goes here def __init__(self, key): self.key = key ``` This PEP does not propose altering or removing the traditional `class` statement; it would continue to work as before. Rationale --------- Python programmers have had a minor problem with classes for years: there's no way to have early-bound circular dependencies between objects. If A depends on B, and B depends on A, there's no linear order that allows you to cleanly declare both. Most of the time, the dependencies were in late-binding code, e.g. A refers to B inside a method. So this was rarely an actual problem at runtime. When this problem did arise, in code run at definition-time, it was usually only a minor headache and could be easily worked around. But the explosion of static type analysis in Python, particularly with the `typing` module and the `mypy` tool, has made circular definition-time dependencies between classes commonplace--and much harder to solve. Here's one simple example: ```Python class A: value: B class B: value: A ``` An attribute of `B` is defined using a type annotation of `A`, and an attribute of `A` is defined using a type annotation of `B`. There's no order to these two definitions that works; either `A` isn't defined yet, or `B` isn't defined yet. Various workarounds and solutions have been proposed to solve this problem, including two PEPs: PEP 563 (automatic stringized annotations) and PEP 649 (delayed evaluation of annotations using functions). But nothing so far has been both satisfying and complete; either it is wordy and clumsy to use (manually stringizing annotations), or it added restrictions and caused massive code breakage for runtime use of annotations (PEP 563), or simply didn't solve every problem (PEP 649). This proposed `forward class` / `continue class` syntax should permit solving *every* forward-reference and circular-reference problem faced in Python, using an elegant and Pythonic new syntax. As a side benefit, `forward class` and `continue class` syntax enables rudimentary separation of "interface" from "implementation", at least for classes. A user seeking to "hide" the implementation details of their code could put their class definitions in one module, and the implementations of those classes in a different module. This new syntax is not intended to replace the traditional `class` declaration syntax in Python. If this PEP were accepted, the `class` statement would still be the preferred mechanism for creating classes in Python; `forward class` should only be used when it confers some specific benefit. Syntax ------ The `forward class` statement is the same as the `class` statement, except it doesn't end with a colon and is not followed by an indented block. Without any base classes or metaclass, the `forward class` statement is as follows: ``` forward class X ``` This would declare class `X`. If `X` needs base classes or metaclass, the corresponding `forward class` statement would be as follows, rendered in a sort of "function prototype" manner: ``` forward class X(*bases, metaclass=object, **kwargs) ``` The `continue class` statement is similar to a `class` statement without any bases or metaclass. It ends with a colon, and is followed by the "class body": continue class X: # class body goes here pass One important difference: the `X` in `continue class X:` is not a *name*, it's an *expression*. This code is valid: ``` forward class X() snodgrass = X continue class snodgrass: # class body goes here pass ``` as well as this: ``` import my_module continue class my_module.X: # class body goes here pass ``` Using this new syntax, the forward-reference problem illustrated in the *Rationale* section above is now easy to solve: ```Python forward class B class A: value: B = None continue class B: value: A = None ``` One final note. Why must the base and metaclass be declared with the `forward class` statement? The point of this new syntax is to allow creating the real class object, permitting users of the class to take references to it early, before it's fully defined. And the class could be declared with a metaclass, and the metaclass could have a `__new__`, which means it's responsible for creating the class object, and this syntax would have to preserve that behavior. (This small detail is about to complicate this proposal a great deal!) #### Semantics of forward-declared class objects `forward class X` declares a class, but the class is explicitly not yet fully defined. It won't be fully defined and ready to be instantiated until after the corresponding `continue class` statement. We'll refer to a class object in this state as a "forward-declared class object". How does this object behave? As per the "consenting adults" rule, the forward-declared class object must permit most operations. You should be able to examine the object, compare it to other objects, inspect some attributes (`__name__`, `__mro__`, `__class__`), and even set attributes. However, the user isn't permitted to instantiate a forward-declared class object until after the corresponding `continue class X`. We ensure this with a new dunder attribute, `__forward__`, which if present tells the Python runtime that this is a forward-declared class object. The `continue class` statement would delete this attribute from the object, after which it could be instantiated. (Users could work around this constraint, or even delete `__forward__` if they so chose--again, the "consenting adults" rule applies.) It's explicitly permissible to create a forward-declared class object that never gets finished with a `continue class` statement. If all you need is an object that represents the class--say, to satisfy static type annotation use cases--a forward-declared class object works fine. A subsequent section will address the complexities of how `forward class` and `continue class` interact with metaclasses. For now, a note about forward-declared class objects declared with a metaclass implementing `__prepare__`. The forward-declared class object *dict* will be the "dict-like object" returned by the `metaclass.__prepare__()` method. This "dict-like object" won't be processed and discarded until after `continue class` processes the class body and calls the appropriate methods in the metaclass. #### Semantics of `continue class` `continue class` may only be run on a class once. (As Eric V. Smith pointed out in response to an early version of this proposal, allowing multiple "continue" declarations on the same class would lead directly to language-condoned monkey-patching.) #### Decorators Both the `forward class` and `continue class` statements support decorators, and the user may use decorators with either or both statements for the same class. But now that we've split the responsibilities of the `class` statement between these two new statements, which decorator goes with which statement becomes a novel concern. In general, decorators that don't examine the contents of the class, but simply want to register the class object and its name, can decorate the `forward class` statement. Also, class decorators that want to return a different object for the class should decorate `forward class`. But all decorators that meaningfully examine the contents of the class should decorate the `continue class` statement. Unfortunately, there are some decorators that can't work properly with either `forward class` *or* `continue class`: a decorator that meaningfully examine the declared contents of that class, but also return an object other than the original class passed in. In that case, the user cannot declare this class with `forward class`; they must declare it with the conventional `class` statement. #### __slots__ This leads us to an example of a decorator that, as of 3.10, wouldn't be usable with classes declared by `forward class`. It's the new 3.10 feature `@dataclass(slots=True)`. When called in this way, dataclass examines the attributes of the class it has decorated, dynamically constructs a new class using `__slots__`, and returns this new class. Since this decorator meaningfully examines the class, it must be used with `continue class`. But, this decorator also returns an object other the original class, which means it's inapproriate for `continue class` and should be called with `forward class`. What to do? We have a separate idea to ameliorate this specific situation. Right now, a class that uses `__slots__` *must* define them in the class body, as that member is read before the class name is bound (or before any descriptors are run). But we can simply relax that, and make processing `__slots__` lazy, so that it isn't examined until the first time the class is *instantiated.* This would mean `@dataclass(slots=True)` could simply return the original class, and thus would work fine when decorating a `continue class` statement. #### Metaclasses The existing semantics of metaclasses present a thorny problem for `forward class` and `continue class`. First, a review of how class definition works. Most of the mechanisms involved with defining a class happen internally to the interpreter. However, there are a number of special methods (aka "dunder methods") called during class construction that are overridable by user code. Empirical testing with the current version of Python (as of this writing, 3.10.4) reveals the order in which all this work is done. When Python executes the definition of this class: ```Python class Foo(BaseClass, metaclass=MetaClass): # class body is here pass ``` these events are visible in this order: 1. Python calls `MetaClass.__prepare__`. 2. Python executes the "class body" for class Foo. 3. Python calls `MetaClass.__new__`. 4. Python calls `BaseClass.__init_subclass__`. (If multiple base classes define `__init_subclass__`, they're called in MRO order.) 5. Python calls `MetaClass.__init__`. 6. The `class` statement binds the class object to the name `Foo`. The big problem this presents for `forward class`: the "class body" is executed before the `MetaClass.__new__`. This is necessary because one of the parameters to `MetaClass.__new__` is `namespace`, the "dict-like object" returned by `MetaClass.__prepare__` and initialized by executing the class body using that object as a sort of locals dictionary. This creates a chicken-and-egg problem: `forward class` needs to define the class object, but the class object is defined by `MetaClass.__new__`, and `MetaClass.__new__` can't run until after the class body, which we don't run until the `continue class` statement, which must be after `forward class`. The unfortunate but necessary solution: split `__new__` into two new special methods on metaclasses, `__new_forward__` and `__new_continue__`. As a reminder, here's the prototype for `__new__` on a metaclass: ```Python def __new__(metaclass, name, bases, namespace, **kwargs): ``` The two new special methods would have the following prototypes: ```Python def __new_forward__(metaclass, name, bases, namespace, **kwargs): def __new_continue__(metaclass, cls, **kwargs): ``` `__new_forward__` creates the class object. It sets the `namespace` member as the class dict, but in general should not examine it contents. (Specifically, `__new_forward__` cannot make any assumptions about whether or not the class body has been executed yet; more on this in a moment.) `__new_continue__` is guaranteed to be called after the class body has been executed. The externally-visible parts of class construction would run in a different order for classes constructed using `forward class` and `continue class`. First, the visible interactions from the `forward class` statement: 1. Python calls `MetaClass.__prepare__`. 2. Python calls `MetaClass.__new_forward__`. 3. The `forward class` statement binds the (forward-declared) class object to the name `Foo`. And here are the visible interactions from the `continue class` statement: 1. Python executes the class body. 2. Python calls `MetaClass.__new_continue__`. 3. Python calls `BaseClass.__init_subclass__`. (If multiple base classes define `__init_subclass__`, they're called in MRO order.) 4. Python calls `MetaClass.__init__`. It's important to note that, while `namespace` is passed in to `__new_forward__`, it's not yet initialized with the class body. It's passed in here because the "dict-like object" returned by `MetaClass.__prepare__` is used as the `__dict__` for the forward-declared class object. (This is also novel. Normally the "dict-like object" is used as the namespace for the class body, then its contents are copied out and it is discarded. Here it will also be used as the `__dict__` for the forward-declared class object until the `continue class` statement executes.) Splitting `__new__` into two methods in this manner has several ramifications for existing code. First, Python still needs to support `MetaClass.__new__` for backwards compatibility with existing code. Therefore, when executing the `class` statement, Python will still call `MetaClass.__new__`. In fact, for maximum backwards compatibility, the order of externally-visible events for the `class` statement should not change at all. The default implementation of `MetaClass.__new__` will be changed to call `__new_forward__` and `__new_continue__`. The implementation will be similar to the following pseudo-code: ```Python def __new__(metaclass, name, bases, namespace, **kwargs): cls = metaclass.__new_forward__(metaclass, name, bases, namespace, **kwargs) metaclass.__new_continue__(metaclass, cls, namespace, **kwargs) return cls ``` This means the order of events will be slightly different between a class defined with the `class` statement and a class defined with the `forward class` and `continue class` statements. With a `class` statement, the class body will be run *before* `__new_forward__` is called, but with a `forward class` statement, the class body will be run *after* `__new_forward__` is called. (This is why `__new_forward__` cannot know in advance whether or not the class body has been called, and the `namespace` has been filled in.) User code that defines its own metaclass with its own `__new__` must also continue to work. But this leads us to a dangerous boundary condition: * if user code defines a metaclass, and * that metaclass defines `__new__` but not `__new_forward__` or `__new_continue__`, and * user code then uses that metaclass in a `forward class` declaration, then Python must throw a `TypeError` exception. This situation is unsafe: clearly the intention with the user's metaclass is to override some behavior in `__new__`, but the `forward new` statement will never call `__new__`. (It's safe to use a metaclass with `forward class` if it doesn't define `__new__`, or if it defines both `__new__` and either `__new_forward__` or `__new_continue__`. It's also safe to use a metaclass with `class` if it defines either `__new_forward__` or `__new_continue__` but not `__new__`, because the default `__new__` will call both `__new_forward__` and `__new_continue__`.) Going forward, best practice for metaclasses would be to only implement `__new_forward__` and `__new_continue__`. Code with metaclasses that wanted to simultaneously support versions of Python with these new dunder methods *and* older versions of Python that predate this change would likely have to conditionally define their own `__new__`, best practices on this approach TBD. #### Interactions between `class`, `forward class`, and `continue class` `class` and `forward class` both bind a name to a newly-created object. Thus, in the same way that you can have two `class` statements that bind and re-bind the same name: ```Python class C: pass class C: pass ``` You can execute `class` and `forward class` statements in any order to bind and re-bind the same name: ```Python class C: pass forward class C ``` This works as expected; when this code executes, the previous objects are dereferenced, and only the last definition of `C` is kept. Executing a `continue class` statement with a class defined by the `class` statement raises a `ValueError` exception. Executing a `continue class` statement with a class defined by the `forward class` statement that has already had `continue class` executed on it raises a `ValueError` exception. It's expected that knowledgeable users will be able to trick Python into executing `continue class` on the same class multiple times by interfering with "dunder" attributes. The same tricks may also permit users to trick Python into executing `continue class` on a class defined by the `class` statement. This is undefined and unsupported behavior, but Python will not prevent it. Final Notes ----------- #### Alternate syntax Instead of `forward class`, we could use `def class`. It's not as immediately clear, which is why this PEP prefers `forward class`. But this alternate syntax has the advantage of not adding a new keyword to the language. #### forward classes and PEP 649 I suggest that `forward class` meshes nicely with PEP 649. PEP 649 solves the forward-reference and circular-reference problem for a lot of use cases, but not all. So by itself it's not quite a complete solution to the problem. This `forward class` proposal should solve *all* the forward-reference and circular-reference problems faced by Python users today. However, its use requires backwards-incompatible code changes to user code. By adding both PEP 649 and `forward class` to Python, we get the best of both worlds. PEP 649 should handle most forward-reference and circular-reference problems, but the user could resort to `forward class` for the stubborn edge cases PEP 649 didn't handle. In particular, combining this PEP with PEP 649 achieves total coverage of the challenges cited by PEP 563's *Rationale* section: > * forward references: when a type hint contains names that have not been > defined yet, that definition needs to be expressed as a string literal; > * type hints are executed at module import time, which is not > computationally free. PEP 649 solves many forward reference problems, and delays the evaluation of annotations until they are used. This PEP solves the remaining forward reference problems. ### A proof-of-concept using decorators I've published a repo with a proof-of-concept of the `forward class` / `continue class` syntax, implemented using decorators. It works surprisingly well, considering. You can find the repo here: https://github.com/larryhastings/forward Naturally, the syntax using this decorators-based version can't be quite as clean. The equivalent declaration for `class X` using these decorators would be as follows: ```Python from forward import * @forward() class X(): ... @continue_(X) class _: # class body goes here pass ``` Specifically: * You have to make the `forward` module available somehow. You can just copy the `forward` directory into the directory you want to experiment in, or you can install it locally in your Python install or venv by installing the `flit` package from PyPI and running `flit install -s` . * You must import and use the two decorators from the `forward` module. The easiest way is with `from forward import *` . * For the `forward class` statement, you instead decorate a conventional class declaration with `@forward()`. The class body should be empty, with either a single `pass` statement or a single ellipsis `...` on a line by itself; the ellipsis form is preferred. You should name this class with the desired final name of your class. * For the `continue class` statement, you instead decorate a conventional class declaration with `@continue_()`, passing in the forward-declared class object as a parameter to the decorator. You can use the original name of the class if you wish, or a throwaway name like `_` as per the example. * You may use additional decorators with either or both of these decorators. However it's vital that `@forward()` and `@continue_()` are the *first* decorators run--that is, they should be on the *bottom* of the stack of decorators. Notes and caveats on the proof-of-concept: * The `continue_` decorator returns the original "forwarded" class object. This is what permits you to stack additional decorators on the class. (But, again, you *must* call the `continue_` decorator first--it should be on the bottom.) * To use `__slots__`, you will have to declare them in the `forward` class. * The proof-of-concept can't support classes that inherit from a class which defines `__init_subclass__`. * Like the proposed syntax, this proof-of-concept doesn't support decorators that both examine the contents of the class *and* return a different object, e.g. `@dataclass(slots=True)` in Python 3.10. * This proof-of-concept doesn't work with metaclasses that override either `__new__` or `__init__`, where those functions examine the `namespace` argument in any meaningful way. #### tools/ There are some tools in the `tools/` directory that will (attempt to) automatically add or remove the `@forward()` decorator to class definitions in Python scripts. It turns this: ```Python class foo(...): pass ``` into this: ```Python @forward() class foo(...): ... @continue_(foo) class _____: pass ``` `tools/edit_file.py` will edit one or more Python files specified on the command-line, making the above change. By default it will toggle the presence of `@forward` decorators. You can also specify explicit behavior: `-a` adds `@forward()` decorators to `class` statements that don't have them. `-r` removes `@forward` decorators, changing back to conventional `class` statements. `-t` requests that it "toggle" the state of `@forward()` decorators. The parser is pretty dumb, so don't run it on anything precious. If it goofs up, sorry! `tools/edit_tree.py` applies `edit_py.py` to all `*.py` files found anywhere under a particular directory. `tools/edit_stdlib.py` was an attempt to intelligently apply `edit_file.py` to the `Lib` tree of a CPython checkout. Sadly, the experiment didn't really work out; it seemed like there were so many exceptions where the brute-force modification didn't work, either due to descriptors, metaclasses, or base classes with `__init_subclass__`, that I gave up on the time investment. It's provided here in a non-functional state in case anyone wants to experiment with it further. Also, it's intentionally delicate; it only works on git checkout trees, and only with one specific revision id: 7b87e8af0cb8df0d76e8ab18a9b12affb4526103 #### Postscript Thanks to Eric V. Smith and Barry Warsaw for proofreading and ideas. Thanks in particular to Eric V. Smith for the idea about making `__slots__` processing lazy. Thanks to Mark Shannon for the idea of prototyping `forward class` and `continue class` using decorators and simply copying the attributes. [View Less]

16 61

Re: Reminder: PEP 479's __future__ about to become the default behavior
by tổng vương April 27, 2022

April 27, 2022

1 0

New functionality for unittest.mock side_effect
by Roy Smith April 26, 2022

April 26, 2022

I often want a side_effect of "if called with foo, return bar" functionality. This is really useful when the order of calls to your mock is indeterminate, so you can't just use an iterable. What I end up doing is writing a little function: def f(x): data = { 'foo': 'bar', 'baz': 'blah', } return data[x] my_mock.side_effect = f sometimes I wrap that up in a lambda, but either way it's fidgety boilerplate which obfuscates the intent. It would be nice if side_effect could accept a … [View More]

1 0

Proposal to deprecate mailcap
by Brett Cannon April 26, 2022

April 26, 2022

A CVE has been opened against mailcap (see https://github.com/python/cpython/issues/68966 for details). I'm not aware of anyone trying to maintain the module and Victor did a search online and didn't find any use of the module in the top 5000 projects on PyPI (see the issue). The module is also under 300 lines of Python code that only ( https://github.com/python/cpython/blob/main/Lib/mailcap.py), so vendoring wouldn't be burdensome. As such, I'm proposing we deprecate mailcap in 3.11 and … [View More]

6 9

Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
by Larry Hastings April 25, 2022

April 25, 2022

Here's one alternate idea for how to implement the "forward class" syntax. The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object? In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. … [View More]This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__. "continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not. The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls. The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object. A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern? A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method: @special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls) This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is. One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this? metaclass.__bind_proxy__(metaclass, None, cls) The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you". A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it. One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like: forward class X ... continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster") Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc. [View Less]

3 3

C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Victor Stinner April 25, 2022

April 25, 2022

Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://… [View More]

7 20

What is Modules/rotatingtree.c for?
by Patrick Reader April 24, 2022

April 24, 2022

I've just noticed Modules/rotatingtree.{h,c}, which don't seem to be used anywhere. Are they just dead code? If so, is there a reason they haven't been removed?

2 1

Proto-PEP part 3: Closing thoughts on "forward class", etc.
by Larry Hastings April 24, 2022

April 24, 2022

Just a quick note from me on the proto-PEP and the two proposed implementations. When I started exploring this approach, I didn't suspect it'd require such sweeping changes to be feasible. Specifically, I didn't think I was going to propose changing the fundamental mechanism used to create class objects. That's an enormous change, and it makes me uncomfortable; I suspect I won't be alone in having that reaction. The alternate implementation with proxy objects was borne of my … [View More]

2 1

Re: Proto-PEP part 1: Forward declaration of classes
by Tobias HT April 23, 2022

April 23, 2022

This seems like it would solve a huge problem that I've often faced myself, but by introducing something that might later on cause problems I anticipate. Like some people suggested earlier, introducing new keyword in python is not a good solution, I feel it just adds to the already existing bulk and increases complexity. Also Like some else mentioned again, this solution doesn't seem pythonic. I don't know what they meant by that, but in hindsight, it does really not feel pythonic. On the … [View More]

2 1

Summary of Python tracker Issues
by Python tracker April 22, 2022

April 22, 2022

ACTIVITY SUMMARY (2022-04-15 - 2022-04-22) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/… [View More]

1 0

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Jason Ansel April 20, 2022

April 20, 2022

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

11 25

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (… [View More]353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

12 38

Updating inspect APIs
by Pablo Galindo Salgado April 20, 2022

April 20, 2022

Hi, We are currently debating in gh-88116 ( https://github.com/python/cpython/issues/88116) what's the best way forward to update the APIs in the inspect module to include the new position information. These APIs are inspect.getframeinfo, inspect.getouterframes, inspect.getinnerframes, inspect.stack and inspect.trace. The problem is that these APIs return a named tuple that now needs to include several new attributes (or one 4 tuple for the positions). Being named tuples, if we add a new … [View More]

7 9

Need help on security vulnerability zlib 1.2.11
by Prasad, PCRaghavendra April 20, 2022

April 20, 2022

Hi All, We are facing some issue with the zlib package 1.2.11. Recently there was a vulnerability in zlib and we had to upgrade to 1.2.12 on all supported platforms We did that in all platforms including windows, python39.dll is now showing 1.2.12 but the problem is we use pyinstaller to generate application exe. This exe is still referring to 1.2.11 we tried lot of things to find how it is linking to 1.2.11, there is no line of sight on this. Can any one please provide some input on this … [View More]

3 2

GSOC
by faresbasousy＠gmail.com April 19, 2022

April 19, 2022

i want to contribute in mailman which is a sub-org under python software in GSOC there's 2 templates to submit i want to know which template is the correct one, the one in mailman's site or the one in python's site

3 2

Do you ever use ceval.c's LLTRACE feature?
by Dennis Sweeney April 19, 2022

April 19, 2022

Hi everyone, I'm looking to improve the output of the interpreter's LLTRACE feature to make it more understandable. This "feature" is undocumented, but it prints out the opcode and oparg of every bytecode instruction executed, along with some info about stack operations, whenever you've built with Py_DEBUG and the name `__ltrace__` is defined in the module. I've found this useful for debugging bits of the compiler and bytecode interpreter. For example, if I make some tweak that instroduces an … [View More]off-by-one error, by default I get a segfault or a rather unhelpful assertion failure at `assert(EMPTY())` or `assert(STACK_LEVEL() <= frame->f_code->co_stacksize)` or similar, at best, with no inducation as to where or why that assertion is failing. But if I enable `__ltrace__` by either setting `__ltrace__=1` in some module or by manually setting `lltrace=1;` in the c code, I can follow what was happening in the interpreter just before the crash. I'd like the output in that scenario to be a bit more helpful. I propose printing opcode names rather than decimal digits, and printing out the name of the current function whenever a stack frame begins executing. I also proprose to print out the full stack contents (almost never very deep) before each bytecode, rather than printing the state piecemeal at each PUSH/POP/STACK_ADJUST macro. I opened issue https://github.com/python/cpython/issues/91462 and PR https://github.com/python/cpython/pull/91463 I later found that this had been explored before by https://github.com/python/cpython/issues/69757, and there was a suggestion that this could be folded into a more generalized bytecode-level tracing feature that is pluggable with python code, similar to sys.settrace(). I would tend to think "YAGNI" -- lltrace is a feature for debugging the c internals of the interpreter, and there are already separate existing features like the `trace` module for tracing through Python code with different goals. I appreciate the simplicity of printf statements at the c level -- it feels more trustworthy than adding a complicated extra feature involving python calls and global state. It's as if I just littered the code with my own debugging print statements, but re-usable and better. I see no documentation anywhere, and there's only one test case in test_lltrace, just testing that there's no segfault. Looking back through the git history, I see that the basic `printf("%d: %d, %d\n", ...);` format goes back to 1990: https://github.com/python/cpython/blob/3f5da24ea304e674a9abbdcffc4d671e32aa… I'm essentially writing to ask: how do you use lltrace? Does anyone rely on the particular format of the output? Would some of these improvements be helpful to you? What else could make it more helpful? Thanks, Dennis Sweeney [View Less]

3 2

Accepting PEP 670 – Convert macros to functions in the Python C API
by Petr Viktorin April 19, 2022

April 19, 2022

Hello, The Steering Council is **Accepting** the latest iteration of PEP 670: Convert macros to functions in the Python C API. Congratulations! - Petr, on behalf of the Steering Council

2 1

Summary of Python tracker Issues
by Python tracker April 15, 2022

April 15, 2022

ACTIVITY SUMMARY (2022-04-08 - 2022-04-15) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/… [View More]

2 1

PEP 687 – Isolating modules in the standard library
by Petr Viktorin April 12, 2022

April 12, 2022

Hello, Please provide any feedback you might have on PEP 687 – Isolating modules in the standard library: https://peps.python.org/pep-0687/ From recent discussions around “what should have a PEP”, it’s clear that this should have been a PEP long ago. Better late than never, I guess! We submit this PEP to explain the changes, seek consensus on whether they are good, propose the remaining changes, and set best practices for new modules. There's a discussion thread on Discourse: https://discuss.python.org/t/pep-687-isolating-modules-in-the-standard-libr…

4 3

Declarative imports
by Malthe April 12, 2022

April 12, 2022

This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing. But firstly, let me present the idea. It is very simple, that Python should have declarative imports, usable anywhere using a simple syntax, @<dotted-name>. For example, `some_regex = @re.compile(...)`. What happens then is that before anything else in that module, that symbol is imported: from re import compile as _mangled_re_compile It … [View More]

17 34

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for … [View More]

2 1

Importing a submodule doesn't always set an attribute on its parent
by dfremont＠ucsc.edu April 11, 2022

April 11, 2022

Hello, I came across what seems like either a bug in the import system or a gap in its documentation, so I'd like to run it by folks here to see if I should submit a bug report. If there's somewhere else more appropriate to discuss this, please let me know. If you import A.B, then remove A from sys.modules and import A.B again, the newly-loaded version of A will not contain an attribute referring to B. Using "collections.abc" as an example submodule from the standard library: >>> … [View More]

7 10

RFC on PEP 681: Data Class Transforms
by Erik De Bonte April 11, 2022

April 11, 2022

PEP 681<https://www.python.org/dev/peps/pep-0681/> improves type checker support for libraries with dataclass-like behaviors by introducing a way to map the behaviors of decorators and classes in those libraries to the behaviors of stdlib dataclass. The PEP will not affect CPython directly except for the addition of the dataclass_transform decorator in typing.py. The decorator is currently in typing_extensions. The canonical discussion thread is on Typing SIG<https://mail.python.org/… [View More]

1 0

Need gh-13143 (bpo-34975: Add start_tls() method to streams API) to be merged
by Oleg Iarygin April 11, 2022

April 11, 2022

Hi everyone, Can PR gh-13143 be reopened and reviewed/merged when possible? It was closed in favor of the new asyncio streams API (gh-13251). Later, the API was cleanly removed (gh-16482) so the proposed PR became relevant again. The added method allows to turn encryption on on demand (and off, as the author expressed his readiness). It's required for a CPython test suit, to port `test.test_ftplib.DummyFTPServer` from asyncore/asynchat before their removal according to PEP 594 (Removing dead … [View More]

2 1

GitHub Issues are now live
by Ezio Melotti April 10, 2022

April 10, 2022

The migration from bugs.python.org to GitHub is now officially complete, and all the issues have been successfully transferred. You can read the full announcement here: https://discuss.python.org/t/github-issues-are-now-live/14967 Best Regards, Ezio Melotti

1 0

About PEPs being discussed on Discourse
by Victor Stinner April 9, 2022

April 9, 2022

Hi, Would it be possible to announce new PEPs on python-dev please? I don't go often to Discourse, like once a month. I don't get any notification by email. I expected new PEPs to be announced on python-dev, but they are not announced here anymore. Is it possible to get Discourse notifications by email, but only for new PEPs? Using mailing lists, it's easy: just select the mailing list that you would like to subscribe to. So I went to Discourse, I click on "New Topics" and I don't see any … [View More]

14 17

Summary of Python tracker Issues
by Python tracker April 8, 2022

April 8, 2022

ACTIVITY SUMMARY (2022-04-01 - 2022-04-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( -7) closed 51841 (+78) total 58987 (+71) Open issues with patches: 2890 Issues opened (49) ================== #47104: Rewrite asyncio.to_thread tests to use IsolatedAsyncioTestCase https://bugs.python.org/issue47104 reopened by kj #47136: The … [View More]

1 0

The GitHub Issues Migration begins today
by Ezio Melotti April 8, 2022

April 8, 2022

In about 1 hour from now, bugs.python.org will become read-only, and the migration to GitHub Issues will begin. Throughout the migration it will not be possible to report new issues or comment on existing ones. Once all the issues have been migrated from bpo to GitHub, you will be able to comment using GitHub Issues. bpo will remain available in read-only mode. For live updates, see https://discuss.python.org/t/github-issues-migration-status-update/14573 Best Regards, Ezio Melotti

1 0

Heads-up: Converting Misc/stable_abi.txt to TOML
by Petr Viktorin April 8, 2022

April 8, 2022

Hello, Now that tomllib is in the stdlib, I'd like to convert the stable ABI manifest (Misc/stable_abi.txt) to TOML to make it easier to work with. To get proper highlighting in editors, I'd like to rename it from .txt to .toml. So, when the command that uses it stops working for you, change `.txt` to `.toml`. (See https://devguide.python.org/c-api/#adding-a-new-definition-to-the-limited-a… -- if you're using make, nothing should change for you.) As always, the file is only intended as … [View More]

1 0

[IMPORTANT] Preparations for 3.11.0 beta 1
by Pablo Galindo Salgado April 8, 2022

April 8, 2022

Hi everyone, We have approximately one month until feature freeze and for 3.11.0b1 to be released. I wanted to take this time to share some planning and considerations with you. Please, read carefully these points as they are important. * 3.11.0b1 is scheduled for Friday, 2022-05-06, which is after the PyCon US sprints. As the sprints normally end with a considerable amount of PRs getting merged and the buildbots having a hard time, I may need to move the release before or after to … [View More]

3 4

PEP 686 – Make UTF-8 mode default
by Inada Naoki April 8, 2022

April 8, 2022

Hi, all. I wrote a new PEP last month. I'm sorry that I forgot to announce it here. The pep is here: https://peps.python.org/pep-0686/ Discussions: * https://discuss.python.org/t/14737 (current thread) * https://discuss.python.org/t/14435 (previous thread) -- Inada Naoki <songofacandy(a)gmail.com>

1 0

[RELEASE] The last Python 3.11 alpha (3.11.0a7) is available - Prepare for beta freeze
by Pablo Galindo Salgado April 6, 2022

April 6, 2022

Brrrrr..... do you feel that? That's the chill of *beta freeze* coming closer. Meanwhile, your friendly CPython release team doesn’t rest and we have prepared a shiny new release for you: Python 3.11.0a7. ************************************************************************************************************************************************************ Dear fellow core developer: This alpha is the last release before feature freeze (Friday, 2022-05-06), so make sure that all new … [View More]features and PEPs are landed in the master branch before we release the first beta. Please, be specially mindfully to check the CI and the buildbots, maybe even using the test-with-buildbots label in GitHub before merging so the release team don’t need to fix a bunch of reference leaks or platform-specific problems on the first beta release. ************************************************************************************************************************************************************ *Go get the new alpha here:* https://www.python.org/downloads/release/python-3110a7/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a7 is the last of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0b1, currently scheduled for Friday, 2022-05-06. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In mathematics, the Dirac delta distribution (δ distribution) is a generalized function or distribution over the real numbers, whose value is zero everywhere except at zero, and whose integral over the entire real line is equal to one. The current understanding of the impulse is as a linear functional that maps every continuous function to its value at zero. The delta function was introduced by physicist Paul Dirac as a tool for the normalization of state vectors. It also has uses in probability theory and signal processing. Its validity was disputed until Laurent Schwartz developed the theory of distributions where it is defined as a linear form acting on functions. Defining this distribution as a "function" as many physicist do is known to be one of the easier ways to annoy mathematicians :) # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower [View Less]

3 6

Descriptions in unittest and avoiding confusion
by Jason R. Coombs April 6, 2022

April 6, 2022

For the edification of all involved, this post summarizes a somewhat surprising behavior in unittest around docstrings. In bpo-46126<https://bugs.python.org/issue46126>, I reported an issue where I’d observed that CPython developers were avoiding the use of docstrings in unittests due to what was perceived as a masking of crucial information. On further investigation, it turns out that with the “descriptions” setting of the TextTestRunner is set to True (the default), unittest will emit … [View More]

9 11

Make HAMT available to python script
by zhang kai April 5, 2022

April 5, 2022

Hi, HAMT is a very useful immutable mapping type. Currently CPython use it internally to implement contextvar. Considering immutable data structure is very useful I hope we can make it available to python script(maybe via collections module). Immutable data structures are fundamental parts of our project. Currently we use a full-featured python immutable data library called pyrsistent. Pyrsistent is very powerful, however the map type in it is implemented in python script not in C. It becomes … [View More]

8 14

code.replace() and Python 3.11 exception table
by Victor Stinner April 1, 2022

April 1, 2022

("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API") On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav(a)gmail.com> wrote: > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner(a)python.org> wrote: > > In Python, sadly the types.CodeType type also has a public constructor > > and many projects break at each Python release because the API > > changes. Hopefully, it seems like the new CodeType.… [View More]

5 6

Summary of Python tracker Issues
by Python tracker April 1, 2022

April 1, 2022

ACTIVITY SUMMARY (2022-03-25 - 2022-04-01) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7153 ( +8) closed 51763 (+61) total 58916 (+69) Open issues with patches: 2902 Issues opened (49) ================== #46907: Update Windows and MacOS installer to SQLite 3.38.2 https://bugs.python.org/issue46907 reopened by ned.deily #47122: Fix the table … [View More]

1 0