Mailman 3 September 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to … [View More]prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian [View Less]

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem,… [View More]

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

Switching to Discourse
by Petr Viktorin Jan. 26, 2023

Jan. 26, 2023

Hello, Currently development discussions are split between multiple communication channels, for example: - python-dev and discuss.python.org for design discussions, - GitHub Issues and Pull Requests for specific changes, - IRC, Discord and private chats for real-time discussions, - Topic-specific channels like typing-sig. While most of these serve different needs, there is too much overlap between python-dev and discuss.python.org. It seems that for most people, this situation is worse than … [View More]sticking to either one platform – even if we don't go with that person's favorite. The discuss.python.org experiment has been going on for quite a while, and while the platform is not without its issues, we consider it a success. The Core Development category is busier than python-dev. According to staff, discuss.python.org is much easier to moderate.. If you're following python-dev but not discuss.python.org, you're missing out. The Steering Council would like to switch from python-dev to discuss.python.org. Practically, this means: - Moving the required PEP announcements to discuss.python.org - Moving discuss.python.org up in the devguide communications page (https://devguide.python.org/communication/) - And that's it? I imagine that the mailing list will stay around for continuing past discussion threads and for announcements, eventually switching to auto-reject incoming messages with a pointer to discuss.python.org. To be clear, discuss.python.org allows editing posts, which is frankly handy for typos and clarifications. Editing alone should not be used for adding new info -- we should cultivate a culture of being friendly to mail users & notification watchers. This probably bears repeating in a few places. We're aware not everyone wants to use the discuss.python.org website, but there are some ways to avoid it: - For new PEPs, you can point your RSS client to https://www.python.org/dev/peps/peps.rss – it's not e-mail, but many email clients have RSS support. You can also watch the Steering Council issues on GitHub (https://github.com/python/steering-council/issues/) for important questions and discussions. - You can use discuss.python.org's “mailing list mode” (which subscribes you to all new posts), possibly with filtering and/or categorizing messages locally. However, we would like to know if this will pose an undue burden to anyone, if there are workflows or usage problems that we are not aware of. As mentioned, this is something the Steering Council thinks is a good idea, but we want to make sure we're aware of all the impact when we make the final decision. – Petr, on behalf of the Steering Council [View Less]

43 114

How about using modern C++ in development of CPython ?
by redradist＠gmail.com Oct. 19, 2022

Oct. 19, 2022

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

Moving to Discourse
by Petr Viktorin Sept. 23, 2022

Sept. 23, 2022

As mentioned previously [0], the Steering Council decided to switch from python-dev to Discourse (discuss.python.org). We're aware that Discourse is not perfect. The previous mail thread [0] lists various shortcomings, as well as some workarounds. However, we don't see anything that would block Python development. Practically, the switch means that: - New PEPs should be announced in the PEPs category on Discourse (rather than on this list), and - The Devguide will list Discourse, rather … [View More]

4 6

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin Sept. 21, 2022

Sept. 21, 2022

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final … [View More]

22 53

Annotating pure functions to improve inline caching/optimization
by Philipp Burch Sept. 19, 2022

Sept. 19, 2022

Hello everyone, the docs on the upcoming 3.11 release state > This [specializing adaptive interpreter] also brings in another concept called inline caching, where Python caches the results of expensive operations directly in the bytecode. I wonder how this caching works, given that the dynamic nature means that virtually every operation could have side effects, causing wrong behaviour when cached. The only mitigation for this that I can imagine is that caching just occurs for basic … [View More]operations defined in the standard library, where it is known that they are free of side effects or "pure". A web search did reveal some discussions[1,2] and a module related to dealing with pure functions, but, as far as I see, not related to optimization. As an example, consider a code like this: ```py @pure def rot_factor(angle_deg: float) -> complex: # This could also be a much more expensive calculation. return cmath.exp(angle_deg / 180 * cmath.pi * 1j) # ... res: List[Tuple(complex, complex, complex, float)] = [] for x in many: res.append(( x * rot_factor(90), x * rot_factor(45), x * rot_factor(-45), x * math.sin(math.pi/8), )) ``` The problem with this code is obvious, every loop iteration calls `rot_factor()` with 90, 45 and -45 and will get exactly the same set of results. The last factor might already be inline cached by the interpreter, since it probably knows that `math.pi` is a constant and `math.sin()` is a pure function. Optimizing this by hand (not considering a list comprehension or other more sophisticated improvements) is easy, but not very pretty: ```py f_p90 = rot_factor(90) f_p45 = rot_factor(45) f_m45 = rot_factor(-45) f_sin = math.sin(math.pi / 8) res: List[Tuple(complex, complex, complex, float)] = [] for x in many: res.append(( x * f_p90, x * f_p45, x * f_m45, x * f_sin, )) ``` I actually find myself often factoring such data out of loops in Python, whereas in C I would just leave that to the optimizer/compiler. An available option would be to use `@lru_cache` for `rot_factor()`, but this will still cause the same dictionary lookups in every iteration and it may not work at all in case the function argument(s) is/are not hashable. Now, if the interpreter understood the `@pure` decorator for `rot_factor()` indicated above would give it the same opportunity to cache the three results throughout the loop, basically creating the manually-optimized code above. For these completely static values, it could even precompute the results and integrate them into the bytecode. Has anything like this been considered already, or is the interpreter itself capable to perform such optimizations? Thanks and best regards, Philipp [1] 'pure' type annotation for mypy: https://github.com/python/mypy/issues/4468 [2] pure-func module: https://pypi.org/project/pure-func/ [View Less]

4 5

glob's new include_hidden parameter
by Mats Wichmann Sept. 13, 2022

Sept. 13, 2022

Just spotted that 3.11 adds an include_hidden. At the moment a little confused because there's an apparent mismatch between docstring and docs. Lib/glob.py: If `include_hidden` is true, the patterns '*', '?', '**' will match hidden directories. Doc/library/glob.rst: If *include_hidden* is true, "``**``" pattern will match hidden directories. ,,, with no mention of the other patterns. Is that just an omission in the rst doc?

3 3

[RELEASE] Python 3.11 release candidate 2 (3.11.0rc2) is available
by Pablo Galindo Salgado Sept. 12, 2022

Sept. 12, 2022

Python 3.11 is one month away, can you believe it? This snake is still trying to bite as it has been an interesting day of fighting fires, release blockers, and a bunch of late bugs but your friendly release team always delivers :) You can get this new release while is still fresh here: https://www.python.org/downloads/release/python-3110rc2/ ## This is the second release candidate of Python 3.11 This release, **3.11.0rc2**, is the last preview before the final release of Python 3.11.0 on … [View More]2022-10-24. Entering the release candidate phase, only reviewed code changes which are clear bug fixes are allowed between this release candidate and the final release. The second candidate and the last planned release preview is currently planned for Monday, 2022-09-05 while the official release is planned for Monday, 2022-10-24. There will be no ABI changes from this point forward in the 3.11 series and the goal is that there will be as few code changes as possible. ## Modification of the final release Due to the fact that we needed to delay the last release candidate by a week and because of personal scheduling problems I am delaying the final release to 2022-10-24 (three weeks from the original date). ## Call to action ⚠️⚠️⚠️⚠️⚠️⚠️⚠️ The 3.11 branch is now accepting changes for 3.11.**1**. To maximize stability, the final release will be cut from the v3.11.0rc2 tag. If you need the release manager (me) to cherry-pick any critical fixes, mark issues as release blockers, and/or add me as a reviewer on a critical backport PR on GitHub. To see which changes are currently cherry-picked for inclusion in 3.11.0, look at the short-lived branch-v3.11.0 https://github.com/python/cpython/tree/branch-v3.11.0 on GitHub. ⚠️⚠️⚠️⚠️⚠️⚠️⚠️ --- #### Core developers: all eyes on the docs now * Are all your changes properly documented? * Did you notice other changes you know of to have insufficient documentation? #### Community members We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.11 compatibilities during this phase. As always, report any issues to [the Python bug tracker ](https://github.com/issues). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Some of the new major new features and changes in Python 3.11 are: ## General changes * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and `except*` * [PEP 680](https://www.python.org/dev/peps/pep-0680/) -- tomllib: Support for Parsing TOML in the Standard Library * [gh-90908](https://github.com/python/cpython/issues/90908) -- Introduce task groups to asyncio * [gh-34627](https://github.com/python/cpython/issues/34627/) -- Atomic grouping (`(?>...)`) and possessive quantifiers (`*+, ++, ?+, {m,n}+`) are now supported in regular expressions. * The [Faster CPython Project](https://github.com/faster-cpython/) is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See [Faster CPython]( https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython) for details. ## Typing and typing language changes * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/) -- Variadic Generics * [PEP 675](https://www.python.org/dev/peps/pep-0675/) -- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/) -- Marking individual TypedDict items as required or potentially-missing * [PEP 681](https://www.python.org/dev/peps/pep-0681/) -- Data Class Transforms <small>(Hey, **fellow core developer,** if a feature you find important is missing from this list, [let Pablo know](mailto:pablogsal@python.org ).)</small> The next release will be the final release of Python 3.11.0, which is currently scheduled for Monday, 2022-10-24. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [ https://github.com/python/cpython/issues](https://github.com/python/cpython… . * [Help fund Python and its community](/psf/donations/). # And now for something completely different In general relativity, a white hole is a theoretical region of spacetime and singularity that cannot be entered from the outside, although energy-matter, light and information can escape from it. In this sense, it is the reverse of a black hole, which can be entered only from the outside and from which energy-matter, light and information cannot escape. White holes appear in the theory of eternal black holes. In addition to a black hole region in the future, such a solution of the Einstein field equations has a white hole region in its past. This region does not exist for black holes that have formed through gravitational collapse, however, nor are there any observed physical processes through which a white hole could be formed. Supermassive black holes are theoretically predicted to be at the centre of every galaxy and that possibly, a galaxy cannot form without one. Stephen Hawking and others have proposed that these supermassive black holes spawn a supermassive white hole. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ If you have any questions, please reach out to me or another member of the release team :) Your friendly release team, Ned Deily @nad https://discuss.python.org/u/nad Steve Dower @steve.dower https://discuss.python.org/u/steve.dower Pablo Galindo Salgado @pablogsal https://discuss.python.org/u/pablogsal [View Less]

1 0

[RELEASE] Python versions 3.10.7, 3.9.14, 3.8.14, 3.7.14 now available with security content
by Łukasz Langa Sept. 7, 2022

Sept. 7, 2022

We have some security content, and plenty of regular bug fixes for 3.10. Let’s dive right in. <https://discuss.python.org/#cve-2020-10735httpscvemitreorgcgi-bincvenamecgi…>CVE-2020-10735 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735> Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError <https://docs.python.org/release/3.10.7/whatsnew/3.10.html#notable-security-…… [View More]> if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. Security releases for 3.9.14, 3.8.14, and 3.7.14 are made available simultaneously to address this issue, along with some less urgent security content. Upgrading your installations is highly recommended. <https://discuss.python.org/#python-3107-2>Python 3.10.7 Get it here: https://www.python.org/downloads/release/python-3107/ <https://www.python.org/downloads/release/python-3107/> This bugfix version of Python was released out-of-schedule to address the CVE, and as such contains a smaller number of changes compared to 3.10.6 (200 commits), or in fact 3.9.7 (187 commits) at the same stage of the release cycle a year ago. But there’s still over a 100 commits in this latest Python version so it’s worth checking out the change log <https://docs.python.org/release/3.10.7/whatsnew/changelog.html>. <https://discuss.python.org/#and-now-for-something-completely-different-3>And now for something completely different In quantum mechanics, the uncertainty principle (also known as Heisenberg’s uncertainty principle) is any of a variety of mathematical inequalities asserting a fundamental limit to the accuracy with which the values for certain pairs of physical quantities of a particle, such as position and momentum or the time and the energy can be predicted from initial conditions. Such variable pairs are known as complementary variables or canonically conjugate variables; and, depending on interpretation, the uncertainty principle limits to what extent such conjugate properties maintain their approximate meaning, as the mathematical framework of quantum physics does not support the notion of simultaneously well-defined conjugate properties expressed by a single value. The uncertainty principle implies that it is in general not possible to predict the value of a quantity with arbitrary certainty, even if all initial conditions are specified. <https://discuss.python.org/#we-hope-you-enjoy-the-new-releases-4>We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Łukasz Langa @ambv <https://discuss.python.org/u/ambv> [View Less]

1 0

[RELEASE] Python 3.11.0 RC2 is blocked
by Pablo Galindo Salgado Sept. 5, 2022

Sept. 5, 2022

Hi everyone, Today was the scheduled release of Python 3.11.0 RC2 but unfortunately, we have a bunch of release blockers. Here are some of them: https://github.com/python/cpython/issues/96569 https://github.com/python/cpython/issues/96559 https://github.com/python/cpython/issues/96572 https://github.com/python/cpython/issues/96046 https://github.com/python/cpython/issues/95027 Here is the current GH query: https://github.com/python/cpython/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda… … [View More]

1 0