Mailman 3 February 2023 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau 17 Oct '23

17 Oct '23

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian

11 14

Python multithreading without the GIL
by Sam Gross 01 Aug '23

01 Aug '23

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

24 48

Mixed Python/C debugging
by Skip Montanaro 26 Apr '23

26 Apr '23

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum 30 Mar '23

30 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

Contributing the Pyston jit?
by Kevin Modzelewski 27 Feb '23

27 Feb '23

Hello all, we on the Pyston team would like to propose the contribution of our JIT <https://github.com/pyston/pyston/blob/pyston_main/Python/aot_ceval_jit.c> into CPython main. We're interested in some initial feedback on this idea before putting in the work to rebase the jit to 3.12 for a PEP and more formal discussion. Our jit is designed to be simple and to generate code quickly, so we believe it's a good point on the design tradeoff curve for potential inclusion. The runtime behavior is intentionally kept almost completely the same as the interpreter, just lowered to machine code and with optimizations applied. Our jit currently targets Python 3.7-3.10, and on 3.8 it achieves a 10% speedup on macrobenchmarks (similar to 3.11). It's hard to estimate the potential speedup of our jit rebased onto 3.12 because there is overlap between what our jit does and the optimizations that have gone into the interpreter since 3.8, but there are several optimizations that would be additive with the current performance work: - Eliminating bytecode dispatch overhead - Mostly-eliminating stack management overhead - Reducing the number of reference count operations in the interpreter - Faster function calls, particularly of C functions - More specialization opportunities, both because a jit is not limited by bytecode limits, but also because it is able to do dynamic specializations that are not possible in an interpreter context There is also room for more optimizations -- in Pyston we've co-optimized the interpreter+jit combination such as by doing more extensive profiling in the interpreter. Our plan would be to submit an initial version that does not contain these optimizations in order to minimize the diff, and add them later. Our jit uses the DynASM assembler library (part of LuaJIT) to generate machine code. Our jit currently supports Mac and Linux, 64-bit ARM and x86_64. Now that we have two architectures supported, adding additional ones is not too much work. We think that our jit fits nicely in the technical roadmap of the Faster CPython project, but conflicts with their plan to build a new custom tracing jit. As mentioned, we'd love to get feedback about the overall appetite for including a jit in CPython! kmod

3 5

[RELEASE] Python 3.11.1, 3.10.9, 3.9.16, 3.8.16, 3.7.16, and 3.12.0 alpha 3 are now available
by Łukasz Langa 18 Feb '23

18 Feb '23

Greetings! We bring you a slew of releases this fine Saint Nicholas / Sinterklaas day. Six simultaneous releases has got to be some record. There’s one more record we broke this time, you’ll see below. In any case, updating is recommended due to security content: 3.7 - 3.12: gh-98739 <https://github.com/python/cpython/issues/98739>: Updated bundled libexpat to 2.5.0 to fix CVE-2022-43680 <https://nvd.nist.gov/vuln/detail/CVE-2022-43680> (heap use-after-free). 3.7 - 3.12: gh-98433 <https://github.com/python/cpython/issues/98433>: The IDNA codec decoder used on DNS hostnames by socket or asyncio related name resolution functions no longer involves a quadratic algorithm to fix CVE-2022-45061 <https://nvd.nist.gov/vuln/detail/CVE-2022-45061>. This prevents a potential CPU denial of service if an out-of-spec excessive length hostname involving bidirectional characters were decoded. Some protocols such as urllib http 3xx redirects potentially allow for an attacker to supply such a name. 3.7 - 3.12: gh-100001 <https://github.com/python/cpython/issues/100001>: python -m http.server no longer allows terminal control characters sent within a garbage request to be printed to the stderr server log. 3.8 - 3.12: gh-87604 <https://github.com/python/cpython/issues/87604>: Avoid publishing list of active per-interpreter audit hooks via the gc module. 3.9 - 3.10 (already released in 3.11+ before): gh-97514 <https://github.com/python/cpython/issues/97514>: On Linux the multiprocessing module returns to using filesystem backed unix domain sockets for communication with the forkserver process instead of the Linux abstract socket namespace. Only code that chooses to use the “forkserver” start method is affected. This prevents Linux CVE-2022-42919 <https://nvd.nist.gov/vuln/detail/CVE-2022-42919> (potential privilege escalation) as abstract sockets have no permissions and could allow any user on the system in the same network namespace (often the whole system) to inject code into the multiprocessing forkserver process. This was a potential privilege escalation. Filesystem based socket permissions restrict this to the forkserver process user as was the default in Python 3.8 and earlier. 3.7 - 3.10: gh-98517 <https://github.com/python/cpython/issues/98517>: Port XKCP’s fix for the buffer overflows in SHA-3 to fix CVE-2022-37454 <https://nvd.nist.gov/vuln/detail/CVE-2022-37454>. 3.7 - 3.9 (already released in 3.10+ before): gh-68966 <https://github.com/python/cpython/issues/68966>: The deprecated mailcap module now refuses to inject unsafe text (filenames, MIME types, parameters) into shell commands to address CVE-2015-20107 <https://nvd.nist.gov/vuln/detail/CVE-2015-20107>. Instead of using such text, it will warn and act as if a match was not found (or for test commands, as if the test failed). <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.12.0 alpha 3 Get it here, read the change log, sing a GPT-3-generated Sinterklaas song: https://www.python.org/downloads/release/python-3120a3/ <https://www.python.org/downloads/release/python-3120a3/> 216 new commits since 3.12.0 alpha 2 last month. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.11.1 Get it here, see the change log, read the recipe for quark soup: https://www.python.org/downloads/release/python-3111/ <https://www.python.org/downloads/release/python-3111/> A whopping 495 new commits since 3.11.0. This is a massive increase of changes comparing to 3.10 at the same stage in the release cycle: there were “only” 339 commits between 3.10.0 and 3.10.1. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.10.9 Get it here, read the change log, see circular patterns: https://www.python.org/downloads/release/python-3109/ <https://www.python.org/downloads/release/python-3109/> 165 new commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.9.16 Get it here, read the change log, consider upgrading to a newer version: https://www.python.org/downloads/release/python-3916/ <https://www.python.org/downloads/release/python-3916/> Security-only release with no binaries. 10 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.8.16 Get it here, see the change log, definitely upgrade to a newer version: https://www.python.org/downloads/release/python-3816/ <https://www.python.org/downloads/release/python-3816/> Security-only release with no binaries. 9 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.7.16 Get it here, read the change log, check PEP 537 <https://peps.python.org/pep-0537/>to confirm EOL is coming to this version in June 2023: https://www.python.org/downloads/release/python-3716/ <https://www.python.org/downloads/release/python-3716/> Security-only release with no binaries. 8 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ <https://www.python.org/psf/> Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Łukasz Langa @ambv <https://discuss.python.org/u/ambv> Thomas Wouters @thomas <https://discuss.python.org/u/thomas>

3 3

[RELEASE] Python 3.11.2, Python 3.10.10 and 3.12.0 alpha 5 are available
by Pablo Galindo Salgado 08 Feb '23

08 Feb '23

Hi everyone, I am happy to report that after solving some last-time problems we have a bunch of fresh releases for you: ### Python 3.12.0 alpha 5 Check the new alpha of 3.12 with some Star Trek vibes: https://www.python.org/downloads/release/python-3120a5/ 210 new commits since 3.12.0a4 last month ### Python 3.11.2 A shipment of bugfixes and security releases for the newest Python! https://www.python.org/downloads/release/python-3112/ 194 new commits since 3.11.1 ### Python 3.10.10 Your trusty Python3.10 just got more stable and secure! https://www.python.org/downloads/release/python-31010/ 131 new commits since 3.10.9 ## We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ Your friendly release team, Ned Deily @nad Steve Dower @steve.dower Pablo Galindo Salgado @pablogsal Łukasz Langa @ambv Thomas Wouters @thomas

1 0

Python Language Summit at PyCon US 2023 in Salt Lake City
by Łukasz Langa 08 Feb '23

08 Feb '23

We’re excited to announce that the signups for the Python Language Summit at PyCon US 2023 are now open. Full details at: https://us.pycon.org/2023/events/language-summit/ <https://us.pycon.org/2023/events/language-summit/> Just like in 2022, we are doing the Summit as an in-person event. We will be following the health and safety guidelines <https://us.pycon.org/2023/about/health-safety-guidelines/> of the wider conference. <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>TL;DR When: Wednesday, April 19, 2023 at 10:00am Where: Salt Palace Convention Center, room TBD Sign up to attend: https://forms.gle/YnWL1Hts6zDtdSgn7 <https://forms.gle/YnWL1Hts6zDtdSgn7> (closes March 5th, 2023 AoE) <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>Who can attend We welcome Python core developers and triage team members, active contributors to CPython and alternative Python implementations, and other community members with a topic to discuss with core developers. <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>Who can propose a discussion topic If you have discussion items; seeking consensus; awaiting decision on a PEP; needing help with your core dev work; or have specific questions that need answers from core developers, please submit a proposal. According to feedback, our audience prefers more discussions and shorter talks. In your proposal, please include the following: why is this topic relevant to the core developers; what is needed from core developers out of this topic. <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>How can I learn what’s happening at the event? Like in 2022, this year’s event will be covered by @AlexWaygood <https://discuss.python.org/u/alexwaygood>. A detailed summary of the event will be published on the Python Software Foundation Blog <https://pyfound.blogspot.com/>. <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>Is this event recorded? Can I watch the live stream? No, there will be no recording and no live stream available. If you’d like to participate in discussions, please sign up to attend. If you’d like to listen in, please wait for Alex’s blog posts after the Summit. <https://discuss.python.org/t/python-language-summit-at-pycon-us-2023-in-sal…>We hope you have a great event! Your Language Summit cat herding team, @ambv <https://discuss.python.org/u/ambv> & @Senthil <https://discuss.python.org/u/senthil>

1 0

PEP 638: Syntactic macros
by Mark Shannon 06 Feb '23

06 Feb '23

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25