Mailman 3 March 2023 - Python-Dev

Python multithreading without the GIL
by Sam Gross 31 Jul '23

31 Jul '23

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

24 48

Mixed Python/C debugging
by Skip Montanaro 26 Apr '23

26 Apr '23

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

Debugging of native extensions on windows
by Rokas Kupstys 14 Mar '23

14 Mar '23

Hello! I am dropping this mail to bring up an issue which cost me three good evenings of time. Now that i figured it out i believe it is quite a serious usability problem. Gist of the problem: i have some C++ code wrapped with SWIG, so a native extension. As with all software - there was a bug. However, no matter what i did - i could not debug it in a native debugger. I ran ".venv/Scripts/python.exe script.py" in a native debugger and breakpoints would not be hit, application would crash eventually. This was especially confusing, because exact same setup worked just fine on linux. I eventually stumbled on to process list showing ".venv/Scripts/python.exe" having spawned a subprocess... Which led me to "PC/launcher.c" which is what ".venv/Scripts/python.exe" really is. I cant find much information about this behavior in documentation even after the fact. All in all, this was quite confusing. So now every time i want to debug a native extension i have to start a program and then attach a debugger to it, instead of just hitting "Debug" button in IDE. It gets worse if crash happens immediately, which means i have to resort to things like adding a message box somewhere to block execution and give me enough time to attach the debugger. It works in the end, but user experience is really not great. But whats worse - this is such a non-obvious behavior that many more people may trip on it and waste their time. Documenting this behavior would be of little help too, as there is no clear path from the issue to the documentation on the matter... So there it is. I am sure it is the way it is for a good reason. However, this is a very error-prone process which is likely to waste people's time. So maybe this behavior could be reconsidered? Or maybe there is a solution already, which escaped me? -- -- Rokas Kupstys

4 5

Re: please consider changing --enable-unicode default to ucs4
by Jonathan Benson 14 Mar '23

14 Mar '23

Sent from my iPhone

2 1

Python 3.12.0 alpha 6 released
by Thomas Wouters 07 Mar '23

07 Mar '23

I'm pleased to announce the release of Python 3.12 alpha 6. https://www.python.org/downloads/release/python-3120a6/ *This is an early developer preview of Python 3.12.* Major new features of the 3.12 series, compared to 3.11 Python 3.12 is still in development. This release, 3.12.0a6 is the sixth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2023-05-08) and, if necessary, may be modified or deleted up until the release candidate phase (2023-07-31). Please keep in mind that this is a preview release and its use is not recommended for production environments. Many new features for Python 3.12 are still being planned and written. Among the new major new features and changes so far: - Even more improved error messages. More exceptions potentially caused by typos now make suggestions to the user. - Support for the Linux perf profiler to report Python function names in traces. - The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623 <https://peps.python.org/pep-0623>. - In the unittest module, a number of long deprecated methods and classes were removed. (They had been deprecated since Python 3.1 or 3.2). - The deprecated smtpd and distutils modules have been removed (see PEP 594 <https://peps.python.org/pep-0594> and PEP 632 <https://peps.python.org/pep-0632>. The setuptools package (installed by default in virtualenvs and many other places) continues to provide the distutils module. - A number of other old, broken and deprecated functions, classes and methods have been removed. - Invalid backslash escape sequences in strings now warn with SyntaxWarning instead of DeprecationWarning, making them more visible. (They will become syntax errors in the future.) - The internal representation of integers has changed in preparation for performance enhancements. (This should not affect most users as it is an internal detail, but it may cause problems for Cython-generated code.) - (Hey, fellow core developer, if a feature you find important is missing from this list, let Thomas know <thomas(a)python.org>.) For more details on the changes to Python 3.12, see What's new in Python 3.12. The next pre-release of Python 3.12 will be 3.12.0a7, currently scheduled for 2023-04-03. More resources Online Documentation <https://docs.python.org/3.12/>. PEP 693 <https://peps.python.org/pep-0693>, the Python 3.12 Release Schedule. Report bugs via GitHub Issues <https://github.com/python/cpython/issues>. Help fund Python and its community <https://www.python.org/psf/donations/>. And now for something completely different Let me not to the marriage of true minds > Admit impediments. Love is not love > Which alters when it alteration finds, > Or bends with the remover to remove: > O, no! it is an ever-fixed mark, > That looks on tempests and is never shaken; > It is the star to every wandering bark, > Whose worth’s unknown, although his height be taken. > Love’s not Time’s fool, though rosy lips and cheeks > Within his bending sickle’s compass come; > Love alters not with his brief hours and weeks, > But bears it out even to the edge of doom. > If this be error, and upon me prov’d, > I never writ, nor no man ever lov’d. *Sonnet 116*, by William Shakespeare. Enjoy the new releases Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Regards from unexpectedly chilly California, Your release team, Thomas Wouters Ned Deily Steve Dower -- Thomas Wouters <thomas(a)python.org>

1 0