Mailman 3 January 2023 - Python-Dev

Python multithreading without the GIL
by Sam Gross 31 Jul '23

31 Jul '23

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

24 48

Mixed Python/C debugging
by Skip Montanaro 26 Apr '23

26 Apr '23

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

[RELEASE] Python 3.11.1, 3.10.9, 3.9.16, 3.8.16, 3.7.16, and 3.12.0 alpha 3 are now available
by Łukasz Langa 18 Feb '23

18 Feb '23

Greetings! We bring you a slew of releases this fine Saint Nicholas / Sinterklaas day. Six simultaneous releases has got to be some record. There’s one more record we broke this time, you’ll see below. In any case, updating is recommended due to security content: 3.7 - 3.12: gh-98739 <https://github.com/python/cpython/issues/98739>: Updated bundled libexpat to 2.5.0 to fix CVE-2022-43680 <https://nvd.nist.gov/vuln/detail/CVE-2022-43680> (heap use-after-free). 3.7 - 3.12: gh-98433 <https://github.com/python/cpython/issues/98433>: The IDNA codec decoder used on DNS hostnames by socket or asyncio related name resolution functions no longer involves a quadratic algorithm to fix CVE-2022-45061 <https://nvd.nist.gov/vuln/detail/CVE-2022-45061>. This prevents a potential CPU denial of service if an out-of-spec excessive length hostname involving bidirectional characters were decoded. Some protocols such as urllib http 3xx redirects potentially allow for an attacker to supply such a name. 3.7 - 3.12: gh-100001 <https://github.com/python/cpython/issues/100001>: python -m http.server no longer allows terminal control characters sent within a garbage request to be printed to the stderr server log. 3.8 - 3.12: gh-87604 <https://github.com/python/cpython/issues/87604>: Avoid publishing list of active per-interpreter audit hooks via the gc module. 3.9 - 3.10 (already released in 3.11+ before): gh-97514 <https://github.com/python/cpython/issues/97514>: On Linux the multiprocessing module returns to using filesystem backed unix domain sockets for communication with the forkserver process instead of the Linux abstract socket namespace. Only code that chooses to use the “forkserver” start method is affected. This prevents Linux CVE-2022-42919 <https://nvd.nist.gov/vuln/detail/CVE-2022-42919> (potential privilege escalation) as abstract sockets have no permissions and could allow any user on the system in the same network namespace (often the whole system) to inject code into the multiprocessing forkserver process. This was a potential privilege escalation. Filesystem based socket permissions restrict this to the forkserver process user as was the default in Python 3.8 and earlier. 3.7 - 3.10: gh-98517 <https://github.com/python/cpython/issues/98517>: Port XKCP’s fix for the buffer overflows in SHA-3 to fix CVE-2022-37454 <https://nvd.nist.gov/vuln/detail/CVE-2022-37454>. 3.7 - 3.9 (already released in 3.10+ before): gh-68966 <https://github.com/python/cpython/issues/68966>: The deprecated mailcap module now refuses to inject unsafe text (filenames, MIME types, parameters) into shell commands to address CVE-2015-20107 <https://nvd.nist.gov/vuln/detail/CVE-2015-20107>. Instead of using such text, it will warn and act as if a match was not found (or for test commands, as if the test failed). <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.12.0 alpha 3 Get it here, read the change log, sing a GPT-3-generated Sinterklaas song: https://www.python.org/downloads/release/python-3120a3/ <https://www.python.org/downloads/release/python-3120a3/> 216 new commits since 3.12.0 alpha 2 last month. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.11.1 Get it here, see the change log, read the recipe for quark soup: https://www.python.org/downloads/release/python-3111/ <https://www.python.org/downloads/release/python-3111/> A whopping 495 new commits since 3.11.0. This is a massive increase of changes comparing to 3.10 at the same stage in the release cycle: there were “only” 339 commits between 3.10.0 and 3.10.1. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.10.9 Get it here, read the change log, see circular patterns: https://www.python.org/downloads/release/python-3109/ <https://www.python.org/downloads/release/python-3109/> 165 new commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.9.16 Get it here, read the change log, consider upgrading to a newer version: https://www.python.org/downloads/release/python-3916/ <https://www.python.org/downloads/release/python-3916/> Security-only release with no binaries. 10 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.8.16 Get it here, see the change log, definitely upgrade to a newer version: https://www.python.org/downloads/release/python-3816/ <https://www.python.org/downloads/release/python-3816/> Security-only release with no binaries. 9 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>Python 3.7.16 Get it here, read the change log, check PEP 537 <https://peps.python.org/pep-0537/>to confirm EOL is coming to this version in June 2023: https://www.python.org/downloads/release/python-3716/ <https://www.python.org/downloads/release/python-3716/> Security-only release with no binaries. 8 commits. <https://discuss.python.org/t/python-3-11-1-3-10-9-3-9-16-3-8-16-3-7-16-and-…>We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ <https://www.python.org/psf/> Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Łukasz Langa @ambv <https://discuss.python.org/u/ambv> Thomas Wouters @thomas <https://discuss.python.org/u/thomas>

3 3

PEP 638: Syntactic macros
by Mark Shannon 05 Feb '23

05 Feb '23

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

Feature Suggestion: "repeat" statement in loops
by Thomas Ratzke 28 Jan '23

28 Jan '23

Hi all, i would like to suggest the following Python feature. It naturally happens that one want's to repeat the current iteration of a for loop for example after an error happened. For this purpose, I usually set a flag and put a while loop inside my for loop. A simple "repeat" statement just like "continue" or "break" would make the code much more readable. This is my solution at the moment with A being checked: for _ in range(n): flag = True while flag: ... if A: flag = False # go to next iteration I would suggest the repeat statement in the following sense for _ in range(n): ... if not A: repeat # repeat current iteration Notice the "not" in the if clause. I am really looking forwars to hear your opinions. Best regards Thomas

6 5

Switching to Discourse
by Petr Viktorin 26 Jan '23

26 Jan '23

Hello, Currently development discussions are split between multiple communication channels, for example: - python-dev and discuss.python.org for design discussions, - GitHub Issues and Pull Requests for specific changes, - IRC, Discord and private chats for real-time discussions, - Topic-specific channels like typing-sig. While most of these serve different needs, there is too much overlap between python-dev and discuss.python.org. It seems that for most people, this situation is worse than sticking to either one platform – even if we don't go with that person's favorite. The discuss.python.org experiment has been going on for quite a while, and while the platform is not without its issues, we consider it a success. The Core Development category is busier than python-dev. According to staff, discuss.python.org is much easier to moderate.. If you're following python-dev but not discuss.python.org, you're missing out. The Steering Council would like to switch from python-dev to discuss.python.org. Practically, this means: - Moving the required PEP announcements to discuss.python.org - Moving discuss.python.org up in the devguide communications page (https://devguide.python.org/communication/) - And that's it? I imagine that the mailing list will stay around for continuing past discussion threads and for announcements, eventually switching to auto-reject incoming messages with a pointer to discuss.python.org. To be clear, discuss.python.org allows editing posts, which is frankly handy for typos and clarifications. Editing alone should not be used for adding new info -- we should cultivate a culture of being friendly to mail users & notification watchers. This probably bears repeating in a few places. We're aware not everyone wants to use the discuss.python.org website, but there are some ways to avoid it: - For new PEPs, you can point your RSS client to https://www.python.org/dev/peps/peps.rss – it's not e-mail, but many email clients have RSS support. You can also watch the Steering Council issues on GitHub (https://github.com/python/steering-council/issues/) for important questions and discussions. - You can use discuss.python.org's “mailing list mode” (which subscribes you to all new posts), possibly with filtering and/or categorizing messages locally. However, we would like to know if this will pose an undue burden to anyone, if there are workflows or usage problems that we are not aware of. As mentioned, this is something the Steering Council thinks is a good idea, but we want to make sure we're aware of all the impact when we make the final decision. – Petr, on behalf of the Steering Council

43 114

Requesting a review on a PR
by Wheeler Law 21 Jan '23

21 Jan '23

Greetings folks, I opened an issue[1] related to respecting the `http.client.HTTPConnection.debuglevel` value; I opened a subsequent PR[2] that fixes the issue back in November and I was wondering if I could get some feedback on it. Kind regards, Wheeler [1]: https://github.com/python/cpython/issues/99352 [2]: https://github.com/python/cpython/pull/99353

2 1

Exception compatibility with aliens
by Frederick Virchanza Gotham 19 Jan '23

19 Jan '23

I have sent this email to two mailing lists for programming language proposals, the one for C++ and the one for Python. If you reply to this email, please make sure you reply to both. You can see the mailing list archives for this month for each list here: C++ : https://lists.isocpp.org/std-proposals/2023/01/date.php Python : https://mail.python.org/archives/list/python-dev@python.org/2023/1/ Both C++ and Python have exception handling, however a C++ program which links with a Python library is unable to handle an exception thrown from Python. Is it conceivable in the future that the C++ Standards Committee and the Python Steering Council could cooperate and furnish each other with an exhaustive list of exceptions thrown from their respective standard libraries, so that both languages can implement compatibility with each other's exceptions? If such a list were to accompany each C++ standard and each Python Language Reference, then for example the C++ standard library could provide classes for handling Python exceptions: namespace std { struct alien_exception /* does not inherit from std::exception */ { enum class language : unsigned int { java=1u, csharp, python }; virtual language lang(void) const noexcept = 0; virtual char const *what(void) const noexcept = 0; }; namespace aliens { namespace java { struct exception : alien_exception { char const *getLocalizedMessage(void) const noexcept; char const *getMessage(void) const noexcept; std::stacktrace getStackTrace(void) const noexcept; char const *what(void) const noexcept override { return this->getMessage(); } language lang(void) const noexcept override { return language::java; } }; } namespace python { struct exception : alien_exception { char const *const *notes(void) const noexcept; /* null terminated */ std::stacktrace traceback() const noexcept; char const *what(void) const noexcept override { return this->notes()[0u]; } language lang(void) const noexcept override { return language::python; } }; } namespace python { struct BaseException : exception {}; struct BaseExceptionGroup : BaseException {}; struct GeneratorExit : BaseException {}; struct KeyboardInterrupt : BaseException {}; struct SystemExit : BaseException {}; struct Exception : BaseException {}; struct ArithmeticError : Exception {}; struct FloatingPointError : ArithmeticError {}; struct OverflowError : ArithmeticError {}; struct ZeroDivisionError : ArithmeticError {}; struct AssertionError : Exception {}; struct AttributeError : Exception {}; struct BufferError : Exception {}; struct EOFError : Exception {}; struct ExceptionGroupBaseExceptionGroup : Exception {}; struct ImportError : Exception {}; struct ModuleNotFoundError : ImportError {}; struct LookupError : Exception {}; struct IndexError : LookupError {}; struct KeyError : LookupError {}; struct MemoryError : Exception {}; struct NameError : Exception {}; struct UnboundLocalError : NameError {}; struct OSError : Exception {}; struct BlockingIOError : OSError {}; struct ChildProcessError : OSError {}; struct ConnectionError : OSError {}; struct BrokenPipeError : ConnectionError {}; struct ConnectionAbortedError : ConnectionError {}; struct ConnectionRefusedError : ConnectionError {}; struct ConnectionResetError : ConnectionError {}; struct FileExistsError : OSError {}; struct FileNotFoundError : OSError {}; struct InterruptedError : OSError {}; struct IsADirectoryError : OSError {}; struct NotADirectoryError : OSError {}; struct PermissionError : OSError {}; struct ProcessLookupError : OSError {}; struct TimeoutError : OSError {}; struct ReferenceError : Exception {}; struct RuntimeError : Exception {}; struct NotImplementedError : RuntimeError {}; struct RecursionError : RuntimeError {}; struct StopAsyncIteration : Exception {}; struct StopIteration : Exception {}; struct SyntaxError : Exception {}; struct IndentationError : SyntaxError {}; struct TabError : IndentationError {}; struct SystemError : Exception {}; struct TypeError : Exception {}; struct ValueError : Exception {}; struct UnicodeError : ValueError {}; struct UnicodeDecodeError : UnicodeError {}; struct UnicodeEncodeError : UnicodeError {}; struct UnicodeTranslateError : UnicodeError {}; struct Warning : Exception {}; struct BytesWarning : Warning {}; struct DeprecationWarning : Warning {}; struct EncodingWarning : Warning {}; struct FutureWarning : Warning {}; struct ImportWarning : Warning {}; struct PendingDeprecationWarning : Warning {}; struct ResourceWarning : Warning {}; struct RuntimeWarning : Warning {}; struct SyntaxWarning : Warning {}; struct UnicodeWarning : Warning {}; struct UserWarning : Warning {}; } // close namespace 'python' } // close namespace 'aliens' class forced_stack_unwind {}; } // close namespace 'std' extern int Some_Func_In_Python_Library(void) { return 7; /* or throws an exception */ } int main(void) { try { Some_Func_In_Python_Library(); } catch ( std::aliens::python::FloatingPointError const & ) { // The specific Python exception we want to handle } catch ( std::aliens::python::exception const & ) { // Any exception thrown from Python } catch ( std::alien_exception const &) { // Maybe the Python library linked with an // Eiffel library that threw an exception } catch ( std::forced_stack_unwind ) { } }

3 2

Re: [std-proposals] Exception compatibility with aliens
by Frederick Virchanza Gotham 18 Jan '23

18 Jan '23

On Wed, Jan 18, 2023 at 3:07 PM Jason McKesson wrote: > > Also, this proposal seems to be missing the biggest issue with > cross-language exception handling: the fact that you can't throw > exceptions across languages. The only thing you *can* do is catch > exceptions on the source language end, convert them into some data > packet, and throw a different exception on the destination language > side. Yes this is what I had in mind. Behind the scenes, the C++ compiler would catch the Python exception and then throw something that C++ can deal with "such as std::aliens::python::exception".

3 2