Mailman 3 February 2022 - Python-Dev

Custom memory profiler and PyTraceMalloc_Track
by Sümer Cip Feb. 15, 2022

Feb. 15, 2022

Hi everyone, I would like to ask a question about an issue that we faced regarding profiling memory usage: We have implemented a custom memory profiler using `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, we are facing an issue with numpy: numpy seems to have its own memory allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc about the allocation. A simple search on GitHub reveals there are more projects using this approach: https://github.com/… [View More]

2 3

Request for feedback: pathlib.AbstractPath prototype
by Barney Gale Feb. 11, 2022

Feb. 11, 2022

Over the last couple of years I've been tidying up the pathlib internals, with a view towards adding an AbstractPath class to the hierarchy. Users would be able to subclass AbstractPath to implement other kinds of filesystems: s3, google cloud storage, pandas, ftp, git, zip and tar files, etc. By implementing some abstract methods (stat(), iterdir(), open(), etc) they'd benefit from a number of derived methods (is_dir(), glob(), read_text(), etc). There's already a healthy ecosystem of PyPI … [View More]

3 8

Summary of Python tracker Issues
by Python tracker Feb. 11, 2022

Feb. 11, 2022

ACTIVITY SUMMARY (2022-02-04 - 2022-02-11) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7178 (+34) closed 51272 (+50) total 58450 (+84) Open issues with patches: 2912 Issues opened (63) ================== #42548: debugger stops at breakpoint of `pass` that is not actually re https://bugs.python.org/issue42548 reopened by iritkatriel #44006: … [View More]

1 0

urllib: addressing inflexibility in scheme-based joining
by Lincoln Auster Feb. 11, 2022

Feb. 11, 2022

Hi, This is a follow-up RFC on PR #30520 (BPO 46337) with regard to urllib's potentially inflexible scheme-based URL manipulation. In my ticket, I proposed supplementing the hard-coded uses_* scheme lists that control URL parsing and joining behavior with an optional enum set that may override behavior when calling a function that would require it. (More detail is in that BPO and code is obviously in that PR.) It's been about a month since I wrote that PR, and it was marked stale a day or two … [View More]

2 1

How to build cpython for x86_64 target on M1 Mac?
by jeff.pek＠autodesk.com Feb. 10, 2022

Feb. 10, 2022

Hi - Newbie post here... We need to be able to produce both arm64 and x86_64 builds of cpython on a single machine. I can see that, by default, it chooses the target architecture based on the current running architecture. The docs say that we can target one platform from the other, but it's not specific about how to do that? I figure this is some sort of 'configure' option -- maybe "--build"? -- but so far nothing I've tried has worked. Can someone help point us in the right direction? What'… [View More]

3 2

PEP 677 (Callable Type Syntax): Rejection notice.
by Gregory P. Smith Feb. 10, 2022

Feb. 10, 2022

Thank you PEP authors for producing a very well written and laid out PEP. That made it easy to understand the proposal, rationale, and alternatives considered. We agree that the existing typing.Callable syntax, while capable, is less than ideal for humans. That said, we deliberated last week and ultimately decided to reject PEP-677 Callable Type Syntax. Why? Several considerations led us here: 1. We feel we need to be cautious when introducing new syntax. Our new parser presents … [View More]understandably exciting opportunities but we don’t want its existence to mean we add new syntax easily. A feature for use only in a fraction of type annotations, not something every Python user uses, did not feel like a strong enough reason to justify the complexity needed to parse this new syntax and provide meaningful error messages. Not only code complexity, humans are also parsers that must look forwards and backwards. 2. While the current Callable[x, y] syntax is not loved, it does work. This PEP isn’t enabling authors to express anything they cannot already. The PEP explicitly chose be conservative and not allow more syntax to express features going beyond what Callable supports. We applaud that decision, starting simple is good. But we can imagine a future where the syntax would desire to be expanded upon. 3. In line with past SC guidance, we acknowledge challenges when syntax desires do not align between typing and Python itself. Each time we add syntax solely for typing it shifts us further in the direction of typing being its own mini-language so we aim to tread lightly in what gets added here. Adopting PEP 677 would lock us into its syntax forever, potentially preventing other future syntax directions. 4. We did not like the visual and cognitive consequence of multiple `->` tokens in a def. Especially when code is not formatted nicely. Though we admit the correlation between Python typing and formatter users is high. ## Recommendation for future syntax enhancements: When proposing a syntax change, low complexity is better. While not always possible, it’s ideal if it could still be described using our old <=3.8 parser. It is important to have in mind that adding syntax that only our modern PEG parser can handle could lead to greater cognitive load and external tooling implementation costs. This should not be read as a ban on PEG-only syntax, we just think it should be used for broadly applicable features or else be relatively unintrusive. Thanks, The 3.11 Python Steering Council [View Less]

2 2

Please update Cython *before* introcuding C API incompatible changes in Python
by Victor Stinner Feb. 10, 2022

Feb. 10, 2022

Hi, It became more and more common that a C API incompatible change introduced in Python breaks Cython and all Python projects using Cython (ex: numpy). Hopefully, usually only some projects using Cython are broken, not all of them. Some of you may remind the PEP 590 (vectorcall) implementation which removed the tp_print member of PyTypeObject in Python 3.8. This change broke a large number of projects using Cython. Cython was updated, the member removal was reverted, etc. It took a few weeks … [View More]to solve the issue. It was unpleasant to have to revert the change (add again tp_print). The tp_print member was removed again in Python 3.9. IMO the PEP 570 (positional-only arguments) implementation went even worse when it added a parameter to PyCode_New(). Cython was modified and released with a change, and *then* the PyCode_New() change was reverted in Python (if I recall correctly)! It was very confusing. There were multiple communication issues between Python and Cython. The first Cython fix was incorrect, etc. Last December, another Python change related to exceptions (bpo-45711) broke Cython on purpose. A commit message says "Add to what's new, because this change breaks things like Cython". It's kind of annoying that the each time we have a long period of time (several weeks) when Cython is unusable and there is pressure on Cython get a fix and then release a new version. At Red Hat, we are rebuilding Python frequently with an up to date Python 3.11: right now, numpy fails to build because of that. -- I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is released - (3) If possible, wait until numpy is released with regenerated Cython code - (4) Introduce the incompatible change in Python Note: Fedora doesn't need (3) since we always regenerated Cython code in numpy. None of these C API incompatible changes are wrong. For each change, there are good reasons to introduce them. I'm only proposing to change *how* we introduce them to avoid a long period of time when Cython and/or numpy are not usable on the development version of Python. -- If you want to test if a C API change is going to break Cython or numpy, you can try my tool: https://github.com/vstinner/pythonci/ Usage: build a modified Python with your change, and then run pythonci with it. pythonci uses pinned version of all dependencies to try to be more reproducible. I update them manually infrequently (see pythonci/requirements.txt). -- Right now, I propose to revert the incompatible change related to exceptions until Cython is prepared for it: https://bugs.python.org/issue45711#msg412264 Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

14 49

Suggestion: a little language for type definitions
by jack.jansen＠cwi.nl Feb. 10, 2022

Feb. 10, 2022

I posted this suggestion earlier in the callable type syntax discussion, at which point it was completely ignored. Possibly because it’s a really stupid idea, but let me post it again on the off chance that it isn’t a stupid idea but was overlooked. > If I can make a wild suggestion: why not create a little language for type specifications? > > If you look at other programming languages you’ll see that the “type definition sub-language” is often completely different from the “… [View More]

15 35

Replace debug runtime checks in release mode with assertions in debug mode
by Victor Stinner Feb. 9, 2022

Feb. 9, 2022

Hi, I would like to replace runtime checks on C function arguments with assertions. For example, replace: PyObject * PyDict_GetItemWithError(PyObject *op, PyObject *key) { if (!PyDict_Check(op)) { PyErr_BadInternalCall(); return NULL; } ... } with: PyObject * PyDict_GetItemWithError(PyObject *op, PyObject *key) { assert(PyDict_Check(op)); ... } I'm not asking to remove all of them. For example, PyList_GetItem(list, index) will continuing raising an … [View More]IndexError if the index is out of range. I'm asking to replace runtime checks with assertions when the C API is "obviously" misused: replace PyErr_BadInternalCall(), _Py_CheckFunctionResult() and _Py_CheckSlotResult() with assertions. The exact scope should be defined. -- Python tries to be nice to C extensions authors and calls PyErr_BadInternalCall() to emit a nice SystemError exception when the C API is misused. There are many sanity checks run at runtime. We pay the cost of these runtime checks even if a C extension is well written (has no bug). In Python 3.6, I added the _Py_CheckFunctionResult() function to check the result of any function call. A function must return a non-NULL object and not raises an exception, or return NULL and raises an exception. Otherwise, SystemError is raised. For example, PyObject_Call() uses it to check the PyTypeObject.tp_call result. In Python 3.10, I added a similar _Py_CheckSlotResult() function to check the result of slot functions. For example, PyObject_Size() checks the result of the PyTypeObject.tp_as_sequence.sq_length slot function. -- I modified Python 3.8 to be able to use a debug Python build as a drop-in replacement of a Python release build: debug and release builds are now ABI compatible. It is no longer needed to rebuild C extensions in debug mode. My first goal was to be able to use gdb on a Python binary built with -O0: https://developers.redhat.com/articles/2021/09/08/debugging-python-c-extens… A debug Python build adds many other debug features useful to develop C extensions: https://docs.python.org/dev/using/configure.html#python-debug-build IMO with Python 3.11, it's now easy enough to get a Python debug build to develop or debug your C extensions. End users should not have to pay the price of these runtime checks. On most Linux distribution, a Python debug build is available as "python3-debug" executable. To check if you are using a debug build, check if the sys.gettotalrefcount() function is available. By the way, it is also possible to build Python in release mode with assertions using "./configure --with-assertions": https://docs.python.org/dev/using/configure.html#cmdoption-with-assertions So, what do you think? -- In 2019, I already proposed this idea on the bug tracker, but I abandoned my idea: https://bugs.python.org/issue37406 Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

8 11

RFC on PEP 675: Arbitrary Literal Strings
by Pradeep Kumar Srinivasan Feb. 8, 2022

Feb. 8, 2022

PEP 675 [1] introduces a supertype for precise literal string types, such as `Literal["foo"]`, called `LiteralString`. The PEP allows libraries to distinguish the type of command strings from data. Powerful, command-executing libraries try to prevent security vulnerabilities by accepting arguments separately from the SQL query or shell command. However, these libraries have no way to prevent programmers from, say, using f-strings to embed arguments within the command string, which can allow … [View More]

3 3