Mailman 3 March 2020 - Python-Dev

Re: Proliferation of tstate arguments.
by Anders Munch 22 Mar '20

22 Mar '20

Chris Angelico [mailto:rosuav@gmail.com]: > And by that logic, globals are even more capable. I don't understand your > point. Isn't the purpose of the tstate parameters to avoid the problem of > being unable to have multiple tstates within the same OS thread? I think I've > missed something here. The point is that because threads can't preempt themselves, this: /*1*/ { Py_something(other_tstate); } and this: /*2*/ { PyInterpreterState* old_tstate = tstate; tstate = other_state; Py_something(); tstate = old_tstate; } is effectively equivalent, provided tstate is thread-local. Both will work equally well from the hypothetical C callback that wants to use a different subinterpreter. That wouldn't be true if tstate was process-wide, because that would be a race condition, some other thread might change tstate. But if tstate is thread-local, there's no race condition. Obviously /*1*/ is cleaner code than /*2*/, but /*2*/ is perfectly sound all the same. regards, Anders

7 8

Changing layout of f_localsplus in frame objects
by Skip Montanaro 21 Mar '20

21 Mar '20

(Apologies if you're seeing this twice. I first posted to the discourse instance.) I first worked on a register-based virtual machine in the 1.5.2 timeframe. That was before PEP 227 (closures) landed. Prior to that, local variables and the stack were contiguous in the f_localsplus array. Based on a comment at the time from Tim Peters (which I recall but can no longer find), the number of required registers won’t be any greater than the maximum extent of the stack. (It makes sense, and worked for me at the time.) That allowed me to repurpose the stack space as my registers and treat the full f_localsplus array as a single large “register file,” eliminating LOAD_FAST_REG and STORE_FAST_REG instructions completely, which I believe was a major win. I want to rearrange the current f_localsplus to make locals and stack neighbors again. My first look at the code suggested there is no good reason it can’t be done, but figured Jeremy Hylton must have had a good reason to prefer the current layout which places cells and frees in the middle. While the necessary changes didn’t look extensive, they did look a bit tedious to get right, which I have confirmed. My first attempt to reorganize f_localsplus from locals/cells/frees/stack to locals/stack/cells/frees has been an abysmal failure. I’ve found a couple mistakes and corrected them. Implementing those corrections caused the types of failures to change (convincing me they were necessary), but did not eliminate them entirely (so, necessary, but not sufficient). I’ve clearly missed something. I’m also fairly ignorant about recent changes to the language (big understatement), so thought that before going any further, I would see if anyone with better current knowledge of frame objects knew of a reason why my desired layout change wouldn’t work. Skip

2 3

Proliferation of tstate arguments.
by Mark Shannon 21 Mar '20

21 Mar '20

Hi, There seems to be a proliferation of `PyThreadState *tstate` arguments being added to API and internal functions. These changes are listed under https://bugs.python.org/issue38644. I think that these changes are misguided. The desired results can be achieved more reliably and more simply in other ways. The changes add bulk to the C-API and may hurt performance. These changes are also causing a lot of churn and merge conflicts (for me at least). So can we please stop making these changes, at least now? Changes on this scale merit a PEP and proper discussion, rather than being added piecemeal without proper review. Cheers, Mark.

8 16

(no subject)
by Dennis Sweeney 20 Mar '20

20 Mar '20

Thanks for the review! > In short, I propose: > def cutprefix(self: str, prefix: str, /) -> str: > if self.startswith(prefix) and prefix: > return self[len(prefix):] > else: > return self > > I call startswith() before testing if pre is non-empty to inherit of > startswith() input type validation. For example, "a".startswith(b'x') > raises a TypeError. This still erroneously accepts tuples and and would return return str subclasses unchanged. If we want to make the Python be the spec with accuracy about type-checking, then perhaps we want: def cutprefix(self: str, prefix: str, /) -> str: if not isinstance(prefix, str): raise TypeError(f'cutprefix() argument must be str, ' f'not {type(prefix).__qualname__}') self = str(self) prefix = str(prefix) if self.startswith(prefix): return self[len(prefix):] else: return self I like the idea to always require these to return the unmodified string. Is there a reason this isn't specified by the ``replace()`` or ``strip`` methods? For accepting multiple prefixes, I can't tell if there's a consensus about whether ``s = s.cutprefix("a", "b", "c")`` should be the same as for prefix in ["a", "b", "c"]: s = s.cutprefix(prefix) or for prefix in ["a", "b", "c"]: if s.startwith(prefix): s = s.cutprefix(prefix) break The latter seems to be harder for users to implement through other means, and it's the behavior that test_concurrent_futures.py has implemented now, so maybe that's what we want. Also, it seems more elegant to me to accept variadic arguments, rather than a single tuple of arguments. Is it worth it to match the related-but-not-the-same API of "startswith" if it makes for uglier Python? My gut reaction is to prefer the varargs, but maybe someone has a different perspective. I can submit a revision to the PEP with some changes tomorrow.

1 0

Requesting review on PEP 615 C Extension Reference Implementation
by Paul Ganssle 20 Mar '20

20 Mar '20

Hi all, The past few weeks I've been working on adding a C extension to the reference implementation for PEP 615 (Support for the IANA Time Zone Database in the Standard Library - PEP link <https://www.python.org/dev/peps/pep-0615/>), but I've had some trouble inducing anyone to review the code <https://twitter.com/pganssle/status/1240019701356539906?s=20>. There's about 2200 lines of C code there that's passing all the tests for the pure Python implementation (which has 100% code coverage) - would anyone be interested in taking a look before I merge it into the reference implementation repo? Any little bit of review (including incremental reviews) would help: https://github.com/pganssle/zoneinfo/pull/15 Note: You'll need to click "Load diff" on the zoneinfo_module.c file to see the actual code, because the diff is so large. The stuff that's immediately visible in the diff is the more straightforward code. I'll note that I am also happy to accept review comments about other parts of the repo, not just the C code, but the C code is a priority since errors there tend to be less forgiving. Thanks! Paul

1 0

[bpo-22699] Cross-compiling and fixing sysconfig
by Steve Dower 20 Mar '20

20 Mar '20

So over on https://bugs.python.org/issue22699 I've been talking to myself as I figure out all the ways that cross-compiling (on Ubuntu, targeting another Linux distro via an SDK) fails. In short, it's either because sysconfig can't provide details about any CPython install other than the currently running one, or it's because our setup.py (for building the CPython extension modules) uses sysconfig. Either way, I'm not about to propose a rewrite to fix either of them without finding those who are most involved in these areas. Please come join me on the bug. Thanks, Steve

3 3

Summary of Python tracker Issues
by Python tracker 20 Mar '20

20 Mar '20

ACTIVITY SUMMARY (2020-03-13 - 2020-03-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7358 ( +1) closed 44420 (+70) total 51778 (+71) Open issues with patches: 2883 Issues opened (44) ================== #39576: Surprising MemoryError in `decimal` with MAX_PREC https://bugs.python.org/issue39576 reopened by pablogsal #39958: Deadlock in _PyInterpreterState_DeleteExceptMain with HEAD_LOC https://bugs.python.org/issue39958 opened by Atle Solbakken #39959: (Possible) bug on multiprocessing.shared_memory https://bugs.python.org/issue39959 opened by dxflores #39960: Using typename.__setattr__ in extension type with Py_TPFLAGS_H https://bugs.python.org/issue39960 opened by Matthias Braun #39961: warning: this use of "defined" may not be portable (Mac OS) https://bugs.python.org/issue39961 opened by clem.wang #39963: Subclassing slice objects https://bugs.python.org/issue39963 opened by mariode1 #39966: mock 3.9 bug: Wrapped objects without __bool__ raise exception https://bugs.python.org/issue39966 opened by aviso #39967: bdb calls linecache.checkcache, resulting in source being diff https://bugs.python.org/issue39967 opened by blueyed #39971: Error on documentation - Quick fix. https://bugs.python.org/issue39971 opened by dxflores #39975: Group of commands running in Python 3.7.6 Shell, but failing a https://bugs.python.org/issue39975 opened by rosarion #39976: Add "**other_popen_kwargs" to subprocess API signatures in doc https://bugs.python.org/issue39976 opened by ncoghlan #39978: Vectorcall implementation should conform to PEP 590. https://bugs.python.org/issue39978 opened by Mark.Shannon #39981: Default values for AST Nodes https://bugs.python.org/issue39981 opened by BTaskaya #39982: FreeBSD: SCTP tests of test_socket fails on AMD64 FreeBSD Shar https://bugs.python.org/issue39982 opened by vstinner #39983: test.regrtest: test marked as failed (env changed), but no war https://bugs.python.org/issue39983 opened by vstinner #39984: Move some ceval fields from _PyRuntime.ceval to PyInterpreterS https://bugs.python.org/issue39984 opened by vstinner #39986: test_os / test_listdir failed as root-directory changed during https://bugs.python.org/issue39986 opened by Matthias Braun #39989: Output closing parenthesis in ast.dump() on separate line https://bugs.python.org/issue39989 opened by serhiy.storchaka #39990: help should evaluate forward reference https://bugs.python.org/issue39990 opened by McSinyx #39994: Redundant code in pprint module. https://bugs.python.org/issue39994 opened by palakjha #39995: test_concurrent_futures: ProcessPoolSpawnExecutorDeadlockTest. https://bugs.python.org/issue39995 opened by vstinner #39996: test_multiprocessing_fork hangs on AMD64 FreeBSD Shared 3.x https://bugs.python.org/issue39996 opened by vstinner #39999: Fix some issues with AST node classes https://bugs.python.org/issue39999 opened by serhiy.storchaka #40001: ignore errors in SimpleCookie https://bugs.python.org/issue40001 opened by aviramha #40002: Cookie load error inconsistency https://bugs.python.org/issue40002 opened by bar.harel #40003: test.regrtest: add an option to run test.bisect_cmd on failed https://bugs.python.org/issue40003 opened by vstinner #40005: Getting different result in python 2.7 and 3.7. https://bugs.python.org/issue40005 opened by Bharatsolanki #40006: enum: Add documentation for _create_pseudo_member_ and composi https://bugs.python.org/issue40006 opened by cool-RR #40007: An attempt to make asyncio.transport.writelines (selector) use https://bugs.python.org/issue40007 opened by tzickel #40010: Inefficient signal handling in multithreaded applications https://bugs.python.org/issue40010 opened by vstinner #40011: Tkinter widget events are of type Tuple https://bugs.python.org/issue40011 opened by Hênio Tierra Sampaio #40012: Avoid Python 2 documentation to appear in Web search results https://bugs.python.org/issue40012 opened by bittner #40013: CSV DictReader parameter documentation https://bugs.python.org/issue40013 opened by Moshe Sambol #40014: os.getgrouplist() can fail on macOS https://bugs.python.org/issue40014 opened by corona10 #40016: Clarify flag case in `re` module https://bugs.python.org/issue40016 opened by cool-RR #40017: Please support CLOCK_TAI in the time module. https://bugs.python.org/issue40017 opened by r3owen #40018: test_ssl fails with OpenSSL 1.1.1e https://bugs.python.org/issue40018 opened by benjamin.peterson #40019: test_gdb should better detect when Python is optimized https://bugs.python.org/issue40019 opened by vstinner #40020: growable_comment_array_add leaks, causes crash https://bugs.python.org/issue40020 opened by Alexander Riccio #40022: 关于列表的基础算法问题 https://bugs.python.org/issue40022 opened by yepan Li #40023: os.writev and socket.sendmsg return value are not ideal https://bugs.python.org/issue40023 opened by tzickel #40024: Add _PyModule_AddType private helper function https://bugs.python.org/issue40024 opened by corona10 #40025: enum: _generate_next_value_ is not called if its definition oc https://bugs.python.org/issue40025 opened by edd07 #40026: Create render_*_diff variants to the *_diff functions in diffl https://bugs.python.org/issue40026 opened by ruoso Most recent 15 issues with no replies (15) ========================================== #40026: Create render_*_diff variants to the *_diff functions in diffl https://bugs.python.org/issue40026 #40025: enum: _generate_next_value_ is not called if its definition oc https://bugs.python.org/issue40025 #40024: Add _PyModule_AddType private helper function https://bugs.python.org/issue40024 #40023: os.writev and socket.sendmsg return value are not ideal https://bugs.python.org/issue40023 #40022: 关于列表的基础算法问题 https://bugs.python.org/issue40022 #40013: CSV DictReader parameter documentation https://bugs.python.org/issue40013 #40012: Avoid Python 2 documentation to appear in Web search results https://bugs.python.org/issue40012 #40011: Tkinter widget events are of type Tuple https://bugs.python.org/issue40011 #40006: enum: Add documentation for _create_pseudo_member_ and composi https://bugs.python.org/issue40006 #40003: test.regrtest: add an option to run test.bisect_cmd on failed https://bugs.python.org/issue40003 #40001: ignore errors in SimpleCookie https://bugs.python.org/issue40001 #39994: Redundant code in pprint module. https://bugs.python.org/issue39994 #39986: test_os / test_listdir failed as root-directory changed during https://bugs.python.org/issue39986 #39983: test.regrtest: test marked as failed (env changed), but no war https://bugs.python.org/issue39983 #39981: Default values for AST Nodes https://bugs.python.org/issue39981 Most recent 15 issues waiting for review (15) ============================================= #40024: Add _PyModule_AddType private helper function https://bugs.python.org/issue40024 #40020: growable_comment_array_add leaks, causes crash https://bugs.python.org/issue40020 #40019: test_gdb should better detect when Python is optimized https://bugs.python.org/issue40019 #40016: Clarify flag case in `re` module https://bugs.python.org/issue40016 #40014: os.getgrouplist() can fail on macOS https://bugs.python.org/issue40014 #40010: Inefficient signal handling in multithreaded applications https://bugs.python.org/issue40010 #40002: Cookie load error inconsistency https://bugs.python.org/issue40002 #40001: ignore errors in SimpleCookie https://bugs.python.org/issue40001 #39999: Fix some issues with AST node classes https://bugs.python.org/issue39999 #39994: Redundant code in pprint module. https://bugs.python.org/issue39994 #39989: Output closing parenthesis in ast.dump() on separate line https://bugs.python.org/issue39989 #39986: test_os / test_listdir failed as root-directory changed during https://bugs.python.org/issue39986 #39984: Move some ceval fields from _PyRuntime.ceval to PyInterpreterS https://bugs.python.org/issue39984 #39981: Default values for AST Nodes https://bugs.python.org/issue39981 #39967: bdb calls linecache.checkcache, resulting in source being diff https://bugs.python.org/issue39967 Top 10 most discussed issues (10) ================================= #22699: Module source files not found when cross-compiling https://bugs.python.org/issue22699 11 msgs #39939: Add str methods to remove prefixes or suffixes https://bugs.python.org/issue39939 10 msgs #10572: Move test sub-packages to Lib/test https://bugs.python.org/issue10572 8 msgs #39879: Update language reference to specify that dict is insertion-or https://bugs.python.org/issue39879 8 msgs #39984: Move some ceval fields from _PyRuntime.ceval to PyInterpreterS https://bugs.python.org/issue39984 8 msgs #38938: Possible performance improvement for heaqq.merge() https://bugs.python.org/issue38938 7 msgs #39982: FreeBSD: SCTP tests of test_socket fails on AMD64 FreeBSD Shar https://bugs.python.org/issue39982 7 msgs #40016: Clarify flag case in `re` module https://bugs.python.org/issue40016 7 msgs #1635741: Py_Finalize() doesn't clear all Python objects at exit https://bugs.python.org/issue1635741 7 msgs #25024: Allow passing "delete=False" to TemporaryDirectory https://bugs.python.org/issue25024 6 msgs Issues closed (69) ================== #20125: We need a good replacement for direct use of load_module(), po https://bugs.python.org/issue20125 closed by brett.cannon #20613: Wrong line information in bytecode generated by compiler modul https://bugs.python.org/issue20613 closed by brett.cannon #20890: Miscellaneous PEP 101 additions https://bugs.python.org/issue20890 closed by brett.cannon #20986: test_startup_imports fails in test_site when executed inside v https://bugs.python.org/issue20986 closed by vstinner #21087: imp.frozen_init() incorrectly removes module during reload https://bugs.python.org/issue21087 closed by brett.cannon #21108: Add examples for importlib in doc https://bugs.python.org/issue21108 closed by brett.cannon #21199: [2.7] Python on 64-bit Windows uses signed 32-bit type for rea https://bugs.python.org/issue21199 closed by vstinner #21242: Generalize configure check for working Python executable https://bugs.python.org/issue21242 closed by brett.cannon #22497: msiexec not creating msvcr90.dll with python -2.7.6.msi https://bugs.python.org/issue22497 closed by steve.dower #26660: tempfile.TemporaryDirectory() cleanup exception if nonwriteabl https://bugs.python.org/issue26660 closed by serhiy.storchaka #27807: Prevent site-packages .pth files from causing test_site failur https://bugs.python.org/issue27807 closed by vstinner #28087: macOS 12 poll syscall returns prematurely https://bugs.python.org/issue28087 closed by vstinner #35312: lib2to3.pgen2.parse.ParseError is not roundtrip pickleable https://bugs.python.org/issue35312 closed by Anthony Sottile #35691: cpython3.7.2 make test failed https://bugs.python.org/issue35691 closed by vstinner #36478: backport of pickle fixes to Python 3.5.7 uses C99 for loops https://bugs.python.org/issue36478 closed by Anthony Sottile #37695: Incorrect error message for `unget_wch(bytes_object)` https://bugs.python.org/issue37695 closed by Anthony Sottile #38373: List overallocation strategy https://bugs.python.org/issue38373 closed by serhiy.storchaka #38500: PEP 523: Add private _PyInterpreterState_SetEvalFrameFunc() fu https://bugs.python.org/issue38500 closed by vstinner #38576: CVE-2019-18348: CRLF injection via the host part of the url pa https://bugs.python.org/issue38576 closed by gregory.p.smith #38744: python 3.8 hang in multiprocessing.Pool() locking on FreeBSD / https://bugs.python.org/issue38744 closed by pablogsal #39087: [C API] No efficient C API to get UTF-8 string from unicode ob https://bugs.python.org/issue39087 closed by inada.naoki #39220: constant folding affects annotations despite 'from __future__ https://bugs.python.org/issue39220 closed by pablogsal #39360: python3.8 regression - ThreadPool join via __del__ hangs forev https://bugs.python.org/issue39360 closed by pablogsal #39507: http library missing HTTP status code 418 "I'm a teapot" https://bugs.python.org/issue39507 closed by serhiy.storchaka #39509: Update HTTP status code to follow IANA https://bugs.python.org/issue39509 closed by serhiy.storchaka #39562: Asynchronous comprehensions don't work in asyncio REPL https://bugs.python.org/issue39562 closed by BTaskaya #39582: ossaudiodev update helpers signature https://bugs.python.org/issue39582 closed by serhiy.storchaka #39585: Delete a pending item in _warning.c https://bugs.python.org/issue39585 closed by serhiy.storchaka #39638: Keep ASDL signatures for AST nodes https://bugs.python.org/issue39638 closed by serhiy.storchaka #39656: shebanged scripts can escape from `venv` depending on how it w https://bugs.python.org/issue39656 closed by vinay.sajip #39677: 3.6+ documentation for MAKE_FUNCTION https://bugs.python.org/issue39677 closed by serhiy.storchaka #39719: tempfile.SpooledTemporaryFile still has softspace property https://bugs.python.org/issue39719 closed by serhiy.storchaka #39797: shutdown() in socketserver.BaseServer should be in a different https://bugs.python.org/issue39797 closed by ned.deily #39824: Multi-phase extension module (PEP 489): don't call m_traverse, https://bugs.python.org/issue39824 closed by vstinner #39845: Argparse on Python 3.7.1 (Windows) appends double quotes to st https://bugs.python.org/issue39845 closed by eryksun #39871: math.copysign raises SystemError with non-float x and custom y https://bugs.python.org/issue39871 closed by mark.dickinson #39887: Duplicate C object description of vectorcallfunc https://bugs.python.org/issue39887 closed by serhiy.storchaka #39915: await_args_list in AsyncMock always refers to the last awaited https://bugs.python.org/issue39915 closed by xtreak #39937: Suggest the usage of Element.iter() instead of iter() in whats https://bugs.python.org/issue39937 closed by xtreak #39946: Remove _PyThreadState_GetFrame https://bugs.python.org/issue39946 closed by vstinner #39956: Numeric Literals vs string "1_1" on input int() or float() or https://bugs.python.org/issue39956 closed by mark.dickinson #39957: bpo39775 not fixed - inspect.Signature.parameters still dict/m https://bugs.python.org/issue39957 closed by inada.naoki #39962: Wrong file.tell() function results (Windows 10/Python 64 3.8.2 https://bugs.python.org/issue39962 closed by eric.smith #39964: adding a string to a list works differently with x+='' compare https://bugs.python.org/issue39964 closed by steven.daprano #39965: await is valid in non async functions if PyCF_ALLOW_TOP_LEVEL_ https://bugs.python.org/issue39965 closed by pablogsal #39968: port extension modules' macros of `get_module_state()` to inli https://bugs.python.org/issue39968 closed by vstinner #39969: Remove Param expression context from AST https://bugs.python.org/issue39969 closed by pablogsal #39970: Combined behavior of datetime.datetime.timestamp() and datetim https://bugs.python.org/issue39970 closed by p-ganssle #39972: Math library Bug Return None for "degrees(0)" https://bugs.python.org/issue39972 closed by SilentGhost #39973: The documentation for PyObject_GenericSetDict() is incorrect https://bugs.python.org/issue39973 closed by serhiy.storchaka #39974: A race condition with GIL releasing exists in stringlib_bytes_ https://bugs.python.org/issue39974 closed by inada.naoki #39977: Python aborts trying to load libcrypto.dylib https://bugs.python.org/issue39977 closed by ned.deily #39979: Cannot tune scrypt with large enough parameters https://bugs.python.org/issue39979 closed by Gle #39980: importlib.resources.path() may return incorrect path when usin https://bugs.python.org/issue39980 closed by jaraco #39985: str.format and string.Formatter subscript behaviors diverge https://bugs.python.org/issue39985 closed by eric.smith #39987: Simplify setting line numbers in the compiler https://bugs.python.org/issue39987 closed by serhiy.storchaka #39988: Remove AugLoad and AugStore expression context from AST https://bugs.python.org/issue39988 closed by serhiy.storchaka #39991: test_uuid.test_netstat_getnode() fails on FreeBSD VM: uuid._ne https://bugs.python.org/issue39991 closed by vstinner #39992: Windows line endings of pyc file detected on Ubuntu https://bugs.python.org/issue39992 closed by serhiy.storchaka #39993: Language Reference - Function definition parameter_list item d https://bugs.python.org/issue39993 closed by serhiy.storchaka #39997: "is" operator doesn't work on method returned from method desc https://bugs.python.org/issue39997 closed by christian.heimes #39998: [C API] Remove PyEval_AcquireLock() and PyEval_ReleaseLock() f https://bugs.python.org/issue39998 closed by vstinner #40000: Improve AST validation for Constant nodes https://bugs.python.org/issue40000 closed by BTaskaya #40004: String comparison with dotted numerical values wrong https://bugs.python.org/issue40004 closed by gvanrossum #40008: Spam https://bugs.python.org/issue40008 closed by zach.ware #40009: Spam https://bugs.python.org/issue40009 closed by zach.ware #40015: logging.Logger.disabled field is redundant https://bugs.python.org/issue40015 closed by rvprasad #40021: Throwing an Exception results in stack overflow https://bugs.python.org/issue40021 closed by benjamin.peterson #40027: re.sub inconsistency beginning with 3.7 https://bugs.python.org/issue40027 closed by mrabarnett

1 0

Looking for a sponsor and feedback on PEP 616: string methods for removing prefixes and suffixes
by Dennis Sweeney 20 Mar '20

20 Mar '20

Hello all! I'm a relatively new contributor looking for a Core Dev sponsor for the following PEP: https://github.com/python/peps/pull/1332 Related: - Python-Ideas Thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS… - Bug Tracker Issue: https://bugs.python.org/issue39939 - Github PR for implementation: https://github.com/python/cpython/pull/18939 Abstract ======== This is a proposal to add two new methods, ``cutprefix`` and ``cutsuffix``, to the APIs of Python's various string objects. In particular, the methods would be added to Unicode ``str`` objects, binary ``bytes`` and ``bytearray`` objects, and ``collections.UserString``. If ``s`` is one these objects, and ``s`` has ``pre`` as a prefix, then ``s.cutprefix(pre)`` returns a copy of ``s`` in which that prefix has been removed. If ``s`` does not have ``pre`` as a prefix, an unchanged copy of ``s`` is returned. In summary, ``s.cutprefix(pre)`` is roughly equivalent to ``s[len(pre):] if s.startswith(pre) else s``. The behavior of ``cutsuffix`` is analogous: ``s.cutsuffix(suf)`` is roughly equivalent to ``s[:-len(suf)] if suf and s.endswith(suf) else s``.

2 1

Re: Proliferation of tstate arguments.
by Anders Munch 18 Mar '20

18 Mar '20

Antoine Pitrou [mailto:solipsis@pitrou.net]: > This example is mixing up the notion of interpreter state and thread state. Sorry about that, I was making a more general point and not paying so much attention to the specific names. The general point is this: A thread-local variable can work as an implied parameter without creating synchronisation problems. You do not have to worry about painting yourself into an architectural corner by using thread-local variables, because anything you can do with an explicit parameter, you could also have done with an implied one in thread-local memory. Correctness is not at stake. Either approach will work. regards, Anders

1 0

PEP 585: Type Hinting Generics In Standard Collections
by Łukasz Langa 16 Mar '20

16 Mar '20

Read in the browser: https://www.python.org/dev/peps/pep-0585/ <https://www.python.org/dev/peps/pep-0585/> Read the source: https://raw.githubusercontent.com/python/peps/master/pep-0585.rst <https://raw.githubusercontent.com/python/peps/master/pep-0585.rst> The following PEP has been discussed on typing-sig already and a prototype implementation exists for it. I'm extending it now for wider feedback on python-dev, with the intent to present the final version for the Steering Council's consideration by mid-March. PEP: 585 Title: Type Hinting Generics In Standard Collections Author: Łukasz Langa <lukasz(a)python.org> Discussions-To: Typing-Sig <typing-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Mar-2019 Python-Version: 3.9 Abstract Static typing as defined by PEPs 484, 526, 544, 560, and 563 was built incrementally on top of the existing Python runtime and constrained by existing syntax and runtime behavior. This led to the existence of a duplicated collection hierarchy in the typing module due to generics (for example typing.List and the built-in list). This PEP proposes to enable support for the generics syntax in all standard collections currently available in the typing module. Rationale and Goals This change removes the necessity for a parallel type hierarchy in the typing module, making it easier for users to annotate their programs and easier for teachers to teach Python. Terminology Generic (n.) -- a type that can be parametrized, typically a container. Also known as a parametric type or a generic type. For example: dict. Parametrized generic -- a specific instance of a generic with the expected types for container elements provided. Also known as a parametrized type. For example: dict[str, int]. Backwards compatibility Tooling, including type checkers and linters, will have to be adapted to recognize standard collections as generics. On the source level, the newly described functionality requires Python 3.9. For use cases restricted to type annotations, Python files with the "annotations" future-import (available since Python 3.7) can parametrize standard collections, including builtins. To reiterate, that depends on the external tools understanding that this is valid. Implementation Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parametrize standard collections directly. Example: from __future__ import annotations def find(haystack: dict[str, list[int]]) -> int: ... Usefulness of this syntax before PEP 585 <https://www.python.org/dev/peps/pep-0585> is limited as external tooling like Mypy does not recognize standard collections as generic. Moreover, certain features of typing like type aliases or casting require putting types outside of annotations, in runtime context. While these are relatively less common than type annotations, it's important to allow using the same type syntax in all contexts. This is why starting with Python 3.9, the following collections become generic using __class_getitem__() to parametrize contained types: tuple # typing.Tuple list # typing.List dict # typing.Dict set # typing.Set frozenset # typing.FrozenSet type # typing.Type collections.deque collections.defaultdict collections.OrderedDict collections.Counter collections.ChainMap collections.abc.Awaitable collections.abc.Coroutine collections.abc.AsyncIterable collections.abc.AsyncIterator collections.abc.AsyncGenerator collections.abc.Iterable collections.abc.Iterator collections.abc.Generator collections.abc.Reversible collections.abc.Container collections.abc.Collection collections.abc.Callable collections.abc.Set # typing.AbstractSet collections.abc.MutableSet collections.abc.Mapping collections.abc.MutableMapping collections.abc.Sequence collections.abc.MutableSequence collections.abc.ByteString collections.abc.MappingView collections.abc.KeysView collections.abc.ItemsView collections.abc.ValuesView contextlib.AbstractContextManager # typing.ContextManager contextlib.AbstractAsyncContextManager # typing.AsyncContextManager re.Pattern # typing.Pattern, typing.re.Pattern re.Match # typing.Match, typing.re.Match Importing those from typing is deprecated. Due to PEP 563 <https://www.python.org/dev/peps/pep-0563> and the intention to minimize the runtime impact of typing, this deprecation will not generate DeprecationWarnings. Instead, type checkers may warn about such deprecated usage when the target version of the checked program is signalled to be Python 3.9 or newer. It's recommended to allow for those warnings to be silenced on a project-wide basis. The deprecated functionality will be removed from the typing module in the first Python version released 5 years after the release of Python 3.9.0. Parameters to generics are available at runtime Preserving the generic type at runtime enables introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild. Just like with the typing module today, the parametrized generic types listed in the previous section all preserve their type parameters at runtime: >>> list[str] list[str] >>> tuple[int, ...] tuple[int, ...] >>> ChainMap[str, list[str]] collections.ChainMap[str, list[str]] This is implemented using a thin proxy type that forwards all method calls and attribute accesses to the bare origin type with the following exceptions: the __repr__ shows the parametrized type; the __origin__ attribute points at the non-parametrized generic class; the __args__ attribute is a tuple (possibly of length 1) of generic types passed to the original __class_getitem__; the __parameters__ attribute is a lazily computed tuple (possibly empty) of unique type variables found in __args__; the __getitem__ raises an exception to disallow mistakes like dict[str][str]. However it allows e.g. dict[str, T][int] and in that case returns dict[str, int]. This design means that it is possible to create instances of parametrized collections, like: >>> l = list[str]() [] >>> list is list[str] False >>> list == list[str] False >>> list[str] == list[str] True >>> list[str] == list[int] False >>> isinstance([1, 2, 3], list[str]) TypeError: isinstance() arg 2 cannot be a parametrized generic >>> issubclass(list, list[str]) TypeError: issubclass() arg 2 cannot be a parametrized generic >>> isinstance(list[str], types.GenericAlias) True Objects created with bare types and parametrized types are exactly the same. The generic parameters are not preserved in instances created with parametrized types, in other words generic types erase type parameters during object creation. One important consequence of this is that the interpreter does not attempt to type check operations on the collection created with a parametrized type. This provides symmetry between: l: list[str] = [] and: l = list[str]() For accessing the proxy type from Python code, it will be exported from the types module as GenericAlias. Pickling or (shallow- or deep-) copying a GenericAlias instance will preserve the type, origin, attributes and parameters. Forward compatibility Future standard collections must implement the same behavior. Reference implementation A proof-of-concept or prototype implementation <https://bugs.python.org/issue39481> exists. Rejected alternatives Do nothing Keeping the status quo forces Python programmers to perform book-keeping of imports from the typing module for standard collections, making all but the simplest annotations cumbersome to maintain. The existence of parallel types is confusing to newcomers (why is there both list and List?). The above problems also don't exist in user-built generic classes which share runtime functionality and the ability to use them as generic type annotations. Making standard collections harder to use in type hinting from user classes hindered typing adoption and usability. Generics erasure It would be easier to implement __class_getitem__ on the listed standard collections in a way that doesn't preserve the generic type, in other words: >>> list[str] <class 'list'> >>> tuple[int, ...] <class 'tuple'> >>> collections.ChainMap[str, list[str]] <class 'collections.ChainMap'> This is problematic as it breaks backwards compatibility: current equivalents of those types in the typing module do preserve the generic type: >>> from typing import List, Tuple, ChainMap >>> List[str] typing.List[str] >>> Tuple[int, ...] typing.Tuple[int, ...] >>> ChainMap[str, List[str]] typing.ChainMap[str, typing.List[str]] As mentioned in the "Implementation" section, preserving the generic type at runtime enables runtime introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild. Additionally, implementing subscripts as identity functions would make Python less friendly to beginners. Say, if a user is mistakenly passing a list type instead of a list object to a function, and that function is indexing the received object, the code would no longer raise an error. Today: >>> l = list >>> l[-1] TypeError: 'type' object is not subscriptable With __class_getitem__ as an identity function: >>> l = list >>> l[-1] list The indexing being successful here would likely end up raising an exception at a distance, confusing the user. Disallowing instantiation of parametrized types Given that the proxy type which preserves __origin__ and __args__ is mostly useful for runtime introspection purposes, we might have disallowed instantiation of parametrized types. In fact, forbidding instantiation of parametrized types is what the typing module does today for types which parallel builtin collections (instantiation of other parametrized types is allowed). The original reason for this decision was to discourage spurious parametrization which made object creation up to two orders of magnitude slower compared to the special syntax available for those builtin collections. This rationale is not strong enough to allow the exceptional treatment of builtins. All other parametrized types can be instantiated, including parallels of collections in the standard library. Moreover, Python allows for instantiation of lists using list() and some builtin collections don't provide special syntax for instantiation. Making isinstance(obj, list[str]) perform a check ignoring generics An earlier version of this PEP suggested treating parametrized generics like list[str] as equivalent to their non-parametrized variants like list for purposes of isinstance() and issubclass(). This would be symmetrical to how list[str]() creates a regular list. This design was rejected because isinstance() and issubclass() checks with parametrized generics would read like element-by-element runtime type checks. The result of those checks would be surprising, for example: >>> isinstance([1, 2, 3], list[str]) True Note the object doesn't match the provided generic type but isinstance() still returns True because it only checks whether the object is a list. If a library is faced with a parametrized generic and would like to perform an isinstance() check using the base type, that type can be retrieved using the __origin__ attribute on the parametrized generic. Making isinstance(obj, list[str]) perform a runtime type check This functionality requires iterating over the collection which is a destructive operation in some of them. This functionality would have been useful, however implementing the type checker within Python that would deal with complex types, nested type checking, type variables, string forward references, and so on is out of scope for this PEP. Note on the initial draft An early version of this PEP discussed matters beyond generics in standard collections. Those unrelated topics were removed for clarity. Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

7 14