Mailman 3 April 2020 - Python-Dev

Question about bytecode stability
by Jonathan Goble 07 Apr '20

07 Apr '20

I'm getting ready to get back into an old personal project with some new ideas. Before I do, I need to ask a question regarding the stability of CPython bytecode. Before you say it, I am well aware and fully understand that the opcodes may change in value and semantics between 3.x and 3.(x+1) without notice, and that's acceptable to me. My question is, are the opcodes guaranteed stable across the lifetime of a single 3.x release? In other words, are they guaranteed to not change values or semantics between 3.x.y and 3.x.(y+1)? Reading through the list of opcodes in the dis documentation, it seems that all changes have occurred in 3.x.0, so it seems the answer would be yes, but then the "CPython implementation detail" paragraph at the top doesn't specify that and is a little vague on whether that's true or not. (For context, my personal project is a BASIC implementation in Python that would compile to Python bytecode.)

5 6

[RELEASE] Python 2.7.18 release candidate 1
by Benjamin Peterson 06 Apr '20

06 Apr '20

Greetings, 2.7.18 release candidate 1, a testing release for the last release of the Python 2.7 series, is now available for download. The CPython core developers stopped applying routine bugfixes to the 2.7 branch on January 1. 2.7.18 will includes fixes that were made between the release of 2.7.17 and the end of 2019. A final—very final!—release is expected in 2 weeks. Downloads are at: https://www.python.org/downloads/release/python-2718rc1/ The full changelog is at https://raw.githubusercontent.com/python/cpython/v2.7.18rc1/Misc/NEWS.d/2.7… Test it out, and let us know if there are any critical problems at https://bugs.python.org/ (This is the last chance!) All the best, Benjamin

1 0

How to enable tracemalloc for the test suite?
by Skip Montanaro 06 Apr '20

06 Apr '20

I've got a memory issue in my modified Python interpreter I'm trying to debug. Output at the end of the problematic unit test looks like this: ... == Tests result: FAILURE then SUCCESS == 1 test OK. 1 re-run test: test_rattlesnake Total duration: 2.9 sec Tests result: FAILURE then SUCCESS Debug memory block at address p=0x55a227969080: API '' 0 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 *** OUCH at p-5: 0x00 *** OUCH at p-4: 0x00 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0x55a227969080 are not all FORBIDDENBYTE (0xfd): at tail+0: 0x00 *** OUCH at tail+1: 0x00 *** OUCH at tail+2: 0x00 *** OUCH at tail+3: 0x00 *** OUCH at tail+4: 0x00 *** OUCH at tail+5: 0x00 *** OUCH at tail+6: 0x00 *** OUCH at tail+7: 0x00 *** OUCH The block was made by call #0 to debug malloc/realloc. Enable tracemalloc to get the memory block allocation traceback Looking at the tracemalloc module docs and trying various command line args (-X tracemalloc=5) or environment variables (PYTHONTRACEMALLOC=5) I'm unable to provoke any different output. The module docs give a tantalizing suggestion: Example of output of the Python test suite: but I see no command line args related to running the test suite with tracemalloc enabled. Pointers appreciated. Skip

3 8

Any thoughts about a control flow optimizer for CPython?
by joannah nanjekye 06 Apr '20

06 Apr '20

Hey all, From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. -- Best, Joannah Nanjekye *"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis*

4 7

Set PyEval_SetTrace to method from class
by Leandro Müller 05 Apr '20

05 Apr '20

Hello. I need to set a PyEval_SetTrace with a new class instance. All examples use simple function. PyEval_SetTrace(trace_trampoline, obj); Following my problem: I start three threads, but I need to check trace C just one thread. If I run a while on C trace on thread, it stops all threads. So, I've been thinking to run Trace with a new instance class C. Any suggestions? Att. Leandro Müller

2 1

Summary of Python tracker Issues
by Python tracker 04 Apr '20

04 Apr '20

ACTIVITY SUMMARY (2020-03-27 - 2020-04-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7401 (+44) closed 44526 (+40) total 51927 (+84) Open issues with patches: 2909 Issues opened (65) ================== #23574: datetime: support leap seconds https://bugs.python.org/issue23574 reopened by vstinner #35455: Solaris thread_time doesn't work with current implementation https://bugs.python.org/issue35455 reopened by BTaskaya #37207: Use PEP 590 vectorcall to speed up calls to range(), list() an https://bugs.python.org/issue37207 reopened by petr.viktorin #39562: Asynchronous comprehensions don't work in asyncio REPL https://bugs.python.org/issue39562 reopened by vstinner #39951: Ignore specific errors when closing ssl connections https://bugs.python.org/issue39951 reopened by Dima.Tisnek #40068: test_threading: ThreadJoinOnShutdown.test_reinit_tls_after_for https://bugs.python.org/issue40068 reopened by vstinner #40093: ThreadPoolExecutor with wait=True shuts down too early https://bugs.python.org/issue40093 opened by fireattack #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 opened by vstinner #40095: Incorrect st_ino returned for ReFS on Windows 10 https://bugs.python.org/issue40095 opened by mbrijun(a)gmail.com #40098: dir() does not return the list of valid attributes for the obj https://bugs.python.org/issue40098 opened by serhiy.storchaka #40100: IDLE: Undesired behavior of emoji in editor. https://bugs.python.org/issue40100 opened by Alex-Python-Programmer #40101: lib2to3 fails on default convert functionality https://bugs.python.org/issue40101 opened by BTaskaya #40102: Improve XLC support for function attributes https://bugs.python.org/issue40102 opened by BTaskaya #40103: ZipFile.extractall is not multiprocess safe with regard to dir https://bugs.python.org/issue40103 opened by nathansilberman #40105: Updating zip comment doesn't truncate the zip file https://bugs.python.org/issue40105 opened by nergall2 #40106: multiprocessor spawn https://bugs.python.org/issue40106 opened by mouse07410 #40107: pathlib: make `_Accessor.open()` return a file object and not https://bugs.python.org/issue40107 opened by barneygale #40110: multiprocessing.Pool.imap() should be lazy https://bugs.python.org/issue40110 opened by kousu #40111: Introspect ssl context: read ALPN and NPN protocols https://bugs.python.org/issue40111 opened by Dima.Tisnek #40113: Turtle demo https://bugs.python.org/issue40113 opened by Davide Golinelli #40116: Regression in memory use of shared key dictionaries for "compa https://bugs.python.org/issue40116 opened by Mark.Shannon #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 opened by omer sela #40119: ensurepip should use different pattern for pip/setuptool wheel https://bugs.python.org/issue40119 opened by anthon #40120: Undefined C behavior going beyond end of struct via a [1] arra https://bugs.python.org/issue40120 opened by gregory.p.smith #40124: Clearer assertion error https://bugs.python.org/issue40124 opened by pgjones #40126: Incorrect error handling in unittest.mock https://bugs.python.org/issue40126 opened by bmclarnon #40127: Documentation of SSL library https://bugs.python.org/issue40127 opened by Christophe Nanteuil #40128: IDLE Show completions pop-up not working on macOS https://bugs.python.org/issue40128 opened by darthur90 #40129: Add test classes for custom __index__, __int__, __float__ and https://bugs.python.org/issue40129 opened by serhiy.storchaka #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 opened by ChrisRands #40133: Provide additional matchers for unittest.mock https://bugs.python.org/issue40133 opened by Flameeyes #40134: Inconsistent ANSI escape code handling on Windows 10 https://bugs.python.org/issue40134 opened by daverove #40135: multiprocessing: test_shared_memory_across_processes() cannot https://bugs.python.org/issue40135 opened by vstinner #40137: TODO list when PEP 573 "Module State Access from C Extension M https://bugs.python.org/issue40137 opened by vstinner #40138: Windows implementation of os.waitpid() truncates the exit stat https://bugs.python.org/issue40138 opened by vstinner #40139: mimetypes module racy https://bugs.python.org/issue40139 opened by ukl #40142: Modify _PyObject_GC_TRACK() to ensure that newly tracked objec https://bugs.python.org/issue40142 opened by vstinner #40143: shutil.rmtree will frequently fail on Windows under heavy load https://bugs.python.org/issue40143 opened by Alexander Riccio #40145: Pyshellext room for binary size improvement https://bugs.python.org/issue40145 opened by Alexander Riccio #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 opened by pablogsal #40148: Add PurePath.with_stem() https://bugs.python.org/issue40148 opened by timhoffm #40149: test_threading leaked [38, 38, 38] references, sum=114 https://bugs.python.org/issue40149 opened by vstinner #40150: (minor) mismatched argument in overlapped_RegisterWaitWithQueu https://bugs.python.org/issue40150 opened by Alexander Riccio #40151: _overlapped room for improvement https://bugs.python.org/issue40151 opened by Alexander Riccio #40152: Coroutine hangs if it performs async operations when handling https://bugs.python.org/issue40152 opened by salgado #40153: json dump with repeated key https://bugs.python.org/issue40153 opened by facundobatista #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 opened by ferferga #40157: SMTP email policy does not encode non-ASCII characters https://bugs.python.org/issue40157 opened by aepsil0n #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 opened by sydefekt #40160: documentation example of os.walk should be less destructive https://bugs.python.org/issue40160 opened by jftuga #40161: Name collisions in pythoncore, preventing unity/jumbo build https://bugs.python.org/issue40161 opened by Alexander Riccio #40163: multissl doesn't support tarballs in /source/old/ https://bugs.python.org/issue40163 opened by vstinner #40164: Upgrade Windows and macOS installer builds to OpenSSL 1.1.1f https://bugs.python.org/issue40164 opened by ned.deily #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 opened by BTaskaya #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 opened by amaajemyfren #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 opened by Beier Liu #40168: import pandas error[python 3.8.] https://bugs.python.org/issue40168 opened by Harish Gajjar #40169: `dis.findlabels()` should accept a code object https://bugs.python.org/issue40169 opened by laike9m #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 opened by vstinner #40171: Attempting to import inaccessible package imports an empty pac https://bugs.python.org/issue40171 opened by xfix #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 opened by nergall2 #40173: test.support.import_fresh_module fails to correctly block subm https://bugs.python.org/issue40173 opened by p-ganssle #40174: HAVE_CLOCK_GETTIME not repected in pytime.c https://bugs.python.org/issue40174 opened by jerome.hamm #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 opened by nergall2 #40176: unterminated string literal tokenization error messages could https://bugs.python.org/issue40176 opened by benjamin.peterson Most recent 15 issues with no replies (15) ========================================== #40176: unterminated string literal tokenization error messages could https://bugs.python.org/issue40176 #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 #40173: test.support.import_fresh_module fails to correctly block subm https://bugs.python.org/issue40173 #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 #40169: `dis.findlabels()` should accept a code object https://bugs.python.org/issue40169 #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 #40164: Upgrade Windows and macOS installer builds to OpenSSL 1.1.1f https://bugs.python.org/issue40164 #40161: Name collisions in pythoncore, preventing unity/jumbo build https://bugs.python.org/issue40161 #40157: SMTP email policy does not encode non-ASCII characters https://bugs.python.org/issue40157 #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 #40152: Coroutine hangs if it performs async operations when handling https://bugs.python.org/issue40152 #40150: (minor) mismatched argument in overlapped_RegisterWaitWithQueu https://bugs.python.org/issue40150 Most recent 15 issues waiting for review (15) ============================================= #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 #40163: multissl doesn't support tarballs in /source/old/ https://bugs.python.org/issue40163 #40160: documentation example of os.walk should be less destructive https://bugs.python.org/issue40160 #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 #40151: _overlapped room for improvement https://bugs.python.org/issue40151 #40148: Add PurePath.with_stem() https://bugs.python.org/issue40148 #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 #40145: Pyshellext room for binary size improvement https://bugs.python.org/issue40145 #40142: Modify _PyObject_GC_TRACK() to ensure that newly tracked objec https://bugs.python.org/issue40142 #40137: TODO list when PEP 573 "Module State Access from C Extension M https://bugs.python.org/issue40137 #40133: Provide additional matchers for unittest.mock https://bugs.python.org/issue40133 Top 10 most discussed issues (10) ================================= #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 29 msgs #37207: Use PEP 590 vectorcall to speed up calls to range(), list() an https://bugs.python.org/issue37207 13 msgs #40120: Undefined C behavior going beyond end of struct via a [1] arra https://bugs.python.org/issue40120 13 msgs #40098: dir() does not return the list of valid attributes for the obj https://bugs.python.org/issue40098 10 msgs #17005: Add a topological sort algorithm https://bugs.python.org/issue17005 8 msgs #39837: Remove Azure Pipelines from GitHub PRs https://bugs.python.org/issue39837 8 msgs #39966: mock 3.9 bug: Wrapped objects without __bool__ raise exception https://bugs.python.org/issue39966 8 msgs #40077: Convert static types to PyType_FromSpec() https://bugs.python.org/issue40077 8 msgs #33725: Python crashes on macOS after fork with no exec https://bugs.python.org/issue33725 7 msgs #39503: [security][CVE-2020-8492] Denial of service in urllib.request. https://bugs.python.org/issue39503 7 msgs Issues closed (42) ================== #15140: PEP 384 inconsistent with implementation https://bugs.python.org/issue15140 closed by gvanrossum #38527: configure script fails to detect "float word ordering" on Sola https://bugs.python.org/issue38527 closed by pablogsal #38636: IDLE regression: toggle tabs and change indent width functions https://bugs.python.org/issue38636 closed by terry.reedy #38972: [venv] Link to instructions to change PowerShell execution pol https://bugs.python.org/issue38972 closed by brett.cannon #39133: threading lib. working improperly on idle window https://bugs.python.org/issue39133 closed by terry.reedy #39682: pathlib.Path objects can be used as context managers https://bugs.python.org/issue39682 closed by pitrou #39780: Add HTTP Response code 103 https://bugs.python.org/issue39780 closed by corona10 #39812: Avoid daemon threads in concurrent.futures https://bugs.python.org/issue39812 closed by pitrou #39982: FreeBSD: SCTP tests of test_socket fails on AMD64 FreeBSD Shar https://bugs.python.org/issue39982 closed by koobs #40018: test_ssl fails with OpenSSL 1.1.1e https://bugs.python.org/issue40018 closed by benjamin.peterson #40019: test_gdb should better detect when Python is optimized https://bugs.python.org/issue40019 closed by vstinner #40020: growable_comment_array_add leaks, causes crash https://bugs.python.org/issue40020 closed by Alexander Riccio #40024: Add PyModule_AddType helper function https://bugs.python.org/issue40024 closed by vstinner #40071: test__xxsubinterpreters leaked [1, 1, 1] references: test_ids https://bugs.python.org/issue40071 closed by vstinner #40086: test_etree is skipped in test_typing due to cElementTree remov https://bugs.python.org/issue40086 closed by xtreak #40087: How to Uninstall Python3.7.3 using cmd? https://bugs.python.org/issue40087 closed by terry.reedy #40088: list.reverse(): slow sublist reverse https://bugs.python.org/issue40088 closed by rhettinger #40096: Support _Py_NO_RETURN on XLC https://bugs.python.org/issue40096 closed by pablogsal #40097: Queue.Empty issue - Python3.8 https://bugs.python.org/issue40097 closed by Python_dev_IL #40099: modify code format of json library for pep7,8 https://bugs.python.org/issue40099 closed by rhettinger #40104: ElementTree does not find elements in a default namespace with https://bugs.python.org/issue40104 closed by Myron Walker #40108: Improve error message for -m option when .py is present https://bugs.python.org/issue40108 closed by pablogsal #40109: List index doesn't work with multiple assignment https://bugs.python.org/issue40109 closed by benjamin.peterson #40112: AIX: xlc - default path changed and no longer recognized https://bugs.python.org/issue40112 closed by vstinner #40114: support maxsize argument for lru_cache's user_function option https://bugs.python.org/issue40114 closed by brendon-zhang(a)hotmail.com #40115: test_asyncio leaked [3, 3, 3] references, sum=9 https://bugs.python.org/issue40115 closed by vstinner #40117: __round__ doesn't behave well with return NotImplemented https://bugs.python.org/issue40117 closed by mark.dickinson #40121: socket module missing audit events https://bugs.python.org/issue40121 closed by steve.dower #40122: The implementation and documentation of "dis.findlables" don't https://bugs.python.org/issue40122 closed by serhiy.storchaka #40123: append() works not correct https://bugs.python.org/issue40123 closed by steven.daprano #40125: update OpenSSL 1.1.1 in multissltests.py to 1.1.1f https://bugs.python.org/issue40125 closed by benjamin.peterson #40130: Remove _PyUnicode_AsKind from private C API https://bugs.python.org/issue40130 closed by serhiy.storchaka #40131: Zipapp example has parameters in the wrong order https://bugs.python.org/issue40131 closed by xtreak #40136: add warning to datetime.replace documentation to not use it fo https://bugs.python.org/issue40136 closed by p-ganssle #40140: test_builtin crashes when runned in parallel mode on solaris https://bugs.python.org/issue40140 closed by vstinner #40141: Add line and column information for keywords in the AST https://bugs.python.org/issue40141 closed by pablogsal #40144: resource: RLIMIT_xxx and RUSAGE_xxx constants are missing on m https://bugs.python.org/issue40144 closed by vstinner #40146: Upgrade Azure Pipelines to OpenSSL 1.1.1f https://bugs.python.org/issue40146 closed by vstinner #40155: AIX: test_builtin.test_input_no_stdout_fileno() hangs https://bugs.python.org/issue40155 closed by vstinner #40156: CodeCov/patch job stills runs on pull requests on 3.5 and 3.6 https://bugs.python.org/issue40156 closed by vstinner #40159: Make python -V -V output arguments of configure https://bugs.python.org/issue40159 closed by Volker Weißmann #40162: Upgrade Travis CI to OpenSSL 1.1.1f https://bugs.python.org/issue40162 closed by vstinner

1 0

PEP 616 -- String methods to remove prefixes and suffixes
by Dennis Sweeney 03 Apr '20

03 Apr '20

Browser Link: https://www.python.org/dev/peps/pep-0616/ PEP: 616 Title: String methods to remove prefixes and suffixes Author: Dennis Sweeney <sweeney.dennis650(a)gmail.com> Sponsor: Eric V. Smith <eric(a)trueblade.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-Mar-2020 Python-Version: 3.9 Post-History: 30-Aug-2002 Abstract ======== This is a proposal to add two new methods, ``cutprefix`` and ``cutsuffix``, to the APIs of Python's various string objects. In particular, the methods would be added to Unicode ``str`` objects, binary ``bytes`` and ``bytearray`` objects, and ``collections.UserString``. If ``s`` is one these objects, and ``s`` has ``pre`` as a prefix, then ``s.cutprefix(pre)`` returns a copy of ``s`` in which that prefix has been removed. If ``s`` does not have ``pre`` as a prefix, an unchanged copy of ``s`` is returned. In summary, ``s.cutprefix(pre)`` is roughly equivalent to ``s[len(pre):] if s.startswith(pre) else s``. The behavior of ``cutsuffix`` is analogous: ``s.cutsuffix(suf)`` is roughly equivalent to ``s[:-len(suf)] if suf and s.endswith(suf) else s``. Rationale ========= There have been repeated issues [#confusion]_ on the Bug Tracker and StackOverflow related to user confusion about the existing ``str.lstrip`` and ``str.rstrip`` methods. These users are typically expecting the behavior of ``cutprefix`` and ``cutsuffix``, but they are surprised that the parameter for ``lstrip`` is interpreted as a set of characters, not a substring. This repeated issue is evidence that these methods are useful, and the new methods allow a cleaner redirection of users to the desired behavior. As another testimonial for the usefulness of these methods, several users on Python-Ideas [#pyid]_ reported frequently including similar functions in their own code for productivity. The implementation often contained subtle mistakes regarding the handling of the empty string (see `Specification`_). Specification ============= The builtin ``str`` class will gain two new methods with roughly the following behavior:: def cutprefix(self: str, pre: str, /) -> str: if self.startswith(pre): return self[len(pre):] return self[:] def cutsuffix(self: str, suf: str, /) -> str: if suf and self.endswith(suf): return self[:-len(suf)] return self[:] The only difference between the real implementation and the above is that, as with other string methods like ``replace``, the methods will raise a ``TypeError`` if any of ``self``, ``pre`` or ``suf`` is not an instace of ``str``, and will cast subclasses of ``str`` to builtin ``str`` objects. Note that without the check for the truthyness of ``suf``, ``s.cutsuffix('')`` would be mishandled and always return the empty string due to the unintended evaluation of ``self[:-0]``. Methods with the corresponding semantics will be added to the builtin ``bytes`` and ``bytearray`` objects. If ``b`` is either a ``bytes`` or ``bytearray`` object, then ``b.cutsuffix()`` and ``b.cutprefix()`` will accept any bytes-like object as an argument. Note that the ``bytearray`` methods return a copy of ``self``; they do not operate in place. The following behavior is considered a CPython implementation detail, but is not guaranteed by this specification:: >>> x = 'foobar' * 10**6 >>> x.cutprefix('baz') is x is x.cutsuffix('baz') True >>> x.cutprefix('') is x is x.cutsuffix('') True That is, for CPython's immutable ``str`` and ``bytes`` objects, the methods return the original object when the affix is not found or if the affix is empty. Because these types test for equality using shortcuts for identity and length, the following equivalent expressions are evaluated at approximately the same speed, for any ``str`` objects (or ``bytes`` objects) ``x`` and ``y``:: >>> (True, x[len(y):]) if x.startswith(y) else (False, x) >>> (True, z) if x != (z := x.cutprefix(y)) else (False, x) The two methods will also be added to ``collections.UserString``, where they rely on the implementation of the new ``str`` methods. Motivating examples from the Python standard library ==================================================== The examples below demonstrate how the proposed methods can make code one or more of the following: Less fragile: The code will not depend on the user to count the length of a literal. More performant: The code does not require a call to the Python built-in ``len`` function. More descriptive: The methods give a higher-level API for code readability, as opposed to the traditional method of string slicing. refactor.py ----------- - Current:: if fix_name.startswith(self.FILE_PREFIX): fix_name = fix_name[len(self.FILE_PREFIX):] - Improved:: fix_name = fix_name.cutprefix(self.FILE_PREFIX) c_annotations.py: ----------------- - Current:: if name.startswith("c."): name = name[2:] - Improved:: name = name.cutprefix("c.") find_recursionlimit.py ---------------------- - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) deccheck.py ----------- This is an interesting case because the author chose to use the ``str.replace`` method in a situation where only a prefix was intended to be removed. - Current:: if funcname.startswith("context."): self.funcname = funcname.replace("context.", "") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Improved:: if funcname.startswith("context."): self.funcname = funcname.cutprefix("context.") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Arguably further improved:: self.contextfunc = funcname.startswith("context.") self.funcname = funcname.cutprefix("context.") test_i18n.py ------------ - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) - Current:: if creationDate.endswith('\\n'): creationDate = creationDate[:-len('\\n')] - Improved:: creationDate = creationDate.cutsuffix('\\n') shared_memory.py ---------------- - Current:: reported_name = self._name if _USE_POSIX and self._prepend_leading_slash: if self._name.startswith("/"): reported_name = self._name[1:] return reported_name - Improved:: if _USE_POSIX and self._prepend_leading_slash: return self._name.cutprefix("/") return self._name build-installer.py ------------------ - Current:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if ((retval.startswith('tcl') or retval.startswith('tk')) and retval.endswith('-src')): retval = retval[:-4] - Improved:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if retval.startswith(('tcl', 'tk')): retval = retval.cutsuffix('-src') Depending on personal style, ``archiveName[:-7]`` could also be changed to ``archiveName.cutsuffix('.tar.gz')``. test_core.py ------------ - Current:: if output.endswith("\n"): output = output[:-1] - Improved:: output = output.cutsuffix("\n") cookiejar.py ------------ - Current:: def strip_quotes(text): if text.startswith('"'): text = text[1:] if text.endswith('"'): text = text[:-1] return text - Improved:: def strip_quotes(text): return text.cutprefix('"').cutsuffix('"') - Current:: if line.endswith("\n"): line = line[:-1] - Improved:: line = line.cutsuffix("\n") fixdiv.py --------- - Current:: def chop(line): if line.endswith("\n"): return line[:-1] else: return line - Improved:: def chop(line): return line.cutsuffix("\n") test_concurrent_futures.py -------------------------- In the following example, the meaning of the code changes slightly, but in context, it behaves the same. - Current:: if name.endswith(('Mixin', 'Tests')): return name[:-5] elif name.endswith('Test'): return name[:-4] else: return name - Improved:: return name.cutsuffix('Mixin').cutsuffix('Tests').cutsuffix('Test') msvc9compiler.py ---------------- - Current:: if value.endswith(os.pathsep): value = value[:-1] - Improved:: value = value.cutsuffix(os.pathsep) test_pathlib.py --------------- - Current:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r[len(clsname) + 1 : -1] - Improved:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r.cutprefix(clsname + '(').cutsuffix(')') Rejected Ideas ============== Expand the lstrip and rstrip APIs --------------------------------- Because ``lstrip`` takes a string as its argument, it could be viewed as taking an iterable of length-1 strings. The API could therefore be generalized to accept any iterable of strings, which would be successively removed as prefixes. While this behavior would be consistent, it would not be obvious for users to have to call ``'foobar'.cutprefix(('foo,))`` for the common use case of a single prefix. Allow multiple prefixes ----------------------- Some users discussed the desire to be able to remove multiple prefixes, calling, for example, ``s.cutprefix('From: ', 'CC: ')``. However, this adds ambiguity about the order in which the prefixes are removed, especially in cases like ``s.cutprefix('Foo', 'FooBar')``. After this proposal, this can be spelled explicitly as ``s.cutprefix('Foo').cutprefix('FooBar')``. Remove multiple copies of a prefix ---------------------------------- This is the behavior that would be consistent with the aforementioned expansion of the ``lstrip/rstrip`` API -- repeatedly applying the function until the argument is unchanged. This behavior is attainable from the proposed behavior via the following:: >>> s = 'foo' * 100 + 'bar' >>> while s != (s := s.cutprefix("foo")): pass >>> s 'bar' The above can be modififed by chaining multiple ``cutprefix`` calls together to achieve the full behavior of the ``lstrip``/``rstrip`` generalization, while being explicit in the order of removal. While the proposed API could later be extended to include some of these use cases, to do so before any observation of how these methods are used in practice would be premature and may lead to choosing the wrong behavior. Raising an exception when not found ----------------------------------- There was a suggestion that ``s.cutprefix(pre)`` should raise an exception if ``not s.startswith(pre)``. However, this does not match with the behavior and feel of other string methods. There could be ``required=False`` keyword added, but this violates the KISS principle. Alternative Method Names ------------------------ Several alternatives method names have been proposed. Some are listed below, along with commentary for why they should be rejected in favor of ``cutprefix`` (the same arguments hold for ``cutsuffix``) ``ltrim`` "Trim" does in other languages (e.g. JavaScript, Java, Go, PHP) what ``strip`` methods do in Python. ``lstrip(string=...)`` This would avoid adding a new method, but for different behavior, it's better to have two different methods than one method with a keyword argument that select the behavior. ``cut_prefix`` All of the other methods of the string API, e.g. ``str.startswith()``, use ``lowercase`` rather than ``lower_case_with_underscores``. ``cutleft``, ``leftcut``, or ``lcut`` The explicitness of "prefix" is preferred. ``removeprefix``, ``deleteprefix``, ``withoutprefix``, etc. All of these might have been acceptable, but they have more characters than ``cut``. Some suggested that the verb "cut" implies mutability, but the string API already contains verbs like "replace", "strip", "split", and "swapcase". ``stripprefix`` Users may benefit from the mnemonic that "strip" means working with sets of characters, while other methods work with substrings, so re-using "strip" here should be avoided. Reference Implementation ======================== See the pull request on GitHub [#pr]_. References ========== .. [#pr] GitHub pull request with implementation (https://github.com/python/cpython/pull/18939) .. [#pyid] Discussion on Python-Ideas (https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS…) .. [#confusion] Comment listing Bug Tracker and StackOverflow issues (https://mail.python.org/archives/list/python-ideas@python.org/message/GRGAF…) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

33 137

Debug C++
by Leandro Müller 03 Apr '20

03 Apr '20

Hi Are there any away to debug C module during python debug? Att. Leandro Müller

4 3

Moving threadstate to thread-local storage.
by Mark Shannon 03 Apr '20

03 Apr '20

Hi, As an experiment, I thought I would try moving the thread state (what you get from _PyThreadState_GET() ) to TLS. https://github.com/python/cpython/compare/master...markshannon:threadstate_… It works, passing all the tests, and seems sound. It is a small patch (< 50 lines) and doesn't increase the overall code size. My branch is GCC/Clang only, so will need a bit of extra code for Windows. It should only need a few more lines; I haven't done it as I don't have a Windows machine to test it on. This is a *much* cleaner approach to removing the global variable than adding lots of extra parameters all over the place. Cheers, Mark.

3 4

Re: Python-Dev Digest, Vol 201, Issue 1
by Ilya Kamenshchikov 03 Apr '20

03 Apr '20

One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > reversed(enumerate('abc', 1)) > Should that yield...? > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > Something else? > My preference would be to treat the starting value as an ending value. My idea is largely that as long as any iterable has defined length and order, it should be reversible, and this should yield the same as if I did this reverse countdown with some pointer. It just so happens that `enumerate` depends fully in this regard on the iterable it enumerates, if it has defined length, then enumerate will also have defined length. Maybe enumerate should also start defining `len` whenever underlying iterable supports it. Regarding the starting value, I think it is important to keep symmetry: enumerate('abc', 1) is [ (1, 'a'), (2, 'b'), (3, 'c') ], therefore reversed(enumerate('abc', 1)) should be [(3, 'c'), (2, 'b'), (1, 'a')] Best Regards, -- Ilya Kamen On Thu, Apr 2, 2020 at 5:55 AM <python-dev-request(a)python.org> wrote: > Send Python-Dev mailing list submissions to > python-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman3/lists/python-dev.python.org/ > or, via email, send a message with subject or body 'help' to > python-dev-request(a)python.org > > You can reach the person managing the list at > python-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > Today's Topics: > > 1. Re: reversed enumerate (Steven D'Aprano) > 2. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > 3. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > > > ---------------------------------------------------------------------- > > Date: Thu, 2 Apr 2020 13:20:02 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-Dev] Re: reversed enumerate > To: python-dev(a)python.org > Cc: python-ideas(a)python.org > Message-ID: <20200402022000.GC16830(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Ilya, > > I'm not sure that this mailing list (Python-Dev) is the right place for > this discussion, I think that Python-Ideas (CCed) is the correct place. > > For the benefit of Python-Ideas, I have left your entire post below, to > establish context. > > [Ilya] > > I needed reversed(enumerate(x: list)) in my code, and have discovered > > that it wound't work. This is disappointing because operation is well > > defined. > > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. Consider: > > def values(): > while True: > yield random.random() > > a, b = reversed(enumerate(values()) > > What should the first pair of (a, b) be? > > However, having said that, I think that your idea is not unreasonable. > `enumerate(it)` in the most general case isn't reversable, but if `it` > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > be too. > > My personal opinion is that this is a fairly obvious and straightforward > enhancement, one which (hopefully!) shouldn't require much, if any, > debate. I don't think we need a new class for this, I think enhancing > enumerate to be reversable if its underlying iterator is reversable > makes good sense. > > But if you can show some concrete use-cases, especially one or two from > the standard library, that would help your case. Or some other languages > which offer this functionality as standard. > > On the other hand, I think that there is a fairly lightweight work > around. Define a helper function: > > def countdown(n): > while True: > yield n > n -= 1 > > then call it like this: > > # reversed(enumerate(seq)) > zip(countdown(len(seq)-1), reversed(seq))) > > So it isn't terribly hard to work around this. But I agree that it would > be nice if enumerate encapsulated this for the caller. > > One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > > reversed(enumerate('abc', 1)) > > Should that yield...? > > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > > Something else? > > My preference would be to treat the starting value as an ending value. > > > Steven > > > On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote: > > Hi, > > > > I needed reversed(enumerate(x: list)) in my code, and have discovered > that > > it wound't work. This is disappointing because operation is well defined. > > It is also well defined for str type, range, and - in principle, but not > > yet in practice - on dictionary iterators - keys(), values(), items() as > > dictionaries are ordered now. > > It would also be well defined on any user type implementing __iter__, > > __len__, __reversed__ - think numpy arrays, some pandas dataframes, > tensors. > > > > That's plenty of usecases, therefore I guess it would be quite useful to > > avoid hacky / inefficient solutions like described here: > > https://code.activestate.com/lists/python-list/706205/. > > > > If deemed useful, I would be interested in implementing this, maybe > > together with __reversed__ on dict keys, values, items. > > > > Best Regards, > > -- > > Ilya Kamen > > > > ----------- > > p.s. > > > > *Sketch* of what I am proposing: > > > > class reversible_enumerate: > > > > def __init__(self, iterable): > > self.iterable = iterable > > self.ctr = 0 > > > > def __iter__(self): > > for e in self.iterable: > > yield self.ctr, e > > self.ctr += 1 > > > > def __reversed__(self): > > try: > > ri = reversed(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > try: > > l = len(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > indexes = range(l-1, -1, -1) > > for i, e in zip(indexes, ri): > > yield i, e > > > > for i, c in reversed(reversible_enumerate("Hello World")): > > print(i, c) > > > > for i, c in reversed(reversible_enumerate([11, 22, 33])): > > > > print(i, c) > > > _______________________________________________ > > Python-Dev mailing list -- python-dev(a)python.org > > To unsubscribe send an email to python-dev-leave(a)python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD… > > Code of Conduct: http://python.org/psf/codeofconduct/ > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > > ------------------------------ > > End of Python-Dev Digest, Vol 201, Issue 1 > ****************************************** >

1 0