Mailman 3 July 2020 - Python-Dev

Stable ABI question.
by Inada Naoki July 3, 2020

July 3, 2020

Hi, folks. I found PyEval_AcquireLock and PyEval_ReleaseLock are deprecated since Python 3.2. But the same time, stable ABI is defined in Python 3.2 too. The deprecated APIs are stable ABI too because `ceval.h` is not excluded from the stable ABI PEP. As far as my understanding, we can not remove them until Python 4.0. Am I right? I will add comment like this. /* This is a part of stable ABI. Do not remove until Python 4.0 */ Regards, -- Inada Naoki <songofacandy(a)gmail.com>

4 10

Flexible assignment targets
by Elliott Chen July 3, 2020

July 3, 2020

I saw in the PEP that "Allow more flexible assignment targets instead" was rejected, but I actually think it's a good idea. The current PEP means there will be two different, subtly incompatible ways to destructure objects (match statement and iterable unpacking). The reasoning given was that most cases involve an if-statement, but there's no reason we can't have both flexible assignment targets and a match statement. I think unifying the concepts will make the language more consistent and easy to learn. I also think destructuring an object with only one possible pattern can be very useful, without the boilerplate to set up an entire match suite and manually raise an exception. This will make the variable assignment statement much more powerful. I don't see a good reason to arbitrarily restrict variable assignment to only allow sequence patterns. This might not be a good idea, but it can even potentially be used as an assert statement, like this: def __init__(self, version, name, point): self.version = 1 | 2 | 3 = type # assert that "version" is between 1 and 3 self.name = str() = string # assert that "name" is the correct type Point(0, self.y) = point # assert that "point" is on the y-axis The constant value pattern would have to be changed to something else, like "@constant", and even though that's also rejected, I think the benefits outweigh the costs here. The special treatment of "_" would have to go too, but there's no reason it can't be used as a wildcard by convention anyway, similar to how it can already be used in for loops. I don't see any benefit in treating it specially. Or the question mark can be the wildcard instead. If this is done, the "match statement" will become equivalent to trying different variable assignments. Maybe it can even be changed to look something like this (possibly with a keyword other than "with"): def is_tuple(node) -> bool: with Node(children=[LParen(), RParen()]) = node: return True or with Node(children=[Leaf(value="("), Node(), Leaf(value=")")]) = node: return True or with _ = node: return False def match_shape(shape): with Point(x, y) = shape: ... or with Rectangle(x, y, _, _) = shape: ... def type_new(cls, *args): with (obj,) = args: return obj.__class__ or with (name, bases, namespace) = args: return cls(name, bases, namespace) or with number_of_args = len(args): raise TypeError(f"need 1 or 3 arguments, not {number_of_args}") which has the benefit that it's obviously the same as a variable assignment (which also makes it obvious that "Point(x, y)" isn't a constructor call), making the match syntax less confusing. It also makes it more intuitive because the similar semantics with existing iterable unpacking become obvious. The expression can be seen right next to the pattern, instead of having to be searched higher up. It can also have different expressions matched for different cases. Maybe it can even be possible to stack "or with" and "elif" clauses on top of each other.

12 21

Summary of Python tracker Issues
by Python tracker July 3, 2020

July 3, 2020

ACTIVITY SUMMARY (2020-06-26 - 2020-07-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7520 (+30) closed 45424 (+44) total 52944 (+74) Open issues with patches: 3042 Issues opened (54) ================== #41004: Hash collisions in IPv4Interface and IPv6Interface https://bugs.python.org/issue41004 reopened by corona10 #41129: Python extension modules fail to build on Mac 10.15.1 (Catalin https://bugs.python.org/issue41129 opened by andrewfg1992 #41130: Improve/fix FreeBSD Bluetooth socket support https://bugs.python.org/issue41130 opened by myfreeweb #41131: Augment random.choices() with the alias method https://bugs.python.org/issue41131 opened by rhettinger #41133: Insufficient description of cyclic garbage collector for C API https://bugs.python.org/issue41133 opened by iabervon #41134: distutils.dir_util.copy_tree FileExistsError when updating sym https://bugs.python.org/issue41134 opened by Tom Hale #41135: Suggested change to http.server.HTTPServer to prevent socket r https://bugs.python.org/issue41135 opened by Michael Rich #41136: argparse uses default encoding when read arguments from file https://bugs.python.org/issue41136 opened by serhiy.storchaka #41137: pdb uses the locale encoding for .pdbrc https://bugs.python.org/issue41137 opened by serhiy.storchaka #41138: trace CLI reads source files using the locale encoding https://bugs.python.org/issue41138 opened by serhiy.storchaka #41139: cgi uses the locale encoding for log files https://bugs.python.org/issue41139 opened by serhiy.storchaka #41140: cgitb uses the locale encoding for log files https://bugs.python.org/issue41140 opened by serhiy.storchaka #41141: remove unneeded handling of '.' and '..' from patlib.Path.iter https://bugs.python.org/issue41141 opened by sir-sigurd #41143: distutils uses the locale encoding for the .pypirc file https://bugs.python.org/issue41143 opened by serhiy.storchaka #41145: EmailMessage.as_string is altering the message state and actua https://bugs.python.org/issue41145 opened by mardiros #41146: Convert signal.default_int_handler() to Argument Clinic https://bugs.python.org/issue41146 opened by serhiy.storchaka #41147: Document that redirect_std{out,err} yield the new stream as th https://bugs.python.org/issue41147 opened by PeterJCLaw #41148: IDLE uses the locale encoding for config files https://bugs.python.org/issue41148 opened by serhiy.storchaka #41149: Threads can fail to start https://bugs.python.org/issue41149 opened by Barney Stratford #41150: pipes uses text files and the locale encodig https://bugs.python.org/issue41150 opened by serhiy.storchaka #41151: Support for new Windows pseudoterminals in the subprocess modu https://bugs.python.org/issue41151 opened by njs #41154: test_pkgutil:test_name_resolution fails on some platforms https://bugs.python.org/issue41154 opened by RJ722 #41157: email.message_from_string() is unable to find the headers for https://bugs.python.org/issue41157 opened by jpatel #41160: Cross-compiling for GNU/Hurd fails https://bugs.python.org/issue41160 opened by mbakke #41162: Clear audit hooks after destructors https://bugs.python.org/issue41162 opened by steve.dower #41163: test_weakref hangs https://bugs.python.org/issue41163 opened by Peter Kuťák #41165: [Python 3.10] Remove APIs deprecated long enough https://bugs.python.org/issue41165 opened by inada.naoki #41167: Add new formats to PyArg_ParseTuple for "str or None" https://bugs.python.org/issue41167 opened by inada.naoki #41168: Lack of proper checking in PyObject_SetAttr leads to segmentat https://bugs.python.org/issue41168 opened by Iman Sharafodin #41169: socket.inet_pton raised when pass an IPv6 address like "[::]" https://bugs.python.org/issue41169 opened by seahoh #41170: Use strnlen instead of strlen when the size i known. https://bugs.python.org/issue41170 opened by Niclas Larsson #41171: Create companion methods of "PyType_FromSpec*" to allow settin https://bugs.python.org/issue41171 opened by WildCard65 #41172: test_peg_generator C tests fail on Windows ARM https://bugs.python.org/issue41172 opened by steve.dower #41173: Windows ARM buildbots cannot upload results https://bugs.python.org/issue41173 opened by steve.dower #41174: asyncio.coroutine decorator returns a non-generator function w https://bugs.python.org/issue41174 opened by a-feld #41175: Static analysis issues reported by GCC 10 https://bugs.python.org/issue41175 opened by cstratak #41176: revise Tkinter mainloop dispatching flag behavior https://bugs.python.org/issue41176 opened by Richard Sheridan #41177: ConvertingList and ConvertingTuple lack iterators and Converti https://bugs.python.org/issue41177 opened by godot_gildor #41178: Registry writes on Windows Store - workaround https://bugs.python.org/issue41178 opened by DataGhost #41180: marshal load bypass code.__new__ audit event https://bugs.python.org/issue41180 opened by tkmk #41182: DefaultSelector fails to detect selector on VMware ESXi https://bugs.python.org/issue41182 opened by akasurde #41183: Workaround or fix for SSL ".._KEY_TOO_SMALL" test failures https://bugs.python.org/issue41183 opened by larry #41185: lib2to3 generation of pickle files is racy https://bugs.python.org/issue41185 opened by barisione #41187: Convert the _msi module to Argument Clinic https://bugs.python.org/issue41187 opened by serhiy.storchaka #41188: Prepare CPython for opaque PyObject structure. https://bugs.python.org/issue41188 opened by WildCard65 #41190: msilib: SetProperty() accepts str, but GetProperty() returns b https://bugs.python.org/issue41190 opened by serhiy.storchaka #41191: PyType_FromModuleAndSpec is not mentioned in 3.9 What's new https://bugs.python.org/issue41191 opened by mdartiailh #41192: Some audit events are undocumented https://bugs.python.org/issue41192 opened by gousaiyang #41195: Interface to OpenSSL's security level https://bugs.python.org/issue41195 opened by mhughes #41197: Async magic methods in contextlib.closing https://bugs.python.org/issue41197 opened by uburuntu #41198: Round built-in function not shows zeros acording significant f https://bugs.python.org/issue41198 opened by Carlos Neves #41199: Docstring convention not followed for dataclasses documentatio https://bugs.python.org/issue41199 opened by JackStraw #41200: Add pickle.loads fuzz test https://bugs.python.org/issue41200 opened by Bruce Day #41202: Allow to provide custom exception handler to asyncio.run() https://bugs.python.org/issue41202 opened by tomaszdrozdz Most recent 15 issues with no replies (15) ========================================== #41202: Allow to provide custom exception handler to asyncio.run() https://bugs.python.org/issue41202 #41200: Add pickle.loads fuzz test https://bugs.python.org/issue41200 #41199: Docstring convention not followed for dataclasses documentatio https://bugs.python.org/issue41199 #41191: PyType_FromModuleAndSpec is not mentioned in 3.9 What's new https://bugs.python.org/issue41191 #41190: msilib: SetProperty() accepts str, but GetProperty() returns b https://bugs.python.org/issue41190 #41188: Prepare CPython for opaque PyObject structure. https://bugs.python.org/issue41188 #41185: lib2to3 generation of pickle files is racy https://bugs.python.org/issue41185 #41182: DefaultSelector fails to detect selector on VMware ESXi https://bugs.python.org/issue41182 #41178: Registry writes on Windows Store - workaround https://bugs.python.org/issue41178 #41174: asyncio.coroutine decorator returns a non-generator function w https://bugs.python.org/issue41174 #41173: Windows ARM buildbots cannot upload results https://bugs.python.org/issue41173 #41172: test_peg_generator C tests fail on Windows ARM https://bugs.python.org/issue41172 #41163: test_weakref hangs https://bugs.python.org/issue41163 #41160: Cross-compiling for GNU/Hurd fails https://bugs.python.org/issue41160 #41157: email.message_from_string() is unable to find the headers for https://bugs.python.org/issue41157 Most recent 15 issues waiting for review (15) ============================================= #41202: Allow to provide custom exception handler to asyncio.run() https://bugs.python.org/issue41202 #41199: Docstring convention not followed for dataclasses documentatio https://bugs.python.org/issue41199 #41195: Interface to OpenSSL's security level https://bugs.python.org/issue41195 #41188: Prepare CPython for opaque PyObject structure. https://bugs.python.org/issue41188 #41187: Convert the _msi module to Argument Clinic https://bugs.python.org/issue41187 #41183: Workaround or fix for SSL ".._KEY_TOO_SMALL" test failures https://bugs.python.org/issue41183 #41182: DefaultSelector fails to detect selector on VMware ESXi https://bugs.python.org/issue41182 #41180: marshal load bypass code.__new__ audit event https://bugs.python.org/issue41180 #41177: ConvertingList and ConvertingTuple lack iterators and Converti https://bugs.python.org/issue41177 #41175: Static analysis issues reported by GCC 10 https://bugs.python.org/issue41175 #41171: Create companion methods of "PyType_FromSpec*" to allow settin https://bugs.python.org/issue41171 #41170: Use strnlen instead of strlen when the size i known. https://bugs.python.org/issue41170 #41165: [Python 3.10] Remove APIs deprecated long enough https://bugs.python.org/issue41165 #41162: Clear audit hooks after destructors https://bugs.python.org/issue41162 #41160: Cross-compiling for GNU/Hurd fails https://bugs.python.org/issue41160 Top 10 most discussed issues (10) ================================= #19335: codeop misclassifies incomplete code with 'nonlocal' https://bugs.python.org/issue19335 17 msgs #41183: Workaround or fix for SSL ".._KEY_TOO_SMALL" test failures https://bugs.python.org/issue41183 15 msgs #41123: Remove Py_UNICODE APIs except PEP 623 https://bugs.python.org/issue41123 12 msgs #41004: Hash collisions in IPv4Interface and IPv6Interface https://bugs.python.org/issue41004 9 msgs #41165: [Python 3.10] Remove APIs deprecated long enough https://bugs.python.org/issue41165 9 msgs #40967: asyncio.Task.all_tasks() and asyncio.Task.current_task() must https://bugs.python.org/issue40967 7 msgs #41100: Build failure on macOS 11 (beta) https://bugs.python.org/issue41100 7 msgs #41169: socket.inet_pton raised when pass an IPv6 address like "[::]" https://bugs.python.org/issue41169 7 msgs #41198: Round built-in function not shows zeros acording significant f https://bugs.python.org/issue41198 7 msgs #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 6 msgs Issues closed (42) ================== #23427: Add sys.orig_argv: original command line arguments passed to t https://bugs.python.org/issue23427 closed by vstinner #24048: remove_module() needs to save/restore exception state https://bugs.python.org/issue24048 closed by ncoghlan #31082: reduce takes iterable, not just sequence https://bugs.python.org/issue31082 closed by corona10 #31590: CSV module incorrectly treats escaped newlines as new records https://bugs.python.org/issue31590 closed by terry.reedy #33665: tkinter.ttk.Scrollbar.fraction() under-estimates length https://bugs.python.org/issue33665 closed by serhiy.storchaka #34542: [TLS] Update test certs to future proof settings https://bugs.python.org/issue34542 closed by larry #39151: Simplify the deep-first-search of the assembler https://bugs.python.org/issue39151 closed by pablogsal #39314: (readline) Autofill the closing parenthesis during auto-comple https://bugs.python.org/issue39314 closed by corona10 #39385: Add an assertNoLogs context manager to unittest TestCase https://bugs.python.org/issue39385 closed by vinay.sajip #40769: Pegen: cover extra surrounding parentheses for invalid annotat https://bugs.python.org/issue40769 closed by pablogsal #40874: Update to libmpdec-2.5.0 https://bugs.python.org/issue40874 closed by skrah #40884: Added defaults parameter for logging.Formatter https://bugs.python.org/issue40884 closed by vinay.sajip #40924: Recent importlib change breaks most recent certifi == 2020.4.5 https://bugs.python.org/issue40924 closed by lukasz.langa #40971: Documentation still mentions 'u' string formatting option https://bugs.python.org/issue40971 closed by petr.viktorin #41007: test_importlib logs ResourceWarning: test_path.CommonTests.tes https://bugs.python.org/issue41007 closed by vstinner #41022: AST sum types is unidentifiable after ast.Constant became a ba https://bugs.python.org/issue41022 closed by BTaskaya #41043: Escape the literal part of the path for glob() https://bugs.python.org/issue41043 closed by serhiy.storchaka #41048: read_mime_types() should read the rule file using UTF-8, not t https://bugs.python.org/issue41048 closed by ned.deily #41076: Pre-feed the parser with the f-string expression location https://bugs.python.org/issue41076 closed by lys.nikolaou #41084: Signify that a SyntaxError comes from an fstring in the error https://bugs.python.org/issue41084 closed by lys.nikolaou #41119: Wrong error message for list/tuple followed by a colon https://bugs.python.org/issue41119 closed by lys.nikolaou #41132: F-String parser uses raw allocator https://bugs.python.org/issue41132 closed by lys.nikolaou #41142: msilib.CAB doesnot support non-ASCII files https://bugs.python.org/issue41142 closed by serhiy.storchaka #41144: IDLE: make Load Module load os.path (posixpath.py, ntpath.py) https://bugs.python.org/issue41144 closed by terry.reedy #41152: IDLE: make sys.stdxxx.encoding always be utf-8 https://bugs.python.org/issue41152 closed by terry.reedy #41153: [easy Doc] "PyPreConfig_InitIsolatedConfig" and "PyPreConfig_I https://bugs.python.org/issue41153 closed by WildCard65 #41155: Tkinter -postoffset not working for TCombobox https://bugs.python.org/issue41155 closed by Sveti007 #41156: Remove Task.all_tasks and Task.current_task https://bugs.python.org/issue41156 closed by inada.naoki #41158: IDLE: rewrite the code for handling file encoding https://bugs.python.org/issue41158 closed by serhiy.storchaka #41159: Nested async dict comprehension fails with SyntaxError https://bugs.python.org/issue41159 closed by serhiy.storchaka #41161: libmpdec-2.5.0 update is missing news entry https://bugs.python.org/issue41161 closed by skrah #41164: allow python to build for macosx-11.0-arm64 https://bugs.python.org/issue41164 closed by ned.deily #41166: CLASS ATTRIBUTES https://bugs.python.org/issue41166 closed by steven.daprano #41179: find_library on macOS Big Sur https://bugs.python.org/issue41179 closed by ned.deily #41181: [macOS] Build macOS installer with LTO and PGO optimizations https://bugs.python.org/issue41181 closed by ned.deily #41184: Reconciling IDLE's Comment Out Region / Uncomment Region with https://bugs.python.org/issue41184 closed by gvanrossum #41186: distutils.version epoch compatibility https://bugs.python.org/issue41186 closed by bar.harel #41189: An exploitable segmentation fault in _PyEval_EvalFrameDefault https://bugs.python.org/issue41189 closed by ned.deily #41193: traceback when exiting on read-only file system https://bugs.python.org/issue41193 closed by vstinner #41194: Python 3.9.0b3 crash on compile() in PyAST_Check() when the _a https://bugs.python.org/issue41194 closed by vstinner #41196: APPDATA directory is different in store installed python https://bugs.python.org/issue41196 closed by steve.dower #41201: Long integer arithmetic https://bugs.python.org/issue41201 closed by christian.heimes

1 0

When can we remove wchar_t* cache from string?
by Inada Naoki July 3, 2020

July 3, 2020

Hi, all. Py_UNICODE has been deprecated since PEP 393 (Flexible string representation). wchar_t* cache in the string object is used only in deprecated APIs. It waste 1 word (8 bytes on 64bit machine) per string instance. The deprecated APIs are documented as "Deprecated since version 3.3, will be removed in version 4.0." See https://docs.python.org/3/c-api/unicode.html#deprecated-py-unicode-apis But when PEP 393 is implemented, no one expects 3.10 will be released. Can we reschedule the removal? My proposal is, schedule the removal on Python 3.11. But we will postpone the removal if we can not remove its usage until it. I grepped the use of the deprecated APIs from top 4000 PyPI packages. result: https://github.com/methane/notes/blob/master/2020/wchar-cache/deprecated-use step: https://github.com/methane/notes/blob/master/2020/wchar-cache/README.md I noticed: * Most of them are generated by Cython. * I reported it to Cython so Cython 0.29.21 will fix them. I expect more than 1 year between Cython 0.29.21 and Python 3.11rc1. * Most of them are `PyUnicode_FromUnicode(NULL, 0);` * We may be able to keep PyUnicode_FromUnicode, but raise error when length>0. Regards, -- Inada Naoki <songofacandy(a)gmail.com>

9 22

Plan to remove Py_UNICODE APis except PEP 623.
by Inada Naoki July 3, 2020

July 3, 2020

Hi, all. I proposed PEP 623 to remove Unicode APIs deprecated by PEP 393. In this thread, I am proposing removal of Py_UNICODE (not Unicode objects) APIs deprecated by PEP 393. Please reply for any comments. ## Undocumented, have Py_DEPRECATED There is no problem to remove them in Python 3.10. I will just do it. * Py_UNICODE_str*** functions -- already removed in https://github.com/python/cpython/pull/21164 * PyUnicode_GetMax() ## Documented and have Py_DEPRECATED * PyLong_FromUnicode * PyUnicode_AsUnicodeCopy * PyUnicode_Encode * PyUnicode_EncodeUTF7 * PyUnicode_EncodeUTF8 * PyUnicode_EncodeUTF16 * PyUnicode_EncodeUTF32 * PyUnicode_EncodeUnicodeEscape * PyUnicode_EncodeRawUnicodeEscape * PyUnicode_EncodeLatin1 * PyUnicode_EncodeASCII * PyUnicode_EncodeCharmap * PyUnicode_TranslateCharmap * PyUnicode_EncodeMBCS These APIs are documented. The document has ``.. deprecated:: 3.3 4.0`` directive. They have been `Py_DEPRECATED` since Python 3.6 too. Plan: Change the document to ``.. deprecated:: 3.0 3.10`` and remove them in Python 3.10. ## PyUnicode_EncodeDecimal It is not documented. It has not been deprecated by Py_DEPRECATED. Plan: Add Py_DEPRECATED in Python 3.9 and remove it in 3.11. ## PyUnicode_TransformDecimalToASCII It is documented, but doesn't have ``deprecated`` directive. It is not deprecated by Py_DEPRECATED. Plan: Add Py_DEPRECATED and ``deprecated 3.3 3.11`` directive in 3.9, and remove it in 3.11. ## _PyUnicode_ToLowercase, _PyUnicode_ToUppercase They are not deprecated by PEP 393, but bpo-12736. They are documented as deprecated, but don't have ``Py_DEPRECATED``. Plan: Add Py_DEPRECATED in 3.9, and remove them in 3.11. Note: _PyUnicode_ToTitlecase has Py_DEPRECATED. It can be removed in 3.10. -- Inada Naoki <songofacandy(a)gmail.com>

10 30

Re: Python Documentation, Python language improvement, and productive discussion (Antoine Pitrou)
by David Antonini July 2, 2020

July 2, 2020

I'm interested in being part of said Docs group! David ________________________________ From: python-dev-request(a)python.org <python-dev-request(a)python.org> Sent: Thursday, July 2, 2020 5:33 PM To: python-dev(a)python.org <python-dev(a)python.org> Subject: Python-Dev Digest, Vol 204, Issue 23 Send Python-Dev mailing list submissions to python-dev(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman3/lists/python-dev.python.org/ or, via email, send a message with subject or body 'help' to python-dev-request(a)python.org You can reach the person managing the list at python-dev-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..." Today's Topics: 1. Re: Python Documentation, Python language improvement, and productive discussion (Antoine Pitrou) 2. Re: Recent PEP-8 change (Antoine Pitrou) 3. Re: Re Re: Recent PEP-8 change (Ivan Pozdeev) (Ivan Pozdeev) ---------------------------------------------------------------------- Date: Fri, 3 Jul 2020 00:10:04 +0200 From: Antoine Pitrou <solipsis(a)pitrou.net> Subject: [Python-Dev] Re: Python Documentation, Python language improvement, and productive discussion To: python-dev(a)python.org Message-ID: <20200703001004.0fca6c23@fsol> Content-Type: text/plain; charset=US-ASCII On Thu, 02 Jul 2020 20:41:54 -0000 "Carol Willing" <willingc(a)willingconsulting.com> wrote: > > Earlier this year at the Python Language Summit, Ned Batchelder and I presented the concept of a Documentation Workgroup and a vision for the next few years: > > - Slidedeck https://speakerdeck.com/willingc/cpython-documentation-the-next-5-years > - Blog post https://pyfound.blogspot.com/2020/04/cpython-documentation-next-5-years.html > > Due to some health issues that I have had the past few months, I haven't yet set up the workgroup. It's a priority of mine for July. We will have an open call for workgroup participants. Kudos for doing this. Having a consistent editorial direction for the documentation is a great idea. Regards Antoine. ------------------------------ Date: Fri, 3 Jul 2020 00:15:29 +0200 From: Antoine Pitrou <solipsis(a)pitrou.net> Subject: [Python-Dev] Re: Recent PEP-8 change To: python-dev(a)python.org Message-ID: <20200703001529.2c9ff416@fsol> Content-Type: text/plain; charset=US-ASCII On Thu, 02 Jul 2020 17:58:44 -0400 Random832 <random832(a)fastmail.com> wrote: > On Thu, Jul 2, 2020, at 05:20, Antoine Pitrou wrote: > > We're not talking about posting "your own writing", we're talking about > > comments (and presumably documentation) in a collective software > > project. There's a need for consistency, however it's > > specified and achieved. > > > > Otherwise why stop at English? I could just as well write my comments > > in French if it's all about individual freedom. Requiring English is > > not inclusive, it forced people like me to painfully adapt to a > > language I wasn't used to. And that has nothing to do with "white > > supremacy". > > Why indeed? Because we're talking about PEP 8, and PEP 8 intends to cover the code style used when writing code in the *Python standard library*. I don't think other Python core developers would like to read code with comments written in French (or, indeed, in Russian or Japanese or...). We're not talking about third-party projects, which indeed choose whatever style and language suit them. Regards Antoine. ------------------------------ Date: Fri, 3 Jul 2020 01:32:38 +0300 From: Ivan Pozdeev <vano(a)mail.mipt.ru> Subject: [Python-Dev] Re: Re Re: Recent PEP-8 change (Ivan Pozdeev) To: python-dev(a)python.org Message-ID: <c822aca3-0270-bbba-6f12-99381efc51a2(a)mail.mipt.ru> Content-Type: multipart/alternative; boundary="------------0905C79342DDD68A7F40BFCF" This is a multi-part message in MIME format. --------------0905C79342DDD68A7F40BFCF Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable At https://git-scm.com/book/en/v2/Git-Branching-Rebasing#_rebase_vs_merge= , they say that the argument of whether to allow overwriting=20 history in VCSes stems from two opposing views on what a project's histor= y should be. One is that is shall be a raw, unedited record of=20 events as they happened; the other is that is should be a (polished and a= dapted for future reading) story of how the project was made. While I'm firmly in camp story (long story short, it makes history much m= ore useful for everything except playing the blame game), in this=20 case, there's a case-specific reason, too. The remarks that stem the controversy have nothing to do with commit's in= tent in technical terms: clarifying wording that cause=20 misunderstanding. They were simply speculations by the author (and misgui= ded ones at that since Strunk & White are actually names). Leaving this commit as it is will leave no trace of any "comment on the c= ommit" and later discussion to a future onlooker. And as a commit=20 is the official codebase, it will still appear as endorsed by the dev tea= m. There'd be no clue that this is not the "final" thoughts on the=20 matter and there something else, somewhere else, and who knows where. While editing the message to just state facts (clarifying the wording) an= d omit specuilations, it will still serve its purpose -- while=20 stating the actual, valid, endorsed by the team (since the concensus is t= hat the change itself was useful and should not be reverted but=20 could be improved further, as a separate initiative), clean from speculat= ions intent of the edit. On 02.07.2020 23:17, David Antonini wrote: > No contention to the contrary, but as a routine, post-merge git history= rewrite, not a grand plan, from what I understand. > > Oh the other hand, an 'official' comment on the commit, recognising the= issue with the original commit message, the following discussion,=20 > and any conclusions that get reached, might be better, in my opinion. I= prefer to recognise and critique, rather than erase, > 'historical' history, as a rule (as opposed to git history). I think si= milar damage is done in this case, when the record, and opportunity=20 > to point to and learn from it, is erased. > > David > > --------------------------------------------------- > Date: Thu, 2 Jul 2020 21:33:56 +0300 > From: Ivan Pozdeev <vano(a)mail.mipt.ru> > Subject: [Python-Dev] Re: Recent PEP-8 change (Antoine Pitrou) > To: python-dev(a)python.org > Message-ID: <e1d9900a-6dae-8bfc-ad0f-a1512cfa87ab(a)mail.mipt.ru> > Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed > On 02.07.2020 21:20, Chris Angelico wrote: > > On Fri, Jul 3, 2020 at 4:09 AM David Mertz <mertz(a)gnosis.cx> wrote: > >>> An issue is that commit messages are uneditable after merge, so som= ething written somewhere suggesting consideration of this would be=20 > a good idea, with authors/mergers bearing this in mind, however unusual= a change on this basis would be. This would be additional burden=20 > on the core dev team, but if commitment is to be made to inclusivity, i= t might be what's necessary. > >> > >> I don't think so. https://docs.github.com/en/github/committing-chang= es-to-your-project/changing-a-commit-message. Interactive rebasing=20 > is perfectly possible, isn't it. I admit my git-fu isn't that strong, b= ut I've done something that I *think* is the same as this.=A0 It's=20 > possible I'm missing some distinction between the trees I've modified a= nd the current one, but I don't think so. > >> > > When you do that sort of rewriting, you're constructing a new and > > independent history and then saying "hey, this is the history I want > > everyone to respect now, thanks". It's full-on Back To The Future > > stuff, and can have annoying or serious consequences with everyone wh= o > > has a clone or fork of the repo. > > > > It would be extremely annoying to anyone who has an open PR at the > > time of the rewrite, but the annoyance would be temporary (hopefully > > one-off). > > > If you are talking about rewriting the PEP8 commit, it has proven to ca= use so much damage that this is warranted despite the=20 > inconveniences IMO. > > > ChrisA > > _______________________________________________ > > Python-Dev mailing list -- python-dev(a)python.org > > To unsubscribe send an email to python-dev-leave(a)python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at https://mail.python.org/archives/list/python-dev@= python.org/message/A2XBFOH5WGEOASSXHHKRWEHMZBN625SU/ > > Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.= org/psf/codeofconduct/> > > -- > > Regards, > > Ivan > > > > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@py= thon.org/message/ZZKGBAROR7TR2M7TM4EYSIIHXTUBQB4Y/ > Code of Conduct: http://python.org/psf/codeofconduct/ > --=20 > Regards, > Ivan --------------0905C79342DDD68A7F40BFCF Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1252"> </head> <body> <font size=3D"2"><span style=3D"font-size:11pt">At <a class=3D"moz-txt-link-freetext" href=3D"https://git-scm.com/bo= ok/en/v2/Git-Branching-Rebasing#_rebase_vs_merge">https://git-scm.com/boo= k/en/v2/Git-Branching-Rebasing#_rebase_vs_merge</a>, they say that the argument of whether to allow overwriting history in VCSes stems from two opposing views on what a project's history should be. One is that is shall be a raw, unedited record of events as they happened; the other is that is should be a (polished and adapted for future reading) story of how the project was made.</span></font> <p><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt">While I'm firmly i= n camp story (long story short, it makes history much more useful for everything except playing the blame game), in this case, there's a case-specific reason, too.</span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt">The remarks that s= tem the controversy have nothing to do with commit's intent in technical terms: clarifying wording that cause misunderstanding. They were simply speculations by the author (and misguided ones at that since </span></font><font size=3D"2"><span style=3D"font-size:11pt"><font size=3D"2"><span style=3D"font-size:11pt">Strunk & White are actually names</span></font></span></font><font size=3D"2"><span style=3D"font-size:11pt">).</span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></p> <p>Leaving this commit as it is will leave no trace of any "<font size=3D"2"><span style=3D"font-size:11pt">comment on the commit" = and later discussion to a future onlooker. And as a commit is the official codebase, it will still appear as endorsed by the dev team. There'd be no clue that this is not the "final" thoughts on the matter and there something else, somewhere else, and who knows where.<br> </span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt"></span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt">While editing the message to just state facts (clarifying the wording) and omit specuilations, it will still serve its purpose -- while stating the actual, valid, endorsed by the team (since the concensus is that the change itself was useful and should not be reverted but could be improved further, as a separate initiative), clean from speculations intent of the edit.<br> </span></font></p> <p><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></p> <p><br> </p> <div class=3D"moz-cite-prefix">On 02.07.2020 23:17, David Antonini wrote:<br> </div> <blockquote type=3D"cite" cite=3D"mid:YQXPR0101MB0872D86CFBD9CACAB41E4E4EB26D0@YQXPR0101MB0872.CANP= RD01.PROD.OUTLOOK.COM"> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1252"> <style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;m= argin-bottom:0;} </style> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <div> <div> <div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt">No contention to the contrary, but as a routine, post-merge git history rewrite, not a grand plan, from what I understand.</span></font></div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt">Oh the ot= her hand, an 'official' comment on the commit, recognising the issue with the original commit message, the following discussion, and any conclusions that get reached, might be better, in my opinion. I prefer to recognise and critique, rather than erase, <br> </span></font></div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt">'historic= al' history, as a rule (as opposed to git history). I think similar damage is done in this case, when the record, and opportunity to point to and learn from it, is erased. <br> </span></font></div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <font size=3D"2"><span style=3D"font-size:11pt">David<br> </span></font></div> <div><font size=3D"2"><span style=3D"font-size:11pt"><br> </span></font></div> <div><font size=3D"2"><span style=3D"font-size:11pt">------= ---------------------------------------------<br> </span></font></div> <div><font size=3D"2"><span style=3D"font-size:11pt">Date: Thu, 2 Jul 2020 21:33:56 +0300<br> </span></font></div> <div><font size=3D"2"><span style=3D"font-size:11pt">From: Ivan Pozdeev <a class=3D"moz-txt-link-rfc2396E" href=3D= "mailto:vano@mail.mipt.ru"><vano(a)mail.mipt.ru></a><br> Subject: [Python-Dev] Re: Recent PEP-8 change (Antoine Pitrou)<br> To: <a class=3D"moz-txt-link-abbreviated" href=3D"mai= lto:python-dev@python.org">python-dev(a)python.org</a><br> Message-ID: <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:e1d= 9900a-6dae-8bfc-ad0f-a1512cfa87ab(a)mail.mipt.ru"><e1d9900a-6dae-8bfc-ad= 0f-a1512cfa87ab(a)mail.mipt.ru></a><br> Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed<br> </span></font></div> </div> </div> </div> <div> <div> <div> <div><font size=3D"2"><span style=3D"font-size:11pt">On 02.07.2020 21:20, Chris Angelico wrote:<br> </span></font></div> </div> </div> </div> <div> <div> <div><font size=3D"2"><span style=3D"font-size:11pt">> On Fri, Jul 3, 2020 at 4:09 AM David Mertz <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:mertz= @gnosis.cx"><mertz(a)gnosis.cx></a> wrote:<br> >>> An issue is that commit messages are uneditable after merge, so something written somewhere suggesting consideration of this would be a good idea, with authors/mergers bearing this in mind, however unusual a change on this basis would be. This would be additional burden on the core dev team, but if commitment is to be made to inclusivity, it might be what's necessary.<br> >><br> >> I don't think so. <a href=3D"https://docs.github.com/en/github/committing-changes-to-your-proj= ect/changing-a-commit-message" target=3D"_blank" rel=3D"noopener noreferrer" moz-do-not-send=3D"true"> https://docs.github.com/en/github/committing-changes-to-your-project/chan= ging-a-commit-message</a>.=A0 Interactive rebasing is perfectly possible, isn't it.=A0 I admit my git-fu isn't that strong, but I've done something that I *think* is the same as this.=A0 It's possible I'm missing some distinction between the trees I've modified and the current one, but I don't think so.<br> >><br> > When you do that sort of rewriting, you're constructing a new and<br> > independent history and then saying "hey, this is the history I want<br> > everyone to respect now, thanks". It's full-on Back To The Future<br> > stuff, and can have annoying or serious consequences with everyone who<br> > has a clone or fork of the repo.<br> ><br> > It would be extremely annoying to anyone who has an open PR at the<br> > time of the rewrite, but the annoyance would be temporary (hopefully<br> > one-off).<br> <br> <br> If you are talking about rewriting the PEP8 commit, it has proven to cause so much damage that this is warranted despite the inconveniences IMO.<br> <br> > ChrisA<br> > _______________________________________________<br= > > Python-Dev mailing list -- <a class=3D"moz-txt-lin= k-abbreviated" href=3D"mailto:python-dev@python.org">python-dev(a)python.or= g</a><br> > To unsubscribe send an email to <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:py= thon-dev-leave(a)python.org">python-dev-leave(a)python.org</a><br> > <a href=3D"https://mail.python.org/mailman3/lists/python= -dev.python.org/" target=3D"_blank" rel=3D"noopener noreferrer" moz-do-not-send=3D"true"> https://mail.python.org/mailman3/lists/python-dev.python.org/</a><br> > Message archived at <a href=3D"https://mail.python.org/archives/list/python-dev@python.org/messa= ge/A2XBFOH5WGEOASSXHHKRWEHMZBN625SU/" target=3D"_blank" rel=3D"noopener noreferrer" moz-do-not-send=3D"true"> https://mail.python.org/archives/list/python-dev@python.org/message/A2XBF= OH5WGEOASSXHHKRWEHMZBN625SU/</a><br> > Code of Conduct: <a href=3D"http://python.org/psf/codeofconduct/" target=3D"_blank" rel=3D"noopener noreferrer" moz-do-not-send=3D"true"> http://python.org/psf/codeofconduct/</a><br> > -- <br> > Regards,<br> > Ivan</span></font></div> </div> </div> <br> </div> <br> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <pre class=3D"moz-quote-pre" wrap=3D"">____________________________= ___________________ Python-Dev mailing list -- <a class=3D"moz-txt-link-abbreviated" href=3D"= mailto:python-dev@python.org">python-dev(a)python.org</a> To unsubscribe send an email to <a class=3D"moz-txt-link-abbreviated" hre= f=3D"mailto:python-dev-leave@python.org">python-dev-leave(a)python.org</a> <a class=3D"moz-txt-link-freetext" href=3D"https://mail.python.org/mailma= n3/lists/python-dev.python.org/">https://mail.python.org/mailman3/lists/p= ython-dev.python.org/</a> Message archived at <a class=3D"moz-txt-link-freetext" href=3D"https://ma= il.python.org/archives/list/python-dev@python.org/message/ZZKGBAROR7TR2M7= TM4EYSIIHXTUBQB4Y/">https://mail.python.org/archives/list/python-dev@pyth= on.org/message/ZZKGBAROR7TR2M7TM4EYSIIHXTUBQB4Y/</a> Code of Conduct: <a class=3D"moz-txt-link-freetext" href=3D"http://python= .org/psf/codeofconduct/">http://python.org/psf/codeofconduct/</a> </pre> <div class=3D"moz-signature">-- <br> Regards,<br> Ivan</div> </blockquote> </body> </html> --------------0905C79342DDD68A7F40BFCF-- ------------------------------ Subject: Digest Footer _______________________________________________ Python-Dev mailing list -- python-dev(a)python.org To unsubscribe send an email to python-dev-leave(a)python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ ------------------------------ End of Python-Dev Digest, Vol 204, Issue 23 *******************************************

1 0

PEP 620: Hide implementation details from the C API
by Victor Stinner July 2, 2020

July 2, 2020

Hi, PEP available at: https://www.python.org/dev/peps/pep-0620/ <introduction> This PEP is the result of 4 years of research work on the C API: https://pythoncapi.readthedocs.io/ It's the third version. The first version (2017) proposed to add a "new C API" and advised C extensions maintainers to opt-in for it: it was basically the same idea as PEP 384 limited C API but in a different color. Well, I had no idea of what I was doing :-) The second version (April 2020) proposed to add a new Python runtime built from the same code base as the regular Python runtime but in a different build mode, the regular Python would continue to be fully compatible. I wrote the third version, the PEP 620, from scratch. It now gives an explicit and concrete list of incompatible C API changes, and has better motivation and rationale sections. The main PEP novelty is the new pythoncapi_compat.h header file distributed with Python to provide new C API functions to old Python versions, the second novelty is the process to reduce the number of broken C extensions. Whereas PEPs are usually implemented in a single Python version, the implementation of this PEP is expected to be done carefully over multiple Python versions. The PEP lists many changes which are already implemented in Python 3.7, 3.8 and 3.9. It defines a process to reduce the number of broken C extensions when introducing the incompatible C API changes listed in the PEP. The process dictates the rhythm of these changes. </introduction> PEP: 620 Title: Hide implementation details from the C API Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-June-2020 Python-Version: 3.10 Abstract ======== Introduce C API incompatible changes to hide implementation details. Once most implementation details will be hidden, evolution of CPython internals would be less limited by C API backward compatibility issues. It will be way easier to add new features. It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations, like tagged pointers. Define a process to reduce the number of broken C extensions. The implementation of this PEP is expected to be done carefully over multiple Python versions. It already started in Python 3.7 and most changes are already completed. The `Process to reduce the number of broken C extensions`_ dictates the rhythm. Motivation ========== The C API blocks CPython evolutions ----------------------------------- Adding or removing members of C structures is causing multiple backward compatibility issues. Adding a new member breaks the stable ABI (PEP 384), especially for types declared statically (e.g. ``static PyTypeObject MyType = {...};``). In Python 3.4, the PEP 442 "Safe object finalization" added the ``tp_finalize`` member at the end of the ``PyTypeObject`` structure. For ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type flag was required to announce if the type structure contains the ``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388 <https://bugs.python.org/issue32388>`_). The ``PyTypeObject.tp_print`` member, deprecated since Python 3.0 released in 2009, has been removed in the Python 3.8 development cycle. But the change broke too many C extensions and had to be reverted before 3.8 final release. Finally, the member was removed again in Python 3.9. C extensions rely on the ability to access directly structure members, indirectly through the C API, or even directly. Modifying structures like ``PyListObject`` cannot be even considered. The ``PyTypeObject`` structure is the one which evolved the most, simply because there was no other way to evolve CPython than modifying it. In the C API, all Python objects are passed as ``PyObject*``: a pointer to a ``PyObject`` structure. Experimenting tagged pointers in CPython is blocked by the fact that a C extension can technically dereference a ``PyObject*`` pointer and access ``PyObject`` members. Small "objects" can be stored as a tagged pointer with no concrete ``PyObject`` structure. Replacing Python garbage collector with a tracing garbage collector would also need to remove ``PyObject.ob_refcnt`` reference counter, whereas currently ``Py_INCREF()`` and ``Py_DECREF()`` macros access directly to ``PyObject.ob_refcnt``. Same CPython design since 1990: structures and reference counting ----------------------------------------------------------------- When the CPython project was created, it was written with one principle: keep the implementation simple enough so it can be maintained by a single developer. CPython complexity grew a lot and many micro-optimizations have been implemented, but CPython core design has not changed. Members of ``PyObject`` and ``PyTupleObject`` structures have not changed since the "Initial revision" commit (1990):: #define OB_HEAD \ unsigned int ob_refcnt; \ struct _typeobject *ob_type; typedef struct _object { OB_HEAD } object; typedef struct { OB_VARHEAD object *ob_item[1]; } tupleobject; Only names changed: ``object`` was renamed to ``PyObject`` and ``tupleobject`` was renamed to ``PyTupleObject``. CPython still tracks Python objects lifetime using reference counting internally and for third party C extensions (through the Python C API). All Python objects must be allocated on the heap and cannot be moved. Why is PyPy more efficient than CPython? ---------------------------------------- The PyPy project is a Python implementation which is 4.2x faster than CPython on average. PyPy developers chose to not fork CPython, but start from scratch to have more freedom in terms of optimization choices. PyPy does not use reference counting, but a tracing garbage collector which moves objects. Objects can be allocated on the stack (or even not at all), rather than always having to be allocated on the heap. Objects layouts are designed with performance in mind. For example, a list strategy stores integers directly as integers, rather than objects. Moreover, PyPy also has a JIT compiler which emits fast code thanks to the efficient PyPy design. PyPy bottleneck: the Python C API --------------------------------- While PyPy is way more efficient than CPython to run pure Python code, it is as efficient or slower than CPython to run C extensions. Since the C API requires ``PyObject*`` and allows to access directly structure members, PyPy has to associate a CPython object to PyPy objects and maintain both consistent. Converting a PyPy object to a CPython object is inefficient. Moreover, reference counting also has to be implemented on top of PyPy tracing garbage collector. These conversions are required because the Python C API is too close to the CPython implementation: there is no high-level abstraction. For example, structures members are part of the public C API and nothing prevents a C extension to get or set directly ``PyTupleObject.ob_item[0]`` (the first item of a tuple). See `Inside cpyext: Why emulating CPython C API is so Hard <https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c…>`_ (Sept 2018) by Antonio Cuni for more details. Rationale ========= Hide implementation details --------------------------- Hiding implementation details from the C API has multiple advantages: * It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations. For example, tagged pointers, and replace the garbage collector with a tracing garbage collector which can move objects. * Adding new features in CPython becomes easier. * PyPy should be able to avoid conversions to CPython objects in more cases: keep efficient PyPy objects. * It becomes easier to implement the C API for a new Python implementation. * More C extensions will be compatible with Python implementations other than CPython. Relationship with the limited C API ----------------------------------- The PEP 384 "Defining a Stable ABI" is in Python 3.4. It introduces the "limited C API": a subset of the C API. When the limited C API is used, it becomes possible to build a C extension only once and use it on multiple Python versions: that's the stable ABI. The main limitation of the PEP 384 is that C extensions have to opt-in for the limited C API. Only very few projects made this choice, usually to ease distribution of binaries, especially on Windows. This PEP moves the C API towards the limited C API. Ideally, the C API will become the limited C API and all C extensions will use the stable ABI, but this is out of this PEP scope. Specification ============= Summary ------- * (**Completed**) Reorganize the C API header files: create ``Include/cpython/`` and ``Include/internal/`` subdirectories. * (**Completed**) Move private functions exposing implementation details to the internal C API. * (**Completed**) Convert macros to static inline functions. * (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and ``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` macros become functions which cannot be used as l-value. * (**Completed**) New C API functions must not return borrowed references. * (**In Progress**) Provide ``pythoncapi_compat.h`` header file. * (**In Progress**) Make structures opaque, add getter and setter functions. * (**Not Started**) Deprecate ``PySequence_Fast_ITEMS()``. * (**Not Started**) Convert ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros to static inline functions. Reorganize the C API header files --------------------------------- The first consumer of the C API was Python itself. There is no clear separation between APIs which must not be used outside Python, and API which are public on purpose. Header files must be reorganized in 3 API: * ``Include/`` directory is the limited C API: no implementation details, structures are opaque. C extensions using it get a stable ABI. * ``Include/cpython/`` directory is the CPython C API: less "portable" API, depends more on the Python version, expose some implementation details, few incompatible changes can happen. * ``Internal/internal/`` directory is the internal C API: implementation details, incompatible changes are likely at each Python release. The creation of the ``Include/cpython/`` directory is fully backward compatible. ``Include/cpython/`` header files cannot be included directly and are included automatically by ``Include/`` header files when the ``Py_LIMITED_API`` macro is not defined. The internal C API is installed and can be used for specific usage like debuggers and profilers which must access structures members without executing code. C extensions using the internal C API are tightly coupled to a Python version and must be recompiled at each Python version. **STATUS**: Completed (in Python 3.8) The reorganization of header files started in Python 3.7 and was completed in Python 3.8: * `bpo-35134 <https://bugs.python.org/issue35134>`_: Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details. * `bpo-35081 <https://bugs.python.org/issue35081>`_: Move internal headers to ``Include/internal/`` Move private functions to the internal C API -------------------------------------------- Private functions which expose implementation details must be moved to the internal C API. If a C extension relies on a CPython private function which exposes CPython implementation details, other Python implementations have to re-implement this private function to support this C extension. **STATUS**: Completed (in Python 3.9) Private functions moved to the internal C API in Python 3.8: * ``_PyObject_GC_TRACK()``, ``_PyObject_GC_UNTRACK()`` Macros and functions excluded from the limited C API in Python 3.9: * ``_PyObject_SIZE()``, ``_PyObject_VAR_SIZE()`` * ``PyThreadState_DeleteCurrent()`` * ``PyFPE_START_PROTECT()``, ``PyFPE_END_PROTECT()`` * ``_Py_NewReference()``, ``_Py_ForgetReference()`` * ``_PyTraceMalloc_NewReference()`` * ``_Py_GetRefTotal()`` Private functions moved to the internal C API in Python 3.9: * GC functions like ``_Py_AS_GC()``, ``_PyObject_GC_IS_TRACKED()`` and ``_PyGCHead_NEXT()`` * ``_Py_AddToAllObjects()`` (not exported) * ``_PyDebug_PrintTotalRefs()``, ``_Py_PrintReferences()``, ``_Py_PrintReferenceAddresses()`` (not exported) Public "clear free list" functions moved to the internal C API an renamed to private functions and in Python 3.9: * ``PyAsyncGen_ClearFreeLists()`` * ``PyContext_ClearFreeList()`` * ``PyDict_ClearFreeList()`` * ``PyFloat_ClearFreeList()`` * ``PyFrame_ClearFreeList()`` * ``PyList_ClearFreeList()`` * ``PyTuple_ClearFreeList()`` * Functions simply removed: * ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``: bound method free list removed in Python 3.9. * ``PySet_ClearFreeList()``: set free list removed in Python 3.4. * ``PyUnicode_ClearFreeList()``: Unicode free list removed in Python 3.3. Convert macros to static inline functions ----------------------------------------- Converting macros to static inline functions have multiple advantages: * Functions have well defined parameter types and return type. * Functions can use variables with a well defined scope (the function). * Debugger can be put breakpoints on functions and profilers can display the function name in the call stacks. In most cases, it works even when a static inline function is inlined. * Functions don't have `macros pitfalls <https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_. Converting macros to static inline functions should only impact very few C extensions which use macros in unusual ways. For backward compatibility, functions must continue to accept any type, not only ``PyObject*``, to avoid compiler warnings, since most macros cast their parameters to ``PyObject*``. Python 3.6 requires C compilers to support static inline functions: the PEP 7 requires a subset of C99. **STATUS**: Completed (in Python 3.9) Macros converted to static inline functions in Python 3.8: * ``Py_INCREF()``, ``Py_DECREF()`` * ``Py_XINCREF()``, ``Py_XDECREF()`` * ``PyObject_INIT()``, ``PyObject_INIT_VAR()`` * ``_PyObject_GC_TRACK()``, ``_PyObject_GC_UNTRACK()``, ``_Py_Dealloc()`` Macros converted to regular functions in Python 3.9: * ``Py_EnterRecursiveCall()``, ``Py_LeaveRecursiveCall()`` (added to the limited C API) * ``PyObject_INIT()``, ``PyObject_INIT_VAR()`` * ``PyObject_GET_WEAKREFS_LISTPTR()`` * ``PyObject_CheckBuffer()`` * ``PyIndex_Check()`` * ``PyObject_IS_GC()`` * ``PyObject_NEW()`` (alias to ``PyObject_New()``), ``PyObject_NEW_VAR()`` (alias to ``PyObject_NewVar()``) * ``PyType_HasFeature()`` (always call ``PyType_GetFlags()``) * ``Py_TRASHCAN_BEGIN_CONDITION()`` and ``Py_TRASHCAN_END()`` macros now call functions which hide implementation details, rather than accessing directly members of the ``PyThreadState`` structure. Make structures opaque ---------------------- All structures of the C API should become opaque: C extensions must use getter or setter functions to get or set structure members. For example, ``tuple->ob_item[0]`` must be replaced with ``PyTuple_GET_ITEM(tuple, 0)``. To be able to move away from reference counting, ``PyObject`` must become opaque. Currently, the reference counter ``PyObject.ob_refcnt`` is exposed in the C API. All structures must become opaque, since they "inherit" from PyObject. For, ``PyFloatObject`` inherits from ``PyObject``:: typedef struct { PyObject ob_base; double ob_fval; } PyFloatObject; Making ``PyObject`` fully opaque requires converting ``Py_INCREF()`` and ``Py_DECREF()`` macros to function calls. This change has an impact on performance. It is likely to be one of the very last changes when making structures opaque. Making ``PyTypeObject`` structure opaque breaks C extensions declaring types statically (e.g. ``static PyTypeObject MyType = {...};``). C extensions must use ``PyType_FromSpec()`` to allocate types on the heap instead. Using heap types has other advantages like being compatible with subinterpreters. Combined with PEP 489 "Multi-phase extension module initialization", it makes a C extension behavior closer to a Python module, like allowing to create more than one module instance. Making ``PyThreadState`` structure opaque requires adding getter and setter functions for members used by C extensions. **STATUS**: In Progress (started in Python 3.8) The ``PyInterpreterState`` structure was made opaque in Python 3.8 (`bpo-35886 <https://bugs.python.org/issue35886>`_) and the ``PyGC_Head`` structure (`bpo-40241 <https://bugs.python.org/issue40241>`_) was made opaque in Python 3.9. Issues tracking the work to prepare the C API to make following structures opaque: * ``PyObject``: `bpo-39573 <https://bugs.python.org/issue39573>`_ * ``PyTypeObject``: `bpo-40170 <https://bugs.python.org/issue40170>`_ * ``PyFrameObject``: `bpo-40421 <https://bugs.python.org/issue40421>`_ * Python 3.9 adds ``PyFrame_GetCode()`` and ``PyFrame_GetBack()`` getter functions, and moves ``PyFrame_GetLineNumber`` to the limited C API. * ``PyThreadState``: `bpo-39947 <https://bugs.python.org/issue39947>`_ * Python 3.9 adds 3 getter functions: ``PyThreadState_GetFrame()``, ``PyThreadState_GetID()``, ``PyThreadState_GetInterpreter()``. Disallow using Py_TYPE() as l-value ----------------------------------- The ``Py_TYPE()`` function gets an object type, its ``PyObject.ob_type`` member. It is implemented as a macro which can be used as an l-value to set the type: ``Py_TYPE(obj) = new_type``. This code relies on the assumption that ``PyObject.ob_type`` can be modified directly. It prevents making the ``PyObject`` structure opaque. New setter functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and ``Py_SET_SIZE()`` are added and must be used instead. The ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` macros must be converted to static inline functions which can not be used as l-value. For example, the ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) becomes:: #define _PyObject_CAST_CONST(op) ((const PyObject*)(op)) static inline PyTypeObject* _Py_TYPE(const PyObject *ob) { return ob->ob_type; } #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob)) **STATUS**: Completed (in Python 3.10) New functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and ``Py_SET_SIZE()`` were added to Python 3.9. In Python 3.10, ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` can no longer be used as l-value and the new setter functions must be used instead. New C API functions must not return borrowed references ------------------------------------------------------- When a function returns a borrowed reference, Python cannot track when the caller stops using this reference. For example, if the Python ``list`` type is specialized for small integers, store directly "raw" numbers rather than Python objects, ``PyList_GetItem()`` has to create a temporary Python object. The problem is to decide when it is safe to delete the temporary object. The general guidelines is to avoid returning borrowed references for new C API functions. No function returning borrowed functions is scheduled for removal by this PEP. **STATUS**: Completed (in Python 3.9) In Python 3.9, new C API functions returning Python objects only return strong references: * ``PyFrame_GetBack()`` * ``PyFrame_GetCode()`` * ``PyObject_CallNoArgs()`` * ``PyObject_CallOneArg()`` * ``PyThreadState_GetFrame()`` Avoid functions returning PyObject** ------------------------------------ The ``PySequence_Fast_ITEMS()`` function gives a direct access to an array of ``PyObject*`` objects. The function is deprecated in favor of ``PyTuple_GetItem()`` and ``PyList_GetItem()``. ``PyTuple_GET_ITEM()`` can be abused to access directly the ``PyTupleObject.ob_item`` member:: PyObject **items = &PyTuple_GET_ITEM(0); The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are converted to static inline functions to disallow that. **STATUS**: Not Started New pythoncapi_compat.h header file ----------------------------------- Making structures opaque requires modifying C extensions to use getter and setter functions. The practical issue is how to keep support for old Python versions which don't have these functions. For example, in Python 3.10, it is no longer possible to use ``Py_TYPE()`` as an l-value. The new ``Py_SET_TYPE()`` function must be used instead:: #if PY_VERSION_HEX >= 0x030900A4 Py_SET_TYPE(&MyType, &PyType_Type); #else Py_TYPE(&MyType) = &PyType_Type; #endif This code may ring a bell to developers who ported their Python code base from Python 2 to Python 3. Python will distribute a new ``pythoncapi_compat.h`` header file which provides new C API functions to old Python versions. Example:: #if PY_VERSION_HEX < 0x030900A4 static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type) { ob->ob_type = type; } #define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type) #endif // PY_VERSION_HEX < 0x030900A4 Using this header file, ``Py_SET_TYPE()`` can be used on old Python versions as well. Developers can copy this file in their project, or even to only copy/paste the few functions needed by their C extension. **STATUS**: In Progress (implemented but not distributed by CPython yet) The ``pythoncapi_compat.h`` header file is currently developer at: https://github.com/pythoncapi/pythoncapi_compat Process to reduce the number of broken C extensions =================================================== Process to reduce the number of broken C extensions when introducing C API incompatible changes listed in this PEP: * Estimate how many popular C extensions are affected by the incompatible change. * Coordinate with maintainers of broken C extensions to prepare their code for the future incompatible change. * Introduce the incompatible changes in Python. The documentation must explain how to port existing code. It is recommended to merge such changes at the beginning of a development cycle to have more time for tests. * Changes which are the most likely to break a large number of C extensions should be announced on the capi-sig mailing list to notify C extensions maintainers to prepare their project for the next Python. * If the change breaks too many projects, reverting the change should be discussed, taking in account the number of broken packages, their importance in the Python community, and the importance of the change. The coordination usually means reporting issues to the projects, or even proposing changes. It does not require waiting for a new release including fixes for every broken project. Since more and more C extensions are written using Cython, rather directly using the C API, it is important to ensure that Cython is prepared in advance for incompatible changes. It gives more time for C extension maintainers to release a new version with code generated with the updated Cython (for C extensions distributing the code generated by Cython). Future incompatible changes can be announced by deprecating a function in the documentation and by annotating the function with ``Py_DEPRECATED()``. But making a structure opaque and preventing the usage of a macro as l-value cannot be deprecated with ``Py_DEPRECATED()``. The important part is coordination and finding a balance between CPython evolutions and backward compatibility. For example, breaking a random, old, obscure and unmaintained C extension on PyPI is less severe than breaking numpy. If a change is reverted, we move back to the coordination step to better prepare the change. Once more C extensions are ready, the incompatible change can be reconsidered. Version History =============== * Version 3, June 2020: PEP rewritten from scratch. Python now distributes a new ``pythoncapi_compat.h`` header and a process is defined to reduce the number of broken C extensions when introducing C API incompatible changes listed in this PEP. * Version 2, April 2020: `PEP: Modify the C API to hide implementation details <https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XK…>`_. * Version 1, July 2017: `PEP: Hide implementation details in the C API <https://mail.python.org/archives/list/python-ideas@python.org/thread/6XATDG…>`_ sent to python-ideas Copyright ========= This document has been placed in the public domain. -- Night gathers, and now my watch begins. It shall not end until my death.

13 37

destructuring with class constructors
by Jim J. Jewett July 2, 2020

July 2, 2020

Nick Coghlan wrote: > On Thu, 2 Jul 2020 at 00:50, Guido van Rossum guido(a)python.org wrote: > > As to allowing object destructuring syntax like > > Point(x, y) = p > > that definitely has merit, but it is a complex subject that should be a separate > > PEP. > This is the main point I'm actually interested in, and I agree it > should be a separate PEP. I also like it, but I don't see a way to actually do it usefully without changing how Python works. The glitch is for cases like: Point(?x, 15) = p If constraints like (y=15) are not allowed, then it doesn't really offer much advantage over a regular constructor like: Point(*p) But if they are allowed, then what does it mean? Point(p[0], 15) would be logical, but still not very valuable. At the very least, it should be able to reject: Point(x, y) = ("Hello", "World") but it isn't clear how much of the constructor (let alone some other arbitrary callable) to run before saying "yeah, it meets the contract", unless type annotations and their "correct" meaning become even more constrained, so that: Point(x:int, y:int) = ("Hello", "World") could fail, because "Hello" is not an int. But that is just a very weak Design By Contract that both constrains the meaning of annotations and still leaves out most interesting contracts. -jJ

1 0

PEP 622 pattern matching syntax
by i.r.galahad July 1, 2020

July 1, 2020

Hello Python Devs, For some time I've been hoping to be able to use pattern matching in Python, so PEP 622 is an exciting proposition. I do have some concerns, mostly about the syntax, because it doesn't currently match the style the rest of Python seems to lay out. The thing I'd most suggest changing is the syntax for constants within patterns from `.CONSTANT`. Dots can already be in patterns as part of attribute access, so the viewer will need to distinguish between different uses when they see one. Instead I propose an at-symbol, like `@CONSTANT`. The `@` is a symbol used in Python syntax already, so it would not require changes to the tokeniser. However, it can never currently appear in patterns; neither decorators nor the matmul operator would create any ambiguity for a human trying to skim over a pattern, unlike the currently proposed syntax. The `@` does share a downside with the dot, which is lack of precedent. The PEP alludes to `.name` having an entirely different use in imports, and the use of `@name` in decorators does not relate to this new pattern syntax either. All the same, `@` would still make for a clearer symbol because it would be more distinct from attributes. (There is a weak argument that `@CONSTANT` already has a precedent because of its use in decorators, because decorators are names that are used at the top level, and because the `@CONSTANT` syntax in patterns would mostly also refer to top level names. However, users will adapt to whichever syntax is settled on.) I have two aesthetic suggestions, which have smaller impacts on clarity than the above. Firstly, replacing `|` for delimiting patterns with the more explicit keyword `or`. Python favours keywords where other languages might use punctuation, such as `range(1, 10)` rather than `1..10`, because full words are closer to natural language than punctuation is. Secondly, I prefer the look of `(_, _) as point` over `point := (_, _)` for named sub-patterns. This one is harder to justify because it's a personal preference. All the same, it's worth considering if you haven't already done so. Finally, I dislike the way the underscore variable name seems like magic. That is, it is a variable name that behaves differently from other names. I recognise that there is a history of Python code treating `_` differently — such as referring to the previous result in the REPL — and I can't offer any better alternatives, just my opinion. I hope you consider these perspectives, especially the change from `.CONSTANT` to `@CONSTANT`. I'm anxious to get my hands on pattern matching syntax, in whatever form it eventually takes. Best wishes, IFcoltransG

1 0

Please refrain from posting on the PEP 8 threads for 24 hours
by Brett Cannon July 1, 2020

July 1, 2020

Regardless of what side you fall on, I think we can agree that emotions are running very high at the moment. Nothing is going to change in at least the next 24 hours, so I am personally asking folks to step back for at least that long and think about: 1. Is what you want to say going to contribute to the discussion? 2. Is it being phrased in a polite, constructive fashion? If after 24 hours of reflection you feel your email still meets these criteria then I would say it's reasonable to send that email.

4 4