Mailman 3 November 2021 - Python-Dev

The current state of typing PEPs
by Barry Warsaw Jan. 24, 2022

Jan. 24, 2022

Does PEP 563 or 649 satisfy static and dynamic typing needs? In the interest of full transparency, we want to let the Python community know that the Steering Council continues to discuss PEP 563 (Postponed Evaluation of Annotations) and PEP 649 (Deferred Evaluation Of Annotations Using Descriptors). We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs. Defer decision on PEP 563 and 649 in 3.11 As such, at this time, the only reasonable path forward that the SC sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users. Calling for help from static and dynamic typing users This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users. We need help in defining those requirements clearly, in uncovering the use cases we’re not aware of, and most importantly, being an interface to typing enthusiasts to help build consensus with either of the proposed PEPs, or some other solution that doesn’t yet have a PEP (such as suggestions made by Łukasz in his blog post here: https://lukasz.langa.pl/61df599c-d9d8-4938-868b-36b67fdb4448/). Volunteer to be a typing PEP shepherd We’re looking for someone(s) who can be neutral as to the solution, knowledgeable enough to understand all sides of the problem, willing to serve as not quite a PEP author, not a PEP Delegate, but more than a PEP shepherd. If you are that person, please let us know! Cheers, -Barry (on behalf of the Python Steering Council)

29 81

RFC on PEP 673: Self Type
by Pradeep Kumar Srinivasan Jan. 20, 2022

Jan. 20, 2022

This PEP [1] introduces a simple and intuitive way to annotate methods and classmethods that return an instance of their class. Such methods and classmethods occur quite frequently, but the existing way to annotate them correctly is quite arcane and error-prone. The PEP introduces a special type `Self` to represent the type of the `self` parameter, similar to the `this` type in TypeScript and the `Self` type in Rust. We have implementations for mypy and pyright. The PEP does not affect CPython directly except for the addition of one special form (Self) to typing.py [2]. Since we have reached consensus on the PEP in typing-sig [3], we wanted to get your comments and suggestions before submitting to the Steering Council. Thanks, Pradeep Kumar Srinivasan James Hilton-Balfe [1]: https://www.python.org/dev/peps/pep-0673/ [2]: Adding `Self` to typing_extensions.py: https://github.com/python/typing/pull/933 [3]: See the comments from typing-sig members on the Google doc: https://docs.google.com/document/d/1ujuSMXDmSIOJpiZyV7mvBEC8P-y55AgSzXcvhrZ…

4 4

SC Acceptance: PEP 646 -- Variadic Generics
by Barry Warsaw Jan. 19, 2022

Jan. 19, 2022

Hello Mark, Matthew, Pradeep, Vincent, and Guido, The Python Steering Council discussed the latest version of PEP 646 (Variadic Generics) at our last meeting, and have unanimously decided to accept the PEP. Congratulations! We want to specifically mention that we appreciate the way you called out the Python grammar changes required by the typing features you proposed. As we’ve said before, the Steering Council strongly believes that the typing language and the “general” Python programming language should remain aligned, so the implications of syntax change proposed in the PEP for typing needed to be addressed for non-typed Python as well. The PEP explains this change well, and does a good job of justifying the semantics and usefulness of the change for non-type related purposes. Please feel free to change the PEP status to Accepted, and to merge your changes to Python 3.11 at your convenience. With our appreciation, -Barry (on behalf of the Python Steering Council)

3 2

PEP 646 (Variadic Generics): final call for comments
by Matthew Rahtz Jan. 18, 2022

Jan. 18, 2022

Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.) Thanks! Matthew and Pradeep

9 19

PEP 674: Disallow using macros as l-value
by Victor Stinner Jan. 12, 2022

Jan. 12, 2022

Hi, I propose to disallow using macros as l-value. Read and comment the plain text below, or read the PEP 674 online, https://python.github.io/peps/pep-0674/ While I'm not sure that the proposed changes are really controversial, I decided to write a formal PEP since the incompatible changes are not following the PEP 387 deprecation process and so a format PEP 387 exception is better. Well, the list of modified macros is also quite long. Moreover, a PEP is a way to document and announce the changes ;-) The Py_TYPE() and Py_SIZE() changes are already approved the Steering Council, but I prefer to list them in the PEP: https://github.com/python/steering-council/issues/79 In practice, I'm only aware of 4 projects impacted by these changes, and I wrote the pythoncapi_compat project which updates automatically C extensions: add Python 3.11 support, without losing support for older Python versions. I already prepared major projects like Cython and numpy for these changes (in total, 14 impacted projects have already been updated). Victor --- PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Oct-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to allow evolving CPython internals and to ease the C API implementation on other Python implementation. In practice, the majority of projects impacted by these incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a struture member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly,and pointers cannot be dereferenced. Disallowing the usage of macros as l-value helps the migration of existing C extensions to HPy by reducing differences between the C API and the HPy API. PyPy cpyext module ------------------ In PyPy, when a Python object is accessed by the Python C API, the PyPy ``cpyext`` module has to convert PyPy object to a CPython object. While PyPy objects are designed to be efficient with the PyPy JIT compiler, CPython objects are less efficient and increase the memory usage. This PEP alone is not enough to get rid of the CPython objects in the PyPy ``cpyext`` module, but it is a step towards this long term goal. PyPy already supports HPy which is a better solution in the long term. Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects impacted by these PEP incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros would require to add a new API to get access to the inner array which is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. In practice, only a minority of third party projects are affected (16 projects are known to be broken) and `most of them have already been updated for these changes <https://bugs.python.org/issue39573#msg401378>`__ (12 on 16). Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes. These two macros have been converted to static inline macro in Python 3.10 alpha versions, but the change had to be reverted since it broke too many projects. In the meanwhile, many projects, like Cython, have been prepared for this change by using ``Py_SET_TYPE()`` and ``Py_SET_SIZE()``. For example, projects using Cython only have to regenerate their outdated generated C code to become compatible. For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI top 5000 projects use these functions as l-value. The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the PyPI top 5000 projects. The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as l-value but only to modify string characters, not to override the ``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member. For example, Cython uses the following code which remains valid:: PyByteArray_AS_STRING(string)[i] = (char) v; This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value). Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not solve issues of the nogil, PyPy and HPy projects. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC3…>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

10 25

Do we need to remove everything that's deprecated?
by Petr Viktorin Dec. 12, 2021

Dec. 12, 2021

We're rebuilding many popular projects with Python 3.11 alpha, and I see many failures like: AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Did you mean: 'RawConfigParser'? (bpo-45173) ImportError: cannot import name 'formatargspec' from 'inspect' (bpo-45320) AttributeError: '[...]Tests' object has no attribute 'failUnless' (bpo-45162) Are these changes necessary? Does it really cost us that much in maintainer effort to keep a well-tested backwards compatibility alias name, or a function that has a better alternative? I think that rather than helping our users, changes like these are making Python projects painful to maintain. If we remove them to make Python easier for us to develop, is it now actually that much easier to maitain? The current backwards compatibility policy (PEP 387) sets a *minimum* timeline for deprecations and removals -- "deprecation period must last at least two years." But it seems like it's not treated as a minimum: if any contributor sends an issue/PR to remove deprecated functionality, it's merged without much discussion. And it's very easy to "solve" these "issues", since everything is already marked for deletion; I fear we get the same kind of bikeshed/powerplant problem https://bikeshed.com/ for changes that explains for discussion. It's just so much easier to do "spring cleaning" than solve other problems. Note that I am criticizing the *process*; the examples I gave have some people's names attached, and I have no doubt the people acted with best intentions. I'm also not talking about code that's buggy, insecure, or genuinely hard to maintain. If deprecation now means "we've come up with a new way to do things, and you have two years to switch", can we have something else that means "there's now a better way to do things; the old way is a bit worse but continues to work as before"?

12 28

Remove asyncore, asynchat and smtpd modules
by Victor Stinner Dec. 3, 2021

Dec. 3, 2021

Hi, The asyncore module is a very old module of the Python stdlib for asynchronous programming, usually to handle network sockets concurrently. It's a common event loop, but its design has many flaws. The asyncio module was added to Python 3.4 with a well designed architecture. Twisted developers, who have like 10 to 20 years of experience in asynchronous programming, helped to design the asyncio API. By design, asyncio doesn't have flaws which would be really hard to fix in asyncore and asynchat. It was decided to start deprecating the asyncore, asynchat and smtpd modules in Python 3.6 released in 2016, 5 years ago. Python 3.10 emits DeprecationWarning. asynchat and smtpd are implemented with asyncore. Open issues in asyncore, asynchat and smtpd have been closed as "wont fix" because these modules are deprecated. These modules are basically no longer maintained. I propose to remove asyncore, aynchat and smtpd in Python 3.11 to reduce the Python maintenance burden, while asyncio remains available in stdlib and is maintained: * asyncore and asynchat can be replaced with asyncio * smtpd can be replaced with aiosmtpd which is based on asyncio: https://aiosmtpd.readthedocs.io/ If someone wants to continue using asyncore, asynchat or smtpd, it's trivial to copy Python 3.10 asyncore.py, asynchat.py and smtpd.py to their project, and maintain these files there. Someone is also free to continue maintaining these modules as third-party projects on PyPI. The removal is discussed at: https://bugs.python.org/issue28533 I wrote a PR to remove the 3 modules: https://github.com/python/cpython/pull/29521 ... in short, the intent is to move the asyncore, asynchat and smtpd maintenance outside the Pyhon project ;-) (if anyone still use them) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

7 11

Oh look, I've been subscribed to python/issues-test-2 notifications again
by Larry Hastings Dec. 3, 2021

Dec. 3, 2021

I guess this is part of the migration from bpo to GitHub issues? Maybe the initial work could be done in a private repo, to cut down on the spurious email notifications to literally everybody subscribed to cpython? Which is a lot of people. //arry/

7 7

Optimizing literal comparisons and contains
by Jeremiah Vivian Nov. 30, 2021

Nov. 30, 2021

Many operations involving two literals are optimized (to a certain level). So it sort of surprises me that literal comparisons are not optimized and literal contains only convert the right operand to a constant if possible. I'd like to implement optimizations for these especially for the literal contains. There is a TODO in the ast optimizer for literal comparisons as well, and that's another reason I would like to have these added.

15 23

Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
by ptmcg＠austin.rr.com Nov. 29, 2021

Nov. 29, 2021

I’ve not been following the thread, but Steve Holden forwarded me the email from Petr Viktorin, that I might share some of the info I found while recently diving into this topic. As part of working on the next edition of “Python in a Nutshell” with Steve, Alex Martelli, and Anna Ravencroft, Alex suggested that I add a cautionary section on homoglyphs, specifically citing “A” (LATIN CAPITAL LETTER A) and “Α” (GREEK CAPITAL LETTER ALPHA) as an example problem pair. I wanted to look a little further at the use of characters in identifiers beyond the standard 7-bit ASCII, and so I found some of these same issues dealing with Unicode NFKC normalization. The first discovery was the overlapping normalization of “ªº” with “ao”. This was quite a shock to me, since I assumed that the inclusion of Unicode for identifier characters would preserve the uniqueness of the different code points. Even ligatures can be used, and will overlap with their multi-character ASCII forms. So we have added a second note in the upcoming edition on the risks of using these “homonorms” (which is a word I just made up for the occasion). To explore the extreme case, I wrote a pyparsing transformer to convert identifiers in a body of Python source to mixed font, equivalent to the original source after NFKC normalization. Here are hello.py, and a snippet from unittest/utils.py: def 𝚑𝓮𝖑𝒍𝑜(): try: 𝔥e𝗅𝕝𝚘︴ = "Hello" 𝕨𝔬r𝓵ᵈ﹎ = "World" ᵖ𝖗𝐢𝘯𝓽(f"{𝗵ｅ𝓵𝔩º_}, {𝖜ₒ𝒓lⅆ︴}!") except 𝓣𝕪ᵖｅ𝖤𝗿ᵣ𝖔𝚛 as ⅇ𝗑c: 𝒑rℹₙₜ("failed: {}".𝕗𝗼ʳᵐªｔ(ᵉ𝐱𝓬)) if _︴ⁿ𝓪𝑚𝕖__ == "__main__": 𝒉eℓˡ𝗈() # snippet from unittest/util.py _𝓟Ⅼ𝖠𝙲𝗘ℋ𝒪Lᴰ𝑬𝕽﹏𝕷𝔼𝗡 = 12 def _𝔰ʰ𝓸ʳ𝕥𝙚𝑛(𝔰, p𝑟𝔢ﬁ𝖝𝕝𝚎𝑛, ｓᵤ𝑓𝗳𝗂𝑥𝗹ₑ𝚗): ˢ𝗸ｉ𝗽 = 𝐥ｅ𝘯(𝖘) - ｐr𝚎𝖋𝐢x𝗅ᵉ𝓷 - 𝒔𝙪ﬀｉ𝘅𝗹𝙚ₙ if sｋi𝘱 > _𝐏𝗟𝖠𝘊𝙴H𝕺Ｌ𝕯𝙀𝘙﹏L𝔈𝒩: 𝘴 = '%s[%d chars]%s' % (𝙨[:𝘱𝐫𝕖𝑓𝕚ｘℓ𝒆𝕟], ₛ𝚔𝒊p, 𝓼[𝓁𝒆𝖓(𝚜) - 𝙨𝚞𝒇ﬁx𝙡ᵉ𝘯:]) return ₛ You should able to paste these into your local UTF-8-aware editor or IDE and execute them as-is. (If this doesn’t come through, you can also see this as a GitHub gist at Hello, World rendered in a variety of Unicode characters (github.com) <https://gist.github.com/ptmcg/bf35d5ada416080d481d789988b6b466> . I have a second gist containing the transformer, but it is still a private gist atm.) Some other discoveries: “·” (ASCII 183) is a valid identifier body character, making “_···” a valid Python identifier. This could actually be another security attack point, in which “s·join(‘x’)” could be easily misread as “s.join(‘x’)”, but would actually be a call to potentially malicious method “s·join”. “_” seems to be a special case for normalization. Only the ASCII “_” character is valid as a leading identifier character; the Unicode characters that normalize to “_” (any of the characters in “︳︴﹍﹎﹏＿”) can only be used as identifier body characters. “︳” especially could be misread as “|” followed by a space, when it actually normalizes to “_”. Potential beneficial uses: I am considering taking my transformer code and experimenting with an orthogonal approach to syntax highlighting, using Unicode groups instead of colors. Module names using characters from one group, builtins from another, program variables from another, maybe distinguish local from global variables. Colorizing has always been an obvious syntax highlight feature, but is an accessibility issue for those with difficulty distinguishing colors. Unlike the “ransom note” code above, code highlighted in this way might even be quite pleasing to the eye. -- Paul McGuire

19 36