Mailman 3 April 2021 - Python-Dev

Let's Fix Class Annotations -- And Maybe Annotations Generally
by Larry Hastings April 24, 2021

April 24, 2021

Howdy howdy. While working on my PEP I stumbled over a lot of behavior by annotations that I found inconsistent and inconvenient. I think there are several problems here that needs fixing. This discussion will probably evolve into a PEP, and I'll be happy to steer that process. But I'm less certain about what the right thing to do is. (Although I do know what I'd prefer!) So let's talk about it! Annotations are represented in Python as a dictionary. They can be present on functions, classes, and modules as an attribute called "__annotations__". We start with: how do you get the annotations from one of these objects? Surely it's as easy as this line from Lib/inspect.py shows us: return func.__annotations__ And yes, that's best practice for getting an annotation from a function object. But consider this line from Lib/functools.py: ann = getattr(cls, '__annotations__', {}) Huh. Why doesn't it simply look at cls.__annotations__? It's because the language declares that __annotations__ on a class or module is optional. Since cls.__annotations__ may not be defined, evaluating that might throw an exception. Three-argument getattr() is much safer, and I assert it's best practice for getting the annotations from a module. But consider this line from Lib/dataclasses.py: cls_annotations = cls.__dict__.get('__annotations__', {}) And a very similar line from Lib/typing.py: ann = base.__dict__.get('__annotations__', {}) Huh! Why is this code skipping the attribute entirely, and examining cls.__dict__? It's because the getattr() approach has a subtle bug when dealing with classes. Consider this example: class A: ax:int=3 class B(A): pass print(getattr(B, '__annotations__', {})) That's right, B *inherits* A.__annotations__! So this prints {'ax': int}. This *can't* the intended behavior of __annotations__ on classes. It's only supposed to contain annotations for the fields of B itself, not those of one of its randomly-selected base classes. But that's how it behaves today--and people have had to work around this behavior for years. Examining the class dict is, sadly, best practice for getting __annotations__ from a class. So, already: three different objects can have __annotations__, and there are three different best practices for getting their __annotations__. Let's zoom out for a moment. Here's the list of predefined data fields you can find on classes: __annotations__ __bases__ __class__ __dict__ __doc__ __module__ __mro__ __name__ __qualname__ All of these describe metadata about the class. In every case *except one*, the field is mandatory, which also means it's never inherited. And in every case *except one*, you cannot delete the field. (Though you *are* allowed to overwrite some of them.) You guessed it: __annotations__ is the exception. It's optional, and you're allowed to delete it. And these exceptions are causing problems. It seems to me that, if the only way to correctly use a language-defined attribute of classes is by rooting around in its __dict__, the design is a misfire. (Much of the above also applies to modules, too. The big difference: since modules lack inheritance, you don't need to look in their __dict__.) Now consider what happens if my "delayed annotations of annotations using descriptors" PEP is accepted. If that happens, pulling __annotations__ out of the class dict won't work if they haven't been generated yet. So today's "best practice" becomes tomorrow's "this code doesn't work". To correctly examine class annotations, code would have to do something like this, which should work correctly in any Python 3.x version: if (getattr(cls, '__co_annotations__', None) or ('__annotations__' in cls.__dict__)): ann = cls.__annotations__ else: ann = {} This is getting ridiculous. Let's move on to a related topic. For each of the objects that can have annotations, what happens if o.__annotations__ is set, and you "del o.__annotations__", then you access o.__annotations__? It depends on what the object is, because each of them behaves differently. You already know what happens with classes: if any of the base classes has __annotations__ set, you'll get the first one you find in the MRO. If none of the bases have __annotations__ set you'll get an AttributeError. For a module, if you delete it then try to access it, you'll always get an AttributeError. For a function, if you delete it then try to get it, the function will create a new empty dict, store it as its new annotations dict, and return that. Why does it do that? I'm not sure. The relevent PEP (3107) doesn't specify this behavior. So, annotations can be set on three different object types, and each of those three have a different behavior when you delete the annotations then try to get them again. As a final topic: what are the permitted types for __annotations__? If you say "o.__annotations__ = <x>", what types are and aren't allowed for <x>? For functions, __annotations__ may be assigned to either None or a dict (an object that passes PyDict_Check). Anything else throws a TypeError. For classes and modules, no checking is done whatsoever, and you can set __annotations__ on those two to any Python object. While "a foolish consistency is the hobgoblin of little minds", I don't see the benefit of setting a module's __annotations__ to 2+3j. I think it's long past time that we cleaned up the behavior of annotations. They should be simple and consistent across all objects that support them. At the very least, I think we should make cls.__annotations__ required rather than optional, so that it's never inherited. What should its default value be? An empty dict would be more compatible, but None would be cheaper. Note that creating the empty dict on the fly, the way function objects do, wouldn't really help--because current best practice means looking in cls.__dict__. I also think you shouldn't be able to delete __annotations__ on any of the three objects (function, class, module). It should always be set, so that the best practice for accessing annotations on an object is always o.__annotations__. If I could wave my magic wand and do whatever I wanted, I'd change the semantics for __annotations__ to the following: * Functions, classes, and modules always have an __annotations__ member set. * "del o.__annotations__" always throws a TypeError. * The language will set __annotations__ to a dict if the object has annotations, or None if it has no annotations. * You may set __annotations__, but you can only set it to either None or a dict (passes PyDict_Check). * You may only access __annotations__ as an attribute, and because it's always set, best practice is to use "o.__annotations__" (though getattr will always work too). * How __annotations__ is stored is implementation-specific behavior; looking in the relevant __dict__ is unsupported. This would grant sanity and consistency to __annotations__ in a way it's never so far enjoyed. The problem is, it's a breaking change. But the existing semantics are kind of terrible, so at this point my goal is to break them. I think the best practice needs to stop requiring examining cls.__dict__; in fact I'd prefer people stop doing it altogether. If we change the behavior as part of a new release of Python, code that examines annotations on classes can do a version check: if (sys.version_info.major >=3 and sys.version_info.minor >= 10): def get_annotations(o): return o.__annotations__ or {} else: def get_annotations(o): # eight or ten lines of complex code goes here ... Or code can just use inspect.get_type_hints(), which is tied to the Python version anyway and should always do the right thing. //arry/

10 44

Keeping Python a Duck Typed Language.
by Mark Shannon April 24, 2021

April 24, 2021

Hi everyone, Once upon a time Python was a purely duck typed language. Then came along abstract based classes, and some nominal typing starting to creep into the language. If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence. Then came type hints. PEP 484 explicitly said that type hints were optional and would *always* be optional. Then came along many typing PEPs that assumed that type hints would only used for static typing, making static typing a bit less optional. Not only that, but the type system proposed by many of these PEPs was clearly nominal, not structural. PEP 544 supports structural typing, but to declare a structural type you must inherit from Protocol. That smells a lot like nominal typing to me. Then came PEP 563 and said that if you wanted to access the annotations of an object, you needed to call typing.get_type_hints() to get annotations in a meaningful form. This smells a bit like enforced static typing to me. Then came PEP 634 (structural pattern matching). Despite having the word 'structural' in the name, PEP 634 insists on nominal typing to distinguish between sequences and mappings. Nominal typing in a dynamically typed language makes little sense. It gains little or no safety, but restricts the programs you can write. Because a class can claim to be a nominal type, but not match it structurally, it can appear to be type safe but fail at runtime. Conversely nominal typing errors can result in failures where structurally typed programs would work. An extreme example of that is this: # Some magic code to mess with collections.abc.Sequence >>> match {}: ... case []: ... print("WTF!") ... WTF! With duck typing this would be impossible (unless you use ctypes to mess with the dict object). To be fair, nominal typing is not always a problem. All exceptions must inherit from BaseException, and it doesn't seem to be a problem in practice. So, lets stick to our promise that type hints will always be optional, and restore duck typing. I'm not suggesting that we get rid type hints and abstract base classes. They are popular for a reason. But let's treat them as useful tools, not warp the rest of the language to fit them. Cheers, Mark. Quick summaries of type systems: https://en.wikipedia.org/wiki/Nominal_type_system https://en.wikipedia.org/wiki/Structural_type_system https://en.wikipedia.org/wiki/Duck_typing Or, if you're really keen: https://www.cis.upenn.edu/~bcpierce/tapl/

24 51

Asking for clarifications in PEP 646 and postponing to Python 3.11
by Brett Cannon April 24, 2021

April 24, 2021

First, the SC is postponing considering accepting https://www.python.org/dev/peps/pep-0646/ any further until Python 3.11. There's unfortunately not enough time before b1 to get requested updates into the PEP, discuss the PEP again, approve it, and then review an implementation thoroughly. Second, this PEP is proposing new syntax for Python but that detail is barely touched upon. For instance, there isn't a direct proposal of what grammar changes that should be made. There's also no discussion of what the exact runtime semantics should be. While a lot could be inferred by "it's allowing * within indexing contexts" (which is basically what the PEP is proposing from a syntactic perspective), that isn't a spec. 😉 As such, we are asking that the PEP be expanded upon to fully explain the proposed syntactic change, it's impact on the language and it's spec, etc. Once this update happens, we will be happy to look at the PEP again for consideration of inclusion in Python 3.11. -Brett (on behalf of the SC)

2 1

Interaction of PEP 563's Annotations with Symbol Table
by Batuhan Taskaya April 24, 2021

April 24, 2021

About 4 months ago, I've created bpo-42725 in order to resolve some of the points below though I decided to postpone it for various reasons that blocked it completely until the point of where we have a beta cut in a week or so. So I'm asking for comments regarding how should we resolve this behavior. Even though PEP 563 blocks having all compiler-related activities for annotations by behaving them strings, this doesn't happen until the symbol is constructed which results with different problems. For example you can use yield in annotation, and since PEP 563 will stringify it you'd expect your function to act accordingly but it currently doesn't. Also references to the upper scope variables will still be contained as closures of the current function, and some other oddities that I've found regarding misc stuff (usage of super() etc). Here is a quick demo: def foo(): for number in range(5): foo: (yield number) return number foo() # ======> this will be a generator We could go with acting these as strings and letting them stay as annotations but making them no-op on the symbol table (which is the straight forward choice) or we could restrict usage of those inside of annotations and raise SyntaxErrors. I've drafted patches for both of those ideas, though it definitely needs more work and there is not much time left so any comments would be appreciated. P.S: I intend to submit a PR with the most likely option by, at latest, end of saturday so that we can give reviewers enough time and also catch the beta freeze considering this is a clear behavioral change.

3 2

Request For Review: Fix test_socket.CANTest.testSendFrame (bpo-40297)
by Karl Ding April 23, 2021

April 23, 2021

Hi all, Could someone help take a look at the following PR, which fixes one of the broken socket module tests that exercises part of the standard library? PR Link: https://github.com/python/cpython/pull/19548 BPO Link: https://bugs.python.org/issue40297 The test can be run locally on Linux by setting up the virtual SocketCAN interface the test expects: # Load SocketCAN vcan kernel module sudo modprobe vcan # Start and set up a virtual SocketCAN interface # Note: The SocketCAN tests expect a network interface called 'vcan0' to exist sudo ip link add dev vcan0 type vcan sudo ip link set up vcan0 # Run the socket tests using locally built Python ./python -m unittest -v test.test_socket.CANTest Thanks! -- Karl Ding

1 0

Summary of Python tracker Issues
by Python tracker April 23, 2021

April 23, 2021

ACTIVITY SUMMARY (2021-04-16 - 2021-04-23) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7474 (-48) closed 48187 (+101) total 55661 (+53) Open issues with patches: 2971 Issues opened (41) ================== #10663: configure shouldn't set a default OPT https://bugs.python.org/issue10663 reopened by Arfrever #28254: Add C API for gc.enable, gc.disable, and gc.isenabled https://bugs.python.org/issue28254 reopened by scoder #38605: [typing] PEP 563: Postponed evaluation of annotations: enable https://bugs.python.org/issue38605 reopened by pablogsal #38659: enum classes cause slow startup time https://bugs.python.org/issue38659 reopened by pablogsal #43874: argparse crashes on subparsers with no dest/metava https://bugs.python.org/issue43874 opened by terence.honles #43875: glob.glob with ** does not always detect symlink loops https://bugs.python.org/issue43875 opened by rrt #43877: Logging Cookbook ambiguity https://bugs.python.org/issue43877 opened by bluebloodpole #43878: ./configure fails on Apple Silicon with coreutils uname https://bugs.python.org/issue43878 opened by keith #43879: Add native_thread_id to PyThreadState https://bugs.python.org/issue43879 opened by Gabriele Tornetta #43880: 3.10 SSL module deprecations https://bugs.python.org/issue43880 opened by christian.heimes #43881: add platform availabity information for os.sched_getaffinity https://bugs.python.org/issue43881 opened by guoci #43882: [security] urllib.parse should sanitize urls containing ASCII https://bugs.python.org/issue43882 opened by orsenthil #43883: Making urlparse WHATWG conformant https://bugs.python.org/issue43883 opened by orsenthil #43884: Cannot cleanly kill a subprocess using high-level asyncio APIs https://bugs.python.org/issue43884 opened by rabraham #43886: Extending/embedding Python documentation outdated/incomplete https://bugs.python.org/issue43886 opened by fede.evol #43888: GitHub Actions CI/CD `Coverage` job is broken on master https://bugs.python.org/issue43888 opened by webknjaz #43889: Pickle performance regression in 3.10 https://bugs.python.org/issue43889 opened by kj #43890: Deadlock when mixing event loops and subprocesses https://bugs.python.org/issue43890 opened by thomas #43891: co_annotations branch caused a crash in stackeffect() in compi https://bugs.python.org/issue43891 opened by larry #43892: Make match patterns explicit in the AST https://bugs.python.org/issue43892 opened by ncoghlan #43893: typing.get_type_hints does not accept type annotations with le https://bugs.python.org/issue43893 opened by jarryshaw #43894: IDLE editor file minor refactoring https://bugs.python.org/issue43894 opened by epaine #43895: Unnecessary Cache of Shared Object Handles https://bugs.python.org/issue43895 opened by Ian.H #43897: Implement support for validation of pattern matching ASTs https://bugs.python.org/issue43897 opened by BTaskaya #43899: separate builtin function https://bugs.python.org/issue43899 opened by alvarezdqal #43901: Add an empty annotations dict to all unannotated classes and m https://bugs.python.org/issue43901 opened by larry #43902: ssl module: add getter for SSL_CTX* and SSL* https://bugs.python.org/issue43902 opened by christian.heimes #43905: dataclasses.astuple does deepcopy on all fields https://bugs.python.org/issue43905 opened by mandolaerik #43907: pickle.py bytearray memoization bug with protocol 5 https://bugs.python.org/issue43907 opened by Carl.Friedrich.Bolz #43908: array.array should remain immutable https://bugs.python.org/issue43908 opened by gvanrossum #43910: cgi.parse_header does not handle escaping correctly https://bugs.python.org/issue43910 opened by msg555 #43912: http.client.BadStatusLine raised and response contains request https://bugs.python.org/issue43912 opened by andymaier #43913: unittest module cleanup functions not run unless tearDownModul https://bugs.python.org/issue43913 opened by rtarpine #43914: Highlight invalid ranges in SyntaxErrors https://bugs.python.org/issue43914 opened by pablogsal #43916: Check that new heap types cannot be created uninitialised https://bugs.python.org/issue43916 opened by pablogsal #43918: anext builtin docstring has no signature text or info about de https://bugs.python.org/issue43918 opened by eriknw #43921: test_ssl fails on Windows buildbots https://bugs.python.org/issue43921 opened by pablogsal #43922: Double dots in quopri transported emails https://bugs.python.org/issue43922 opened by Julien Castiaux #43923: Can't create generic NamedTuple as of py3.9 https://bugs.python.org/issue43923 opened by FHTMitchell #43924: print unexpected values in infinite loop https://bugs.python.org/issue43924 opened by rafihassan190041234 #43925: Add hangul syllables to unicodedata.decomposititon https://bugs.python.org/issue43925 opened by frederic.grosshans Most recent 15 issues with no replies (15) ========================================== #43925: Add hangul syllables to unicodedata.decomposititon https://bugs.python.org/issue43925 #43924: print unexpected values in infinite loop https://bugs.python.org/issue43924 #43923: Can't create generic NamedTuple as of py3.9 https://bugs.python.org/issue43923 #43918: anext builtin docstring has no signature text or info about de https://bugs.python.org/issue43918 #43913: unittest module cleanup functions not run unless tearDownModul https://bugs.python.org/issue43913 #43910: cgi.parse_header does not handle escaping correctly https://bugs.python.org/issue43910 #43907: pickle.py bytearray memoization bug with protocol 5 https://bugs.python.org/issue43907 #43897: Implement support for validation of pattern matching ASTs https://bugs.python.org/issue43897 #43894: IDLE editor file minor refactoring https://bugs.python.org/issue43894 #43890: Deadlock when mixing event loops and subprocesses https://bugs.python.org/issue43890 #43881: add platform availabity information for os.sched_getaffinity https://bugs.python.org/issue43881 #43879: Add native_thread_id to PyThreadState https://bugs.python.org/issue43879 #43875: glob.glob with ** does not always detect symlink loops https://bugs.python.org/issue43875 #43870: C API Functions Bypass __import__ Override https://bugs.python.org/issue43870 #43863: [Windows] test_distutils logs: DeprecationWarning: bdist_msi https://bugs.python.org/issue43863 Most recent 15 issues waiting for review (15) ============================================= #43922: Double dots in quopri transported emails https://bugs.python.org/issue43922 #43921: test_ssl fails on Windows buildbots https://bugs.python.org/issue43921 #43918: anext builtin docstring has no signature text or info about de https://bugs.python.org/issue43918 #43914: Highlight invalid ranges in SyntaxErrors https://bugs.python.org/issue43914 #43910: cgi.parse_header does not handle escaping correctly https://bugs.python.org/issue43910 #43908: array.array should remain immutable https://bugs.python.org/issue43908 #43907: pickle.py bytearray memoization bug with protocol 5 https://bugs.python.org/issue43907 #43902: ssl module: add getter for SSL_CTX* and SSL* https://bugs.python.org/issue43902 #43897: Implement support for validation of pattern matching ASTs https://bugs.python.org/issue43897 #43894: IDLE editor file minor refactoring https://bugs.python.org/issue43894 #43893: typing.get_type_hints does not accept type annotations with le https://bugs.python.org/issue43893 #43892: Make match patterns explicit in the AST https://bugs.python.org/issue43892 #43891: co_annotations branch caused a crash in stackeffect() in compi https://bugs.python.org/issue43891 #43888: GitHub Actions CI/CD `Coverage` job is broken on master https://bugs.python.org/issue43888 #43881: add platform availabity information for os.sched_getaffinity https://bugs.python.org/issue43881 Top 10 most discussed issues (10) ================================= #43892: Make match patterns explicit in the AST https://bugs.python.org/issue43892 17 msgs #38659: enum classes cause slow startup time https://bugs.python.org/issue38659 16 msgs #43817: Add inspect.get_annotations() https://bugs.python.org/issue43817 11 msgs #43908: array.array should remain immutable https://bugs.python.org/issue43908 11 msgs #43878: ./configure fails on Apple Silicon with coreutils uname https://bugs.python.org/issue43878 9 msgs #43901: Add an empty annotations dict to all unannotated classes and m https://bugs.python.org/issue43901 8 msgs #37751: In codecs, function 'normalizestring' should convert both spac https://bugs.python.org/issue37751 7 msgs #43804: "Building C and C++ Extensions on Windows" docs are very out-o https://bugs.python.org/issue43804 7 msgs #43795: Implement PEP 652 -- Maintaining the Stable ABI https://bugs.python.org/issue43795 6 msgs #43833: Unexpected Parsing of Numeric Literals Concatenated with Boole https://bugs.python.org/issue43833 6 msgs Issues closed (84) ================== #7247: test_fcntl_64_bit from test_fcntl.py fails in Python 2.6.4 https://bugs.python.org/issue7247 closed by gregory.p.smith #25786: contextlib.ExitStack introduces a cycle in exception __context https://bugs.python.org/issue25786 closed by iritkatriel #27763: Add complex case to test_builtin abs() https://bugs.python.org/issue27763 closed by mark.dickinson #28022: SSL releated deprecation for 3.6 https://bugs.python.org/issue28022 closed by christian.heimes #28453: SSLObject.selected_alpn_protocol() not documented https://bugs.python.org/issue28453 closed by christian.heimes #28724: Add method send_io, recv_io to the socket module. https://bugs.python.org/issue28724 closed by methane #30069: External library behave differently when loaded by Python (may https://bugs.python.org/issue30069 closed by iritkatriel #30226: Modernize make_ssl_certs https://bugs.python.org/issue30226 closed by christian.heimes #30800: zlib.compressobj took too much memory on window https://bugs.python.org/issue30800 closed by iritkatriel #31213: __context__ reset to None in nested exception https://bugs.python.org/issue31213 closed by iritkatriel #31453: Debian Sid/Buster: Cannot enable TLS 1.0/1.1 with PROTOCOL_TLS https://bugs.python.org/issue31453 closed by christian.heimes #31711: ssl.SSLSocket.send(b"") fails https://bugs.python.org/issue31711 closed by christian.heimes #32813: SSL shared_ciphers implementation wrong - returns configured b https://bugs.python.org/issue32813 closed by christian.heimes #32951: Prohibit direct instantiation of SSLSocket and SSLObject https://bugs.python.org/issue32951 closed by christian.heimes #33023: Unable to copy ssl.SSLContext https://bugs.python.org/issue33023 closed by christian.heimes #33808: ssl.get_server_certificate fails with openssl 1.1.0 but works https://bugs.python.org/issue33808 closed by christian.heimes #34001: LibreSSL does not tolerate setting minimum_version greater tha https://bugs.python.org/issue34001 closed by christian.heimes #34028: Python 3.7.0 wont compile with SSL Support 1.1.0 > alledged mi https://bugs.python.org/issue34028 closed by christian.heimes #34078: Broken CRL functionality in ssl.py https://bugs.python.org/issue34078 closed by christian.heimes #34271: Please support logging of SSL master secret by env variable SS https://bugs.python.org/issue34271 closed by christian.heimes #34391: test_ftplib is failing with TLS 1.3 https://bugs.python.org/issue34391 closed by christian.heimes #34669: test_ssl fails if SSLv2 is enabled https://bugs.python.org/issue34669 closed by christian.heimes #34670: Add set_post_handshake_auth for TLS 1.3 https://bugs.python.org/issue34670 closed by christian.heimes #35114: ssl.RAND_status docs describe it as returning True/False; actu https://bugs.python.org/issue35114 closed by christian.heimes #35136: test_ssl fails in AMD64 FreeBSD CURRENT Shared 3.6 buildbot, O https://bugs.python.org/issue35136 closed by christian.heimes #35306: [Windows] OSError when testing whether pathlib.Path('*') exist https://bugs.python.org/issue35306 closed by steve.dower #35422: misleading error message from ssl.get_server_certificate() whe https://bugs.python.org/issue35422 closed by christian.heimes #35888: ssl module - could not get the server certificate w/o complete https://bugs.python.org/issue35888 closed by christian.heimes #36076: ssl.get_server_certificate should use SNI https://bugs.python.org/issue36076 closed by christian.heimes #36137: SSL verification fails for some sites inside windows docker co https://bugs.python.org/issue36137 closed by christian.heimes #36868: New behavior of OpenSSL hostname verification not exposed, inc https://bugs.python.org/issue36868 closed by christian.heimes #37550: SSL Pip Error https://bugs.python.org/issue37550 closed by christian.heimes #38822: [Windows] Inconsistent os.stat behavior for directory with Acc https://bugs.python.org/issue38822 closed by steve.dower #39344: Getting error while importing ssl " import _ssl # if we can't https://bugs.python.org/issue39344 closed by christian.heimes #40079: NULL pointer deref on error path in _ssl debughelpers.c https://bugs.python.org/issue40079 closed by christian.heimes #40482: _hashlib: register Python names as OpenSSL aliases https://bugs.python.org/issue40482 closed by christian.heimes #40849: Expose X509_V_FLAG_PARTIAL_CHAIN ssl flag https://bugs.python.org/issue40849 closed by christian.heimes #41195: Interface to OpenSSL's security level https://bugs.python.org/issue41195 closed by christian.heimes #41701: Buildbot web page: connection lost after 1 minute, then displa https://bugs.python.org/issue41701 closed by vstinner #41943: unittest.assertLogs passes unexpectedly https://bugs.python.org/issue41943 closed by iritkatriel #42128: Structural Pattern Matching (PEP 634) https://bugs.python.org/issue42128 closed by brandtbucher #42166: corrupted size vs. prev_size https://bugs.python.org/issue42166 closed by christian.heimes #42333: Port ssl module to heap types and module state (PEP 573) https://bugs.python.org/issue42333 closed by christian.heimes #42422: types.CodeType() has no bytecode verifier https://bugs.python.org/issue42422 closed by benjamin.peterson #42746: python3.7.3 - ssl.SSLContext() - "Killed" https://bugs.python.org/issue42746 closed by christian.heimes #42854: OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex() https://bugs.python.org/issue42854 closed by christian.heimes #43000: All SSL requests fail with WRONG_VERSION_NUMBER when a packet https://bugs.python.org/issue43000 closed by christian.heimes #43284: sys.getwindowsversion().platform_version is incorrect https://bugs.python.org/issue43284 closed by steve.dower #43362: Bad free in py_sha3_new_impl function https://bugs.python.org/issue43362 closed by christian.heimes #43472: [security][subinterpreters] Add auditing hooks to subinterpret https://bugs.python.org/issue43472 closed by steve.dower #43475: Worst-case behaviour of hash collision with float NaN https://bugs.python.org/issue43475 closed by rhettinger #43522: SSLContext.hostname_checks_common_name appears to have no effe https://bugs.python.org/issue43522 closed by christian.heimes #43538: [Windows] support args and cwd in os.startfile() https://bugs.python.org/issue43538 closed by steve.dower #43602: Include Decimal's in numbers.Real https://bugs.python.org/issue43602 closed by rhettinger #43607: urllib's request.pathname2url not compatible with extended-len https://bugs.python.org/issue43607 closed by steve.dower #43641: Update `ssl.PROTOCOL_TLSv1_2` docs since it is not the newest https://bugs.python.org/issue43641 closed by christian.heimes #43669: PEP 644: Require OpenSSL 1.1.1 or newer https://bugs.python.org/issue43669 closed by christian.heimes #43756: About updating audit events when function gains new arguments https://bugs.python.org/issue43756 closed by steve.dower #43802: Seg fault on macOS using multiprocessing.JoinableQueue https://bugs.python.org/issue43802 closed by jacobtylerwalls #43811: Run GHA CI with multiple OpenSSL versions https://bugs.python.org/issue43811 closed by christian.heimes #43834: Use context manager in StringIO example https://bugs.python.org/issue43834 closed by rhettinger #43837: Operator precedence documentation could be more clear https://bugs.python.org/issue43837 closed by rhettinger #43846: Control stack usage in large expressions https://bugs.python.org/issue43846 closed by Mark.Shannon #43852: [sqlite3] Improve tuple creation https://bugs.python.org/issue43852 closed by berker.peksag #43856: Docs for importlib.metadata should mention Python version https://bugs.python.org/issue43856 closed by jaraco #43859: Improve syntax error for indentation errors https://bugs.python.org/issue43859 closed by pablogsal #43866: Installation files of the Python https://bugs.python.org/issue43866 closed by steve.dower #43872: Unable to compile cpython in Visual Studio 2019 https://bugs.python.org/issue43872 closed by shreyanavigyan #43873: bz2.open() docs give conflicting defaults for mode https://bugs.python.org/issue43873 closed by ned.deily #43876: argparse documentation contrasting nargs '*' vs. '+' is misle https://bugs.python.org/issue43876 closed by rhettinger #43885: ResourceWarning: unclosed <ssl.SSLSocket ...> test_pha_require https://bugs.python.org/issue43885 closed by christian.heimes #43887: it seems that sorted built-in always return floats before int https://bugs.python.org/issue43887 closed by steven.daprano #43896: Update the Sphinx directive for super from function to class https://bugs.python.org/issue43896 closed by rhettinger #43898: Python fails to import .dylib but works when extension is chan https://bugs.python.org/issue43898 closed by ned.deily #43900: string comprehension https://bugs.python.org/issue43900 closed by rhettinger #43903: round() produces incorrect results with certain values https://bugs.python.org/issue43903 closed by eric.smith #43904: Documentation for argparse, missed opportunity https://bugs.python.org/issue43904 closed by rhettinger #43906: turtle graphics don't work https://bugs.python.org/issue43906 closed by eric.smith #43909: Fuzz dis module and find crashes for dis.dis(), dis.get_instr https://bugs.python.org/issue43909 closed by serhiy.storchaka #43911: Queue.get() memory leak https://bugs.python.org/issue43911 closed by rhettinger #43915: Add PCbuild/blurb.bat https://bugs.python.org/issue43915 closed by steve.dower #43917: An error in classmethod example in the documentation of descri https://bugs.python.org/issue43917 closed by rhettinger #43919: Potential bug in the "request" package. https://bugs.python.org/issue43919 closed by xtreak #43920: OpenSSL 3.0.0: handle empty cadata consistently https://bugs.python.org/issue43920 closed by christian.heimes

1 0

Re: Keeping Python a Duck Typed Language.
by Luciano Ramalho April 22, 2021

April 22, 2021

On Wed, Apr 21, 2021 at 4:44 PM Christopher Barker <pythonchb(a)gmail.com> wrote: >> You say this like it's a bad thing, but how is this avoidable, even in >> principle? Structural typing lets you check whether Foo is duck-shaped >> -- has appropriate attribute names, etc. But quacking like a duck is >> harder: you also have to implement the Sequence behavioral contract, >> and realistically the only way to know that is if the author of Foo >> tells you. > > > But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck. I agree with Christopher here. For duck typing, all we care is that the method(s) we need are implemented in a sensible way. > And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever). Also agree, and very often the function cares only about one single method. > And a lot of the standard library works exactly this way. Totally. I found it astonishing that PEP 484 did not allow to correctly annotate LOTS of functions in the standard library, including built-ins people use every day like `sorted()`. Fortunately, that was corrected with PEP 544. > But there is the "culture" of Python -- and it has been very much shifting toward more typing I disagree because Python is too widespread these days to talk about "the culture". We'll just need to learn to coexist as groups that use the language differently. -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there? Maybe you mean this one: "Python 3 types in the wild: a tale of two type systems" Metadata: https://dl.acm.org/doi/10.1145/3426422.3426981 PDF: http://hirzels.com/martin/papers/dls20-python-types.pdf > And I've seen a lot more isinstance(Some_ABC) code lately as well. I believe a lot of that is people coming from statically typed languages and not feeling comfortable with duck typing, while for some other reason not adopting type hints. > From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). It is considered best practice in some places. I hope we as a community never adopt it as the canonical way. > Maybe it is -- but if the community is moving that way, we should be honest about it. Clearly parts of the community are moving that way. The paper I linked shows the use of type hints is still very limited in their sample. >> I'm not even sure that this *is* nominal typing. You could just as >> well argue that "the operation `isinstance(..., Sequence)` returns >> `True`" is just another of the behavioral constraints that are >> required to quack like a sequence. It is not just nominal typing because of the subclass hook. > I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this). Yes. I wish Alex joined this conversation. Cheers, Luciano On Wed, Apr 21, 2021 at 8:28 PM Luciano Ramalho <luciano(a)ramalho.org> wrote: > > On Wed, Apr 21, 2021 at 4:44 PM Christopher Barker <pythonchb(a)gmail.com> wrote: > >> You say this like it's a bad thing, but how is this avoidable, even in > >> principle? Structural typing lets you check whether Foo is duck-shaped > >> -- has appropriate attribute names, etc. But quacking like a duck is > >> harder: you also have to implement the Sequence behavioral contract, > >> and realistically the only way to know that is if the author of Foo > >> tells you. > > > > > > But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck. > > I agree with Christopher here. For duck typing, all we care is that > the method(s) we need are implemented in a sensible way. > > > And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever). > > Also agree, and very often the function cares only about one single method. > > > And a lot of the standard library works exactly this way. > > Totally. I found it astonishing that PEP 484 did not allow to > correctly annotate LOTS of functions in the standard library, > including built-ins people use every day like `sorted()`. Fortunately, > that was corrected with PEP 544. > > > But there is the "culture" of Python -- and it has been very much shifting toward more typing > > I disagree because Python is too widespread these days to talk about > "the culture". We'll just need to learn to coexist as groups that use > the language differently. > > -- A recent publication (sorry can't find it now -- my google fu is > failing me) examined code on PyPi and found a lot of type hints -- > many of which were apparently not being used with a static type > checker. So why were they there? > > Maybe you mean this one: > > "Python 3 types in the wild: a tale of two type systems" > Metadata: > https://dl.acm.org/doi/10.1145/3426422.3426981 > PDF: > http://hirzels.com/martin/papers/dls20-python-types.pdf > > > And I've seen a lot more isinstance(Some_ABC) code lately as well. > > I believe a lot of that is people coming from statically typed > languages and not feeling comfortable with duck typing, while for some > other reason not adopting type hints. > > > From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). > > It is considered best practice in some places. I hope we as a > community never adopt it as the canonical way. > > > Maybe it is -- but if the community is moving that way, we should be honest about it. > > Clearly parts of the community are moving that way. The paper I linked > shows the use of type hints is still very limited in their sample. > > >> I'm not even sure that this *is* nominal typing. You could just as > >> well argue that "the operation `isinstance(..., Sequence)` returns > >> `True`" is just another of the behavioral constraints that are > >> required to quack like a sequence. > > It is not just nominal typing because of the subclass hook. > > > I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this). > > Yes. I wish Alex joined this conversation. > > Cheers, > > Luciano > > -- > Luciano Ramalho > | Author of Fluent Python (O'Reilly, 2015) > | http://shop.oreilly.com/product/0636920032519.do > | Technical Principal at ThoughtWorks > | Twitter: @ramalhoorg -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

1 0

Re: Keeping Python a Duck Typed Language
by David Antonini April 21, 2021

April 21, 2021

I think what Christopher says is fair. My question (with little understanding of the underlying infrastructure) is can we define a 'type' that communicates that? Something like 'object that implements .read', or '.upper and .__iter__ for a spring-like object's? toonarmycaptain -------- Original message -------- From: python-dev-request(a)python.org Date: 4/21/21 2:44 PM (GMT-06:00) To: python-dev(a)python.org Subject: Python-Dev Digest, Vol 213, Issue 128 Send Python-Dev mailing list submissions to python-dev(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman3/lists/python-dev.python.org/ or, via email, send a message with subject or body 'help' to python-dev-request(a)python.org You can reach the person managing the list at python-dev-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..." Today's Topics: 1. Re: PEP 654: Exception Groups and except* [REPOST] (srkunze(a)mail.de) 2. Re: PEP 654: Exception Groups and except* [REPOST] (Yury Selivanov) 3. Re: Keeping Python a Duck Typed Language. (Christopher Barker) ---------------------------------------------------------------------- Message: 1 Date: Wed, 21 Apr 2021 20:48:29 +0200 From: "srkunze(a)mail.de" <srkunze(a)mail.de> Subject: [Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST] To: Irit Katriel <iritkatriel(a)googlemail.com> Cc: Irit Katriel <iritkatriel(a)yahoo.com>, python-dev <python-dev(a)python.org>, Yury Selivanov <yselivanov(a)gmail.com> Message-ID: <04512110-64B5-4AB3-AE54-AE6889C77C30(a)mail.de> Content-Type: text/plain; charset=utf-8 Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn. Or am I missing something here? Couldn’t we achieve our goal without these two new classes? Splitting, wrapping and exception handling semantics are perfectly fine and serve their purposes. So, of course this still needs to be implemented. > On 20. Apr 2021, at 23:26, Irit Katriel <iritkatriel(a)googlemail.com> wrote: > > I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*). > > >> On 20 Apr 2021, at 22:12, srkunze(a)mail.de wrote: >> >> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use. >> >> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case. >> >> A usual datastructure for a tree is to store all child nodes at the parent node. >> >> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP. >> >> Illustrative Example: >>>>> bundle_exc.__group__ >> [IOError(123), RuntimerError(‘issue somewhere’)] >> >> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute. >> >> The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that). >> >> Sven >> >>> >>>> On 20. Apr 2021, at 22:05, Irit Katriel <iritkatriel(a)yahoo.com> wrote: >>> >>> Hi Sven, >>> >>> I don’t follow. What would the value of __group__ be and how would it work? >>> >>> Irit >>> >>>>> On 20 Apr 2021, at 20:44, srkunze(a)mail.de wrote: >>>> >>>> Hi Irit, >>>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP: >>>> Why not extending BaseException by __group__ among __cause__ and __context__? >>>> Would this reduce some of the added complexity and thus increase broader acceptance? >>>> Cheers, >>>> Sven >> ------------------------------ Message: 2 Date: Wed, 21 Apr 2021 11:58:23 -0700 From: Yury Selivanov <yselivanov.ml(a)gmail.com> Subject: [Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST] To: "srkunze(a)mail.de" <srkunze(a)mail.de> Cc: Irit Katriel <iritkatriel(a)googlemail.com>, Irit Katriel <iritkatriel(a)yahoo.com>, python-dev <python-dev(a)python.org>, Yury Selivanov <yselivanov(a)gmail.com> Message-ID: <CA+St6D0V8-bOAQUgGVsQ1kT_7NkRd4Q96rUnk7h8xHRwNSrwCw(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Wed, Apr 21, 2021 at 11:50 AM srkunze(a)mail.de <srkunze(a)mail.de> wrote: > > Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn. > > Or am I missing something here? Couldn’t we achieve our goal without these two new classes? No, we can't. What you are proposing would make it very hard for users to understand at a glance if what you have in an innocently looking `except Exception` is correct or not. In my async/await code I'd have to always check the `__group__` attribute to make sure it's not an exception group in disguise. So while you're "simplifying" the proposal by removing a couple of types, you're complicating it in all other places. Besides, I don't think that adding the ExceptionGroup type is a controversial idea that needs any simplification. Yury ------------------------------ Message: 3 Date: Wed, 21 Apr 2021 12:36:34 -0700 From: Christopher Barker <pythonchb(a)gmail.com> Subject: [Python-Dev] Re: Keeping Python a Duck Typed Language. To: Nathaniel Smith <njs(a)pobox.com> Cc: Python Dev <python-dev(a)python.org> Message-ID: <CALn7ch9VvMx1b=AB46On=x5xvbaP1usGAbAgfe12DsQUKYBiUA(a)mail.gmail.com> Content-Type: multipart/alternative; boundary="00000000000023a40905c080af6d" Thanks Mark for posting this. I know some of us are uneasy about the pace of the typing train .... On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith <njs(a)pobox.com> wrote: > > If you guarded your code with `isinstance(foo, Sequence)` then I could > > not use it with my `Foo` even if my `Foo` quacked like a sequence. I was > > forced to use nominal typing; inheriting from Sequence, or explicitly > > registering as a Sequence. > > You say this like it's a bad thing, but how is this avoidable, even in > principle? Structural typing lets you check whether Foo is duck-shaped > -- has appropriate attribute names, etc. But quacking like a duck is > harder: you also have to implement the Sequence behavioral contract, > and realistically the only way to know that is if the author of Foo > tells you. > But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck. So yes, isinstance(obj, Sequence) is really the only way to know that obj is a Sequence in every important way -- but if you only need it to do one or two things like a Sequence, then you don't care. And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever). And a lot of the standard library works exactly this way. Two examples (chosen arbitrarily, I just happen to have thought about how they work): json.load() simply calls ``fp.read()``, and passes the result on down to json.loads(). That's it -- no checking of anything. If fp does not have a read() method, you get an AttributeError. If fp has a read() method, but it returns something other than a string, then you get some other Exception. And if it returns a string, but that string isn't valid JSON, you get yet another kind of error. In short, json.load(fp, ...) requires fp to have a read() method that returns a valid JSON string. But it doesn't check, nor does it need to, if it's getting an actual io.TextIOBase object. Is that the right one? I'm not totally sure, which I kind of think makes my point -- I've been using "file-like" objects for years (decades) without worrying about it. Example 2: The str.translate method takes: "a mapping of Unicode ordinals to Unicode ordinals, strings, or None" Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs go on to say: The table must implement lookup/indexing via __getitem__, for instance a dictionary or list. Ah -- so we don't need a Mapping -- we need anything indexable by an integer that contains "ordinals, strings, or None". What the heck ABC could we use for that? The ABCs do have an already complex hierarchy of containers, but there is no "Indexable", (quacks) and certainly no "indexable and returns these particular things. (quacks a certain way). (maybe there's something in the typing module that would work for static typing -- I have no idea). I'm pretty sure this particular API was designed to accommodate the old py2 str.translate, which took a length-256 sequence, while also accommodating full Unicode, which would have required a 2^32 length sequence to do the same thing :-) But again -- this is duck typing, built into the stdlib, and it works just fine. Granted, until PEP 563 (kind of) , there has been nothing that weakens or disallows such duck typing -- those of us that want to write fully duck-typed code can continue to do so. But there is the "culture" of Python -- and it has been very much shifting toward more typing -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there? And I've seen a lot more isinstance(Some_ABC) code lately as well. >From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). Maybe it is -- but if the community is moving that way, we should be honest about it. > I'm not even sure that this *is* nominal typing. You could just as > well argue that "the operation `isinstance(..., Sequence)` returns > `True`" is just another of the behavioral constraints that are > required to quack like a sequence. > I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this). The big distinction is whether we want to know if the object is a duck, or if we only need it to do one or two things like a duck. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython -------------- next part -------------- A message part incompatible with plain text digests has been removed ... Name: not available Type: text/html Size: 6277 bytes Desc: not available ------------------------------ Subject: Digest Footer _______________________________________________ Python-Dev mailing list -- python-dev(a)python.org To unsubscribe send an email to python-dev-leave(a)python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ ------------------------------ End of Python-Dev Digest, Vol 213, Issue 128 ********************************************

1 0

Should Python typing leverage PEP 263 as a pre-processor step?
by Luciano Ramalho April 20, 2021

April 20, 2021

How about leveraging the `# coding=<encoding name>` hook that exists since 2001 to enable the alternative syntax some are advocating for type hints? PEP 263—Defining Python Source Code Encodings https://www.python.org/dev/peps/pep-0263/ I've seen experiments in the wild using that to support syntax extensions to Python. Just for fun, years ago I wrote a POC for Sucuri—a Python dialect with the keywords in Portuguese. The pre-processor simply replaced the Portuguese keywords with the English ones. Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

5 6

PEP 563 in light of PEP 649
by Łukasz Langa April 20, 2021

April 20, 2021

Hi all, I got pinged to voice my opinion on PEP 649 as the instigator of PEP 563. I'm sorry, this is long, and a separate thread, because it deals with three things: - Goals set for PEP 563 and how it did in practice; - PEP 649 and how it addresses those same goals; - can we cleanly adopt PEP 649? First off, it looks like this isn't worded clearly enough in the PEP itself so let me summarize what the goals of PEP 563 were: Goal 1. To get rid of the forward reference problem, e.g. when a type is declared lower in the file than its use. A cute special case of this is when a class has a method that accepts or returns objects of its own type. Goal 2. To somewhat decouple the syntax of type annotations from the runtime requirements, allowing for better expressibility. Goal 3. To make annotations affect runtime characteristics of typed modules less, namely import time and memory usage. Now, did PEP 563 succeed in its goals? Well, partially at best. Let's see. In terms of Goal 1, it turned out that `typing.get_type_hints()` has limits that make its use in general costly at runtime, and more importantly insufficient to resolve all types. The most common example deals with non-global context in which types are generated (e.g. inner classes, classes within functions, etc.). But one of the crown examples of forward references: classes with methods accepting or returning objects of their own type, also isn't properly handled by `typing.get_type_hints()` if a class generator is used. There's some trickery we can do to connect the dots but in general it's not great. As for Goal 2, it became painfully obvious that a number of types used for static typing purposes live outside of the type annotation context. So while PEP 563 tried to enable a backdoor for more usable static typing syntax, it ultimately couldn't. This is where PEP 585 and later PEP 604 came in, filling the gap by doing the sad but necessary work of enabling this extended typing syntax in proper runtime Python context. This is what should have been done all along and it makes PEP 563 in this context irrelevant as of Python 3.9 (for PEP 585) and 3.10 (for PEP 604). However, to the extent of types used within annotations, the PEP 563 future-import allows using the new cute typing syntax already for Python 3.7+ compatible code. So library authors can already adopt the lowercase type syntax of PEP 585 and the handy pipe syntax for unions of PEP 604. And even for non-type annotation uses that can be successfully barred by a `if TYPE_CHECKING` block, like type aliases, type variables, and such. Of course that has no chance of working with `typing.get_type_hints()`. Now, Goal 3 is a different matter. As Inada Naoki demonstrated somewhere in the preceding discussion here, PEP 563 made fully type-annotationed codebase import pretty much as fast as non-annotated code, and through the joys of string interning, use relatively little extra memory. At the time PEP 563, a popular concern around static typing in Python was that it slows down runtime while it's only useful for static analysis. While we (the typing crowd) were always sure the "only useful as a better linter" is dismissive, the performance argument had to go. So where does this leave us today? Runtime use of types was somewhat overly optimistically treated as solvable with `typing.get_type_hints()`. Now Pydantic and other similar tools show that this isn't sadly the case. Without the future-import, they could ignore the problem until Python 3.10 but no longer. I was somewhat surprised this was the case because forward references as strings could always be used. So I guess the answer there was to just not use them if you want your runtime tool to work. Fair enough. PEP 649 addresses this runtime usage of type annotations head on in a way that eluded me when I first set out to solve this problem. Back then, Larry and Mark Shannon did voice their opinion that through some clever frame object storage, "implicit lambdas", we can address the issue of forward references. This seemed unattractive to me at the time because it didn't deal with our Goal 2 and our understanding was that it actually makes Goal 3 worse by holding on to all frames in memory where type annotations appear, and by creating massive duplication of equivalent annotations in memory due to lack of a mechanism similar to string interning. Now those issues are somewhat solved in the final PEP 649 and this makes for an interesting compromise for us to make. I say "compromise" because as Inada Naoki measured, there's still a non-zero performance cost of PEP 649 versus PEP 563: - code size: +63% - memory: +62% - import time: +60% Will this hurt some current users of typing? Yes, I can name you multiple past employers of mine where this will be the case. Is it worth it for Pydantic? I tend to think that yes, it is, since it is a significant community, and the operations on type annotations it performs are in the sensible set for which `typing.get_type_hints()` was proposed. However, there are some big open questions about how to adopt PEP 649. Question 1. What should happen to code that already adopted `from __future__ import annotations`? Future imports were never thought of as feature toggles so if PEP 563 isn't going to become the default, it should get removed. However, removing it needs a deprecation period, otherwise code that already adopted the future-import will fail to execute (even if you leave a dummy future-import there -- forward references, remember?). So deprecation, and with that, a rather tricky situation where PEP 649 will have to support files with the future-import. Now PEP 649 says that an object cannot both have __annotations__ and __co_annotations__ set at the same time though, so I guess files with the future-import would necessarily be treated as some deprecated code that might or might not be translatable to PEP 649 co-annotations. Question 2. If PEP 649 deprecates PEP 563, will the use of the future-import for early adoption of PEP 585 and PEP 604 syntax become disallowed? Ultimately this is my biggest worry -- that there doesn't seem to be a clear adoption path for this change. If we just take it wholesale and declare PEP 563 a failure, that bars library authors from using PEP 585/604 syntax until Python 3.10 becomes their lowest supported version. That's October 2025 at the earliest if we're looking at 3.9 lifespan. That would be a significant wrench thrown in typing adoption as PEP 585 and 604 provide significant usability advantages, to the point where often modules with non-trivial annotations don't even have to import the typing module at all. Question 3. Additionally, it's unclear to me whether PEP 649 allows for any relaxed syntax in type annotations that might not be immediately valid in a given version of Python. Say, hypothetically (please don't bikeshed this!), we adopt PEP 649 in Python 3.10 and in Python 3.11 we come up with a shorthand for optional types using a question mark. Can I put the question mark there in Python 3.10 at all? Second made up example: let's say in Python 3.12 we allow passing a function as a type (meaning "a callable just LIKE this one"). Can I put this callable there now in Python 3.10? Now, should we postpone with until Python 3.11? We should not, at least not in terms of figuring out a clear migration path from here to there, ideally such that existing end users of the PEP 563 future-import can just live their lives as if nothing ever happened. Given that the goal of the future-import was largely to say "I don't care about those annotations at runtime", maybe PEP 649 could somehow adopt the existing future-import? `typing.get_type_hints()` would do what it does today anyway. The only users affected would be those directly using the strings in PEP 563 annotations, and it's unclear whether this is a real problem. There are a hundred pages of results on GitHub [1]_ that include `__annotations__` so this would have to be sought through. If we don't make it in time for including this in Python 3.10, then so be it. But let's not wait until April 2022 ;-) i-regret-nothing'ly yours, Łukasz _[1] https://github.com/search?l=Python&q=__annotations__&type=Code

10 17