Mailman 3 March 2020 - Python-Dev

Re: Proliferation of tstate arguments.
by Anders Munch March 22, 2020

March 22, 2020

Chris Angelico [mailto:rosuav@gmail.com]: > And by that logic, globals are even more capable. I don't understand your > point. Isn't the purpose of the tstate parameters to avoid the problem of > being unable to have multiple tstates within the same OS thread? I think I've > missed something here. The point is that because threads can't preempt themselves, this: /*1*/ { Py_something(other_tstate); } and this: /*2*/ { PyInterpreterState* old_tstate = tstate; tstate = … [View More]

7 8

Changing layout of f_localsplus in frame objects
by Skip Montanaro March 21, 2020

March 21, 2020

(Apologies if you're seeing this twice. I first posted to the discourse instance.) I first worked on a register-based virtual machine in the 1.5.2 timeframe. That was before PEP 227 (closures) landed. Prior to that, local variables and the stack were contiguous in the f_localsplus array. Based on a comment at the time from Tim Peters (which I recall but can no longer find), the number of required registers won’t be any greater than the maximum extent of the stack. (It makes sense, and worked … [View More]

2 3

Proliferation of tstate arguments.
by Mark Shannon March 21, 2020

March 21, 2020

Hi, There seems to be a proliferation of `PyThreadState *tstate` arguments being added to API and internal functions. These changes are listed under https://bugs.python.org/issue38644. I think that these changes are misguided. The desired results can be achieved more reliably and more simply in other ways. The changes add bulk to the C-API and may hurt performance. These changes are also causing a lot of churn and merge conflicts (for me at least). So can we please stop making these … [View More]

8 16

(no subject)
by Dennis Sweeney March 20, 2020

March 20, 2020

Thanks for the review! > In short, I propose: > def cutprefix(self: str, prefix: str, /) -> str: > if self.startswith(prefix) and prefix: > return self[len(prefix):] > else: > return self > > I call startswith() before testing if pre is non-empty to inherit of > startswith() input type validation. For example, "a".startswith(b'x') > raises a TypeError. This still erroneously accepts tuples and and would return return str … [View More]

1 0

Requesting review on PEP 615 C Extension Reference Implementation
by Paul Ganssle March 20, 2020

March 20, 2020

Hi all, The past few weeks I've been working on adding a C extension to the reference implementation for PEP 615 (Support for the IANA Time Zone Database in the Standard Library - PEP link <https://www.python.org/dev/peps/pep-0615/>), but I've had some trouble inducing anyone to review the code <https://twitter.com/pganssle/status/1240019701356539906?s=20>. There's about 2200 lines of C code there that's passing all the tests for the pure Python implementation (which has 100% code … [View More]

1 0

[bpo-22699] Cross-compiling and fixing sysconfig
by Steve Dower March 20, 2020

March 20, 2020

So over on https://bugs.python.org/issue22699 I've been talking to myself as I figure out all the ways that cross-compiling (on Ubuntu, targeting another Linux distro via an SDK) fails. In short, it's either because sysconfig can't provide details about any CPython install other than the currently running one, or it's because our setup.py (for building the CPython extension modules) uses sysconfig. Either way, I'm not about to propose a rewrite to fix either of them without finding those … [View More]

3 3

Summary of Python tracker Issues
by Python tracker March 20, 2020

March 20, 2020

ACTIVITY SUMMARY (2020-03-13 - 2020-03-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7358 ( +1) closed 44420 (+70) total 51778 (+71) Open issues with patches: 2883 Issues opened (44) ================== #39576: Surprising MemoryError in `decimal` with MAX_PREC https://bugs.python.org/issue39576 reopened by pablogsal #39958: Deadlock in … [View More]

1 0

Looking for a sponsor and feedback on PEP 616: string methods for removing prefixes and suffixes
by Dennis Sweeney March 20, 2020

March 20, 2020

Hello all! I'm a relatively new contributor looking for a Core Dev sponsor for the following PEP: https://github.com/python/peps/pull/1332 Related: - Python-Ideas Thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS… - Bug Tracker Issue: https://bugs.python.org/issue39939 - Github PR for implementation: https://github.com/python/cpython/pull/18939 Abstract ======== This is a proposal to add two new methods, ``cutprefix`` and ``cutsuffix``, to the APIs of … [View More]

2 1

Re: Proliferation of tstate arguments.
by Anders Munch March 18, 2020

March 18, 2020

Antoine Pitrou [mailto:solipsis@pitrou.net]: > This example is mixing up the notion of interpreter state and thread state. Sorry about that, I was making a more general point and not paying so much attention to the specific names. The general point is this: A thread-local variable can work as an implied parameter without creating synchronisation problems. You do not have to worry about painting yourself into an architectural corner by using thread-local variables, because anything you can … [View More]

1 0

PEP 585: Type Hinting Generics In Standard Collections
by Łukasz Langa March 16, 2020

March 16, 2020

Read in the browser: https://www.python.org/dev/peps/pep-0585/ <https://www.python.org/dev/peps/pep-0585/> Read the source: https://raw.githubusercontent.com/python/peps/master/pep-0585.rst <https://raw.githubusercontent.com/python/peps/master/pep-0585.rst> The following PEP has been discussed on typing-sig already and a prototype implementation exists for it. I'm extending it now for wider feedback on python-dev, with the intent to present the final version for the Steering … [View More]Council's consideration by mid-March. PEP: 585 Title: Type Hinting Generics In Standard Collections Author: Łukasz Langa <lukasz(a)python.org> Discussions-To: Typing-Sig <typing-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Mar-2019 Python-Version: 3.9 Abstract Static typing as defined by PEPs 484, 526, 544, 560, and 563 was built incrementally on top of the existing Python runtime and constrained by existing syntax and runtime behavior. This led to the existence of a duplicated collection hierarchy in the typing module due to generics (for example typing.List and the built-in list). This PEP proposes to enable support for the generics syntax in all standard collections currently available in the typing module. Rationale and Goals This change removes the necessity for a parallel type hierarchy in the typing module, making it easier for users to annotate their programs and easier for teachers to teach Python. Terminology Generic (n.) -- a type that can be parametrized, typically a container. Also known as a parametric type or a generic type. For example: dict. Parametrized generic -- a specific instance of a generic with the expected types for container elements provided. Also known as a parametrized type. For example: dict[str, int]. Backwards compatibility Tooling, including type checkers and linters, will have to be adapted to recognize standard collections as generics. On the source level, the newly described functionality requires Python 3.9. For use cases restricted to type annotations, Python files with the "annotations" future-import (available since Python 3.7) can parametrize standard collections, including builtins. To reiterate, that depends on the external tools understanding that this is valid. Implementation Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parametrize standard collections directly. Example: from __future__ import annotations def find(haystack: dict[str, list[int]]) -> int: ... Usefulness of this syntax before PEP 585 <https://www.python.org/dev/peps/pep-0585> is limited as external tooling like Mypy does not recognize standard collections as generic. Moreover, certain features of typing like type aliases or casting require putting types outside of annotations, in runtime context. While these are relatively less common than type annotations, it's important to allow using the same type syntax in all contexts. This is why starting with Python 3.9, the following collections become generic using __class_getitem__() to parametrize contained types: tuple # typing.Tuple list # typing.List dict # typing.Dict set # typing.Set frozenset # typing.FrozenSet type # typing.Type collections.deque collections.defaultdict collections.OrderedDict collections.Counter collections.ChainMap collections.abc.Awaitable collections.abc.Coroutine collections.abc.AsyncIterable collections.abc.AsyncIterator collections.abc.AsyncGenerator collections.abc.Iterable collections.abc.Iterator collections.abc.Generator collections.abc.Reversible collections.abc.Container collections.abc.Collection collections.abc.Callable collections.abc.Set # typing.AbstractSet collections.abc.MutableSet collections.abc.Mapping collections.abc.MutableMapping collections.abc.Sequence collections.abc.MutableSequence collections.abc.ByteString collections.abc.MappingView collections.abc.KeysView collections.abc.ItemsView collections.abc.ValuesView contextlib.AbstractContextManager # typing.ContextManager contextlib.AbstractAsyncContextManager # typing.AsyncContextManager re.Pattern # typing.Pattern, typing.re.Pattern re.Match # typing.Match, typing.re.Match Importing those from typing is deprecated. Due to PEP 563 <https://www.python.org/dev/peps/pep-0563> and the intention to minimize the runtime impact of typing, this deprecation will not generate DeprecationWarnings. Instead, type checkers may warn about such deprecated usage when the target version of the checked program is signalled to be Python 3.9 or newer. It's recommended to allow for those warnings to be silenced on a project-wide basis. The deprecated functionality will be removed from the typing module in the first Python version released 5 years after the release of Python 3.9.0. Parameters to generics are available at runtime Preserving the generic type at runtime enables introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild. Just like with the typing module today, the parametrized generic types listed in the previous section all preserve their type parameters at runtime: >>> list[str] list[str] >>> tuple[int, ...] tuple[int, ...] >>> ChainMap[str, list[str]] collections.ChainMap[str, list[str]] This is implemented using a thin proxy type that forwards all method calls and attribute accesses to the bare origin type with the following exceptions: the __repr__ shows the parametrized type; the __origin__ attribute points at the non-parametrized generic class; the __args__ attribute is a tuple (possibly of length 1) of generic types passed to the original __class_getitem__; the __parameters__ attribute is a lazily computed tuple (possibly empty) of unique type variables found in __args__; the __getitem__ raises an exception to disallow mistakes like dict[str][str]. However it allows e.g. dict[str, T][int] and in that case returns dict[str, int]. This design means that it is possible to create instances of parametrized collections, like: >>> l = list[str]() [] >>> list is list[str] False >>> list == list[str] False >>> list[str] == list[str] True >>> list[str] == list[int] False >>> isinstance([1, 2, 3], list[str]) TypeError: isinstance() arg 2 cannot be a parametrized generic >>> issubclass(list, list[str]) TypeError: issubclass() arg 2 cannot be a parametrized generic >>> isinstance(list[str], types.GenericAlias) True Objects created with bare types and parametrized types are exactly the same. The generic parameters are not preserved in instances created with parametrized types, in other words generic types erase type parameters during object creation. One important consequence of this is that the interpreter does not attempt to type check operations on the collection created with a parametrized type. This provides symmetry between: l: list[str] = [] and: l = list[str]() For accessing the proxy type from Python code, it will be exported from the types module as GenericAlias. Pickling or (shallow- or deep-) copying a GenericAlias instance will preserve the type, origin, attributes and parameters. Forward compatibility Future standard collections must implement the same behavior. Reference implementation A proof-of-concept or prototype implementation <https://bugs.python.org/issue39481> exists. Rejected alternatives Do nothing Keeping the status quo forces Python programmers to perform book-keeping of imports from the typing module for standard collections, making all but the simplest annotations cumbersome to maintain. The existence of parallel types is confusing to newcomers (why is there both list and List?). The above problems also don't exist in user-built generic classes which share runtime functionality and the ability to use them as generic type annotations. Making standard collections harder to use in type hinting from user classes hindered typing adoption and usability. Generics erasure It would be easier to implement __class_getitem__ on the listed standard collections in a way that doesn't preserve the generic type, in other words: >>> list[str] <class 'list'> >>> tuple[int, ...] <class 'tuple'> >>> collections.ChainMap[str, list[str]] <class 'collections.ChainMap'> This is problematic as it breaks backwards compatibility: current equivalents of those types in the typing module do preserve the generic type: >>> from typing import List, Tuple, ChainMap >>> List[str] typing.List[str] >>> Tuple[int, ...] typing.Tuple[int, ...] >>> ChainMap[str, List[str]] typing.ChainMap[str, typing.List[str]] As mentioned in the "Implementation" section, preserving the generic type at runtime enables runtime introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild. Additionally, implementing subscripts as identity functions would make Python less friendly to beginners. Say, if a user is mistakenly passing a list type instead of a list object to a function, and that function is indexing the received object, the code would no longer raise an error. Today: >>> l = list >>> l[-1] TypeError: 'type' object is not subscriptable With __class_getitem__ as an identity function: >>> l = list >>> l[-1] list The indexing being successful here would likely end up raising an exception at a distance, confusing the user. Disallowing instantiation of parametrized types Given that the proxy type which preserves __origin__ and __args__ is mostly useful for runtime introspection purposes, we might have disallowed instantiation of parametrized types. In fact, forbidding instantiation of parametrized types is what the typing module does today for types which parallel builtin collections (instantiation of other parametrized types is allowed). The original reason for this decision was to discourage spurious parametrization which made object creation up to two orders of magnitude slower compared to the special syntax available for those builtin collections. This rationale is not strong enough to allow the exceptional treatment of builtins. All other parametrized types can be instantiated, including parallels of collections in the standard library. Moreover, Python allows for instantiation of lists using list() and some builtin collections don't provide special syntax for instantiation. Making isinstance(obj, list[str]) perform a check ignoring generics An earlier version of this PEP suggested treating parametrized generics like list[str] as equivalent to their non-parametrized variants like list for purposes of isinstance() and issubclass(). This would be symmetrical to how list[str]() creates a regular list. This design was rejected because isinstance() and issubclass() checks with parametrized generics would read like element-by-element runtime type checks. The result of those checks would be surprising, for example: >>> isinstance([1, 2, 3], list[str]) True Note the object doesn't match the provided generic type but isinstance() still returns True because it only checks whether the object is a list. If a library is faced with a parametrized generic and would like to perform an isinstance() check using the base type, that type can be retrieved using the __origin__ attribute on the parametrized generic. Making isinstance(obj, list[str]) perform a runtime type check This functionality requires iterating over the collection which is a destructive operation in some of them. This functionality would have been useful, however implementing the type checker within Python that would deal with complex types, nested type checking, type variables, string forward references, and so on is out of scope for this PEP. Note on the initial draft An early version of this PEP discussed matters beyond generics in standard collections. Those unrelated topics were removed for clarity. Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

7 14