Mailman 3 June 2022 - Python-ideas

PEP 671 (late-bound arg defaults), next round of discussion!
by Chris Angelico 29 Apr '23

29 Apr '23

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere. *PEP 671: Syntax for late-bound function argument defaults* Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it? 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? (It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.) 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? 4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.) 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best! The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go. ChrisA

41 355

zip(x, y, z, strict=True)
by Ram Rachum 01 Dec '22

01 Dec '22

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Confusing naming of Optional type should be changed
by nverich07＠gmail.com 12 Jul '22

12 Jul '22

(I apologize in advance if I've posted anything incorrectly before, I believe I might have sent this in python-dev instead but not sure as it's not appearing in the posts for my account). I am aware this is clarified in the Python documentation for the typing module but I and others have thought the naming of Optional is still quite confusing by itself. "Note that this is not the same concept as an optional argument, which is one that has a default. An optional argument with a default does not require the Optional qualifier on its type annotation just because it is optional." - typing.Optional docs Google defines optional as this, "available to be chosen but not obligatory." Pretend we have a function like this: def test_func(param: Optional[str]) -> None: ... The argument param is typed as Optional[str] meaning Union[str, None] or str | None. Optional here if we follow the definition above, basically means it can be str but not required or obligated to be that type. See the problem with the naming? This is a function where param can be None or str, not just it can be str but not obligated. Some interpretations may think optional means left to one's choice as well. The docs even have to clarify the use of Optional with typing because of this confusion. I believe this has been proposed before (not sure) but something like Nullable or Noneable (not sure about naming) would make much more sense in the context of typing. def test_func(param: Noneable[str]) -> None: ... It also would work and still make sense if there is a default value for the argument: def test_func(param: Noneable[str] = None) -> None: ... def test_func(param: Noneable[str] = "hello world") -> None: ...

5 9

dataclass field argument to allow converting value on init
by Dexter Hill 02 Jul '22

02 Jul '22

The idea is to have a `default_factory` like argument (either in the `field` function, or a new function entirely) that takes a function as an argument, and that function, with the value provided by `__init__`, is called and the return value is used as the value for the respective field. For example: ```py @dataclass class Foo: x: str = field(init_fn=chr) f = Foo(65) f.x # "A" ``` The `chr` function is called, given the value `65` and `x` is set to its return value of `"A"`. I understand that there is both `__init__` and `__post_init__` which can be used for this purpose, but sometimes it isn't ideal to override them. If you overrided `__init__`, and were using `__post_init__`, you would need to manually call it, and in my case, `__post_init__` is implemented on a base class, which all other classes inherit, and so overloading it would require re-implementing the logic from it (and that's ignoring the fact that you also need to type the field with `InitVar` to even have it passed to `__post_init__` in the first place). I've created a proof of concept, shown below: ```py def initfn(fn, default=None): class Inner: def __set_name__(_, owner_cls, owner_name): old_setattr = getattr(owner_cls, "__setattr__") def __setattr__(self, attr_name, value): if attr_name == owner_name: # Bypass `__setattr__` self.__dict__[attr_name] = fac(value) else: old_setattr(self, attr_name, value) setattr(owner_cls, "__setattr__", __setattr__) def fac(value): if isinstance(value, Inner): return default return fn(value) return field(default=Inner()) ``` It makes use of the fact that providing `default` as an argument to `field` means it checks the value for a `__set_name__` function, and calls it with the class and field name as arguments. Overriding `__setattr__` is just used to catch when a value is being assigned to a field, and if that field's name matches the name given to `__set_name__`, it calls the function on the value, at sets the field to that instead. It can be used like so: ```py @dataclass class Foo: x: str = initfn(fn=chr, default="Z") f = Foo(65) f2 = Foo() f.x # "A" f2.x # "Z" ``` It adds a little overhead, especially with having to override `__setattr__` however, I believe it would have very little overhead if directly implemented in the dataclass library. Even in the case of being able to override one of the init functions, I still think it would be nice to have as a quality of life feature as I feel calling a function is too simple to want to override the functions, if that makes sense. Thanks. Dexter

5 22

Typescript-Like Notational Shortcuts
by kevinlu1248＠gmail.com 01 Jul '22

01 Jul '22

On the theme of the addition of pip operators to indicate union, I was wondering what people think of using the following shortcuts for type hints. * type[] be a short-hand for List[type] * [type1, type2, type3] be a shorthand for Tuple[type1, type2, type3] * similar ideas for dictionaries and callables with (perhaps with javascript-like arrow notations?) I just think something like this would be easier to read. For example, a lot of functions that return tuples generally are immediately unpacked when it's used, so I think this is a more clear and concise way of stating that this is that it returns three elements: an element of type1, an element of type2 and an element of type3. In other words, the fact that it's a tuple does not matter much.

3 2

Pip & gpg story
by J. Pic 28 Jun '22

28 Jun '22

Hi Currently we can upload signed packages on pypi. Shouldn't pip have a keyring of thrusted projects or developers and enforce whitelisting of untrusted packages, either through a requirement flag or through an interactive question in CLI? I think this would help with user security if we want to keep pypi open for upload to all on the long term. Thanks for your feedback

3 2

Generalized deferred computation in Python
by David Mertz, Ph.D. 26 Jun '22

26 Jun '22

Here is a very rough draft of an idea I've floated often, but not with much specification. Take this as "ideas" with little firm commitment to details from me. PRs, or issues, or whatever, can go to https://github.com/DavidMertz/peps/blob/master/pep-9999.rst as well as mentioning them in this thread. PEP: 9999 Title: Generalized deferred computation Author: David Mertz <dmertz(a)gnosis.cx> Discussions-To: https://mail.python.org/archives/list/python-ideas@python.org/thread/ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 21-Jun-2022 Python-Version: 3.12 Post-History: Abstract ======== This PEP proposes introducing the soft keyword ``later`` to express the concept of deferred computation. When an expression is preceded by the keyword, the expression is not evaluated but rather creates a "thunk" or "deferred object." Reference to the deferred object later in program flow causes the expression to be executed at that point, and for both the value and type of the object to become the result of the evaluated expression. Motivation ========== "Lazy" or "deferred" evaluation is useful paradigm for expressing relationships among potentially expensive operations prior their actual computation. Many functional programming languages, such as Haskell, build laziness into the heart of their language. Within the Python ecosystem, the popular scientific library `dask-delayed <dask-delayed>`_ provides a framework for lazy evaluation that is very similar to that proposed in this PEP. .. _dask-delayed: https://docs.dask.org/en/stable/delayed.html Examples of Use =============== While the use of deferred computation is principally useful when computations are likely to be expensive, the simple examples shown do not necessarily use such expecially spendy computations. Most of these are directly inspired by examples used in the documentation of dask-delayed. In dask-delayed, ``Delayed`` objects are create by functions, and operations create a *directed acyclic graph* rather than perform actual computations. For example:: >>> import dask >>> @dask.delayed ... def later(x): ... return x ... >>> output = [] >>> data = [23, 45, 62] >>> for x in data: ... x = later(x) ... a = x * 3 ... b = 2**x ... c = a + b ... output.append(c) ... >>> total = sum(output) >>> total Delayed('add-8f4018dbf2d3c1d8e6349c3e0509d1a0') >>> total.compute() 4611721202807865734 >>> total.visualize() .. figure:: pep-9999-dag.png :align: center :width: 50% :class: invert-in-dark-mode Figure 1. Dask DAG created from simple operations. Under this PEP, the soft keyword ``later`` would work in a similar manner to this dask.delayed code. But rather than requiring calling ``.compute()`` on a ``Delayed`` object to arrive at the result of a computation, every reference to a binding would perform the "compute" *unless* it was itself a deferred expression. So the equivalent code under this PEP would be:: >>> output = [] >>> data = [23, 45, 62] >>> for later x in data: ... a = later (x * 3) ... b = later (2**x) ... c = later (a + b) ... output.append(later c) ... >>> total = later sum(output) >>> type(total) # type() does not un-thunk <class 'DeferredObject'> >>> if value_needed: ... print(total) # Actual computation occurs here 4611721202807865734 In the example, we assume that the built-in function `type()` is special in not counting as a reference to the binding for purpose of realizing a computation. Alternately, some new special function like `isdeferred()` might be used to check for ``Deferred`` objects. In general, however, every regular reference to a bound object will force a computation and re-binding on a ``Deferred``. This includes access to simple names, but also similarly to instance attributes, index positions in lists or tuples, or any other means by which an object may be referenced. Rejected Spellings ================== A number of alternate spellings for creating a ``Deferred`` object are possible. This PEP-author has little preference among them. The words ``defer`` or ``delay``, or their past participles ``deferred`` and ``delayed`` are commonly used in discussions of lazy evaluation. All of these would work equally well as the suggested soft keyword ``later``. The keyword ``lazy`` is not completely implausible, but does not seem to read as well. No punctuation is immediately obvious for this purpose, although surrounding expressions with backticks is somewhat suggestive of quoting in Lisp, and perhaps slightly reminiscent of the ancient use of backtick for shell commands in Python 1.x. E.g.:: might_use = `math.gcd(a, math.factorial(b))` Relationship to PEP-0671 ======================== The concept of "late-bound function argument defaults" is introduced in :pep:`671`. Under that proposal, a special syntactic marker would be permitted in function signatures with default arguments to allow the expressions indicated as defaults to be evaluated at call time rather than at runtime. In current Python, we might write a toy function such as:: def func(items=[], n=None): if n is None: n = len(items) items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 Using the :pep:`671` approach this could be simplified somewhat as:: def func(items=[], n=>len(items)): # late-bound defaults act as if bound here items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 Under the current PEP, evaluation of a ``Deferred`` object only occurs upon reference. That is, for the current toy function, the evaluation would not occur until the ``print(n)`` line.:: def func(items=[], n=later len(items)): items.append("Hello") print(n) func([1, 2, 3]) # prints: 4 To completely replicate the behavior of PEP-0671, an extra line at the start of the function body would be required:: def func(items=[], n=later len(items)): n = n # Evaluate the Deferred and re-bind the name n items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 References ========== https://github.com/DavidMertz/peps/blob/master/pep-9999.rst Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

18 55

Adopting better naming conventions (food for thought)
by abed...＠gmail.com 24 Jun '22

24 Jun '22

I was reading some of Kevlin Henney's views on naming conventions <https://kevlinhenney.medium.com/exceptional-naming-6e3c8f5bffac>, and I realized that, of the 65 classes that make up Python's built-in exceptions <https://docs.python.org/3/library/exceptions.html#exception-hierarchy>, only 8 (by my count) satisfy Henney's pretty basic concept of a good Exception name: *The name should represent whatever the problem is, and should do so directly and specifically. To add Exception to the end is either redundant — so remove it — or an indication of a poor name — so rename it.* Those are: SystemExit, KeyboardInterrupt, GeneratorExit, Exception, StopIteration, StopAsyncIteration, Warning, and BaseException (though Henney also discourages the use of "Base" in a base-class's name) I'm sure there are others throughout the standard library too. I always caution novice programmers to err on the side of pragmatism over dogmatic adherence to "sacred truths". To that end: 1) I realize changing built-in exception names would break backwards compatibility and probably isn't worth doing. This post is more intended as food for thought going forward and to bring more attention to Kevlin Henney's ideas. 2) I think many of the built-in exceptions are actually fine, particularly ones that represent a broad class of more specific exceptions like OSError. I don't think OSError has a specific enough name to be raised on its own (perhaps it should be abstract or otherwise not directly instantiable?), but catching error categories can be pretty helpful even if they don't lend themselves to explicit names. I would suggest a more ideal naming scheme would be something like the following: - BaseException -> Exception # The text for GeneratorExit <https://docs.python.org/3/library/exceptions.html#GeneratorExit> seems to indicate that in Python, an Error is considered a subclass of an Exception, so this naming seems more appropriate though it contradicts other views on the distinction between Exception and Error <https://stackoverflow.com/questions/5813614/what-is-difference-between-erro…> . - SystemExit - *KeyboardInterrupt -> UserInterrupt *#This is supposed to capture a user-initiated exit, whether it came from a keyboard or a magic wand is generally much less relevant - GeneratorExit - Exception -> Error - *StopIteration* - *StopAsyncIteration* - *ArithmeticError* - FloatingPointError -> Not used, get rid of it? - OverflowError -> Overflow - ZeroDivisionError -> ZeroDivision or Undefined - *AssertionError -> Contradiction or AssertionViolation* - *AttributeError* - *+ AttributeNotFound or AttributeUnavailable* - *+ AttributeUnassignable* - *BufferError* - *<specific errors>* - *EOFError -> EndOfFile* - *ImportError * - ModuleNotFoundError -> ModuleNotFound - *LookupError * - *IndexError -> IndexOutOfBounds* - *KeyError -> KeyNotFound* - *MemoryError -> OutOfMemory * - *NameError -> InvalidName (Not needed?) * - UnboundLocalError -> UnboundLocal - *OSError * - BlockingIOError -> BlockingIO - ChildProcessError -> ChildProcessFailed - ConnectionError - BrokenPipeError -> BrokenPipe - ConnectionAbortedError -> ConnectionAborted - ConnectionRefusedError -> ConnectionRefused - ConnectionResetError -> ConnectionReset - FileExistsError -> FileAlreadyExists - FileNotFoundError -> FileNotFound - InterruptedError -> Interrupted - IsADirectoryError -> NotAFile - NotADirectoryError -> NotADirectory - PermissionError -> PermissionDenied - ProcessLookupError -> ProcessNotFound - TimeoutError -> Timeout - *ReferenceError -> ReferentCollected? * - *RuntimeError * - NotImplementedError -> NotImplemented - RecursionError -> MaxRecursionDepthExceeded or RecursionOverflow - *SyntaxError * - IndentationError - TabError -> InconsistentIndentation - *SystemError -> SystemFailure* - *TypeError -> IncompatibleType * - *ValueError * - UnicodeError - UnicodeDecodeError - UnicodeEncodeError - UnicodeTranslateError - *Warning* # Should this be under Error? - DeprecationWarning -> Deprecated - PendingDeprecationWarning -> Obsolete - RuntimeWarning - SyntaxWarning - UserWarning - FutureWarning - ImportWarning - UnicodeWarning - BytesWarning - EncodingWarning - ResourceWarning

2 1

Ellipsis (...) to be roughly synonymous with *<varname> in destructuring but without capture.
by Steve Jorgensen 22 Jun '22

22 Jun '22

This is based on previous discussions of possible ways of matching all remaining items during destructuring but without iterating of remaining final items. This is not exactly a direct replacement for that idea though, and skipping iteration of final items might or might not be part of the goal. In this proposal, the ellipsis (...) can be used in the expression on the left side of the equals sign in destructuring anywhere that `*<varname>` can appear and has approximately the same meaning. The difference is that when the ellipsis is used, the matched items are not stored in variables. This can be useful when the matched data might be very large. ..., last_one = <expression> a, ..., z = <expression> first_one, ... = <expression> Additionally, when the ellipsis comes last and the data is being retrieved by iterating, stop retrieving items since that might be expensive and we know that we will not use them. Alternative A: Still iterate over items when the ellipsis comes last (for side effects) but introduce a new `final_elipsis` object that is used to stop iteration. The negation of `ellipsis` (e.g. `-...`) could return `final_ellipsis` in that case. Alternative B: Still iterate over items when the ellipsis comes last (for side effects) and don't provide any new means of skipping iteration over final items. The programmer can use islice to achieve that.

2 3

Re: Add a line_offsets() method to str
by Inada Naoki 21 Jun '22

21 Jun '22

On Tue, Jun 21, 2022 at 3:49 AM Marco Sulla <Marco.Sulla.Python(a)gmail.com> wrote: > > On Sun, 19 Jun 2022 at 03:06, Inada Naoki <songofacandy(a)gmail.com> wrote: > > FWIW, I had proposed str.iterlines() to fix incompatibility between > > IO.readlines() and str.splitlines(). > > It's a good idea IMHO. In your mind, str.iterlines() will find only > \n, \r and \r\n, as IO.readlines() do? I ask this because > str.splitlines() finds all line boundary chars. > I don't remember the old discussion, but I think so. Another idea is adding some option to `splitlines()` that uses only universal newlines. (e.g. str.splitlines(keepends=False, *, ascii=False)) Then we can add `str.iterlines` that have the same arguments to `str.splitlines`. -- Inada Naoki <songofacandy(a)gmail.com>

1 0

Python-ideas June 2022