Python-ideas

python-ideas@python.org

51 participants
5435 discussions

dataclass field argument to allow converting value on init
by Dexter Hill 25 Jun '22

25 Jun '22

The idea is to have a `default_factory` like argument (either in the `field` function, or a new function entirely) that takes a function as an argument, and that function, with the value provided by `__init__`, is called and the return value is used as the value for the respective field. For example: ```py @dataclass class Foo: x: str = field(init_fn=chr) f = Foo(65) f.x # "A" ``` The `chr` function is called, given the value `65` and `x` is set to its return value of `"A"`. I understand that there is both `__init__` and `__post_init__` which can be used for this purpose, but sometimes it isn't ideal to override them. If you overrided `__init__`, and were using `__post_init__`, you would need to manually call it, and in my case, `__post_init__` is implemented on a base class, which all other classes inherit, and so overloading it would require re-implementing the logic from it (and that's ignoring the fact that you also need to type the field with `InitVar` to even have it passed to `__post_init__` in the first place). I've created a proof of concept, shown below: ```py def initfn(fn, default=None): class Inner: def __set_name__(_, owner_cls, owner_name): old_setattr = getattr(owner_cls, "__setattr__") def __setattr__(self, attr_name, value): if attr_name == owner_name: # Bypass `__setattr__` self.__dict__[attr_name] = fac(value) else: old_setattr(self, attr_name, value) setattr(owner_cls, "__setattr__", __setattr__) def fac(value): if isinstance(value, Inner): return default return fn(value) return field(default=Inner()) ``` It makes use of the fact that providing `default` as an argument to `field` means it checks the value for a `__set_name__` function, and calls it with the class and field name as arguments. Overriding `__setattr__` is just used to catch when a value is being assigned to a field, and if that field's name matches the name given to `__set_name__`, it calls the function on the value, at sets the field to that instead. It can be used like so: ```py @dataclass class Foo: x: str = initfn(fn=chr, default="Z") f = Foo(65) f2 = Foo() f.x # "A" f2.x # "Z" ``` It adds a little overhead, especially with having to override `__setattr__` however, I believe it would have very little overhead if directly implemented in the dataclass library. Even in the case of being able to override one of the init functions, I still think it would be nice to have as a quality of life feature as I feel calling a function is too simple to want to override the functions, if that makes sense. Thanks. Dexter

5 12

Generalized deferred computation in Python
by David Mertz, Ph.D. 25 Jun '22

25 Jun '22

Here is a very rough draft of an idea I've floated often, but not with much specification. Take this as "ideas" with little firm commitment to details from me. PRs, or issues, or whatever, can go to https://github.com/DavidMertz/peps/blob/master/pep-9999.rst as well as mentioning them in this thread. PEP: 9999 Title: Generalized deferred computation Author: David Mertz <dmertz(a)gnosis.cx> Discussions-To: https://mail.python.org/archives/list/python-ideas@python.org/thread/ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 21-Jun-2022 Python-Version: 3.12 Post-History: Abstract ======== This PEP proposes introducing the soft keyword ``later`` to express the concept of deferred computation. When an expression is preceded by the keyword, the expression is not evaluated but rather creates a "thunk" or "deferred object." Reference to the deferred object later in program flow causes the expression to be executed at that point, and for both the value and type of the object to become the result of the evaluated expression. Motivation ========== "Lazy" or "deferred" evaluation is useful paradigm for expressing relationships among potentially expensive operations prior their actual computation. Many functional programming languages, such as Haskell, build laziness into the heart of their language. Within the Python ecosystem, the popular scientific library `dask-delayed <dask-delayed>`_ provides a framework for lazy evaluation that is very similar to that proposed in this PEP. .. _dask-delayed: https://docs.dask.org/en/stable/delayed.html Examples of Use =============== While the use of deferred computation is principally useful when computations are likely to be expensive, the simple examples shown do not necessarily use such expecially spendy computations. Most of these are directly inspired by examples used in the documentation of dask-delayed. In dask-delayed, ``Delayed`` objects are create by functions, and operations create a *directed acyclic graph* rather than perform actual computations. For example:: >>> import dask >>> @dask.delayed ... def later(x): ... return x ... >>> output = [] >>> data = [23, 45, 62] >>> for x in data: ... x = later(x) ... a = x * 3 ... b = 2**x ... c = a + b ... output.append(c) ... >>> total = sum(output) >>> total Delayed('add-8f4018dbf2d3c1d8e6349c3e0509d1a0') >>> total.compute() 4611721202807865734 >>> total.visualize() .. figure:: pep-9999-dag.png :align: center :width: 50% :class: invert-in-dark-mode Figure 1. Dask DAG created from simple operations. Under this PEP, the soft keyword ``later`` would work in a similar manner to this dask.delayed code. But rather than requiring calling ``.compute()`` on a ``Delayed`` object to arrive at the result of a computation, every reference to a binding would perform the "compute" *unless* it was itself a deferred expression. So the equivalent code under this PEP would be:: >>> output = [] >>> data = [23, 45, 62] >>> for later x in data: ... a = later (x * 3) ... b = later (2**x) ... c = later (a + b) ... output.append(later c) ... >>> total = later sum(output) >>> type(total) # type() does not un-thunk <class 'DeferredObject'> >>> if value_needed: ... print(total) # Actual computation occurs here 4611721202807865734 In the example, we assume that the built-in function `type()` is special in not counting as a reference to the binding for purpose of realizing a computation. Alternately, some new special function like `isdeferred()` might be used to check for ``Deferred`` objects. In general, however, every regular reference to a bound object will force a computation and re-binding on a ``Deferred``. This includes access to simple names, but also similarly to instance attributes, index positions in lists or tuples, or any other means by which an object may be referenced. Rejected Spellings ================== A number of alternate spellings for creating a ``Deferred`` object are possible. This PEP-author has little preference among them. The words ``defer`` or ``delay``, or their past participles ``deferred`` and ``delayed`` are commonly used in discussions of lazy evaluation. All of these would work equally well as the suggested soft keyword ``later``. The keyword ``lazy`` is not completely implausible, but does not seem to read as well. No punctuation is immediately obvious for this purpose, although surrounding expressions with backticks is somewhat suggestive of quoting in Lisp, and perhaps slightly reminiscent of the ancient use of backtick for shell commands in Python 1.x. E.g.:: might_use = `math.gcd(a, math.factorial(b))` Relationship to PEP-0671 ======================== The concept of "late-bound function argument defaults" is introduced in :pep:`671`. Under that proposal, a special syntactic marker would be permitted in function signatures with default arguments to allow the expressions indicated as defaults to be evaluated at call time rather than at runtime. In current Python, we might write a toy function such as:: def func(items=[], n=None): if n is None: n = len(items) items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 Using the :pep:`671` approach this could be simplified somewhat as:: def func(items=[], n=>len(items)): # late-bound defaults act as if bound here items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 Under the current PEP, evaluation of a ``Deferred`` object only occurs upon reference. That is, for the current toy function, the evaluation would not occur until the ``print(n)`` line.:: def func(items=[], n=later len(items)): items.append("Hello") print(n) func([1, 2, 3]) # prints: 4 To completely replicate the behavior of PEP-0671, an extra line at the start of the function body would be required:: def func(items=[], n=later len(items)): n = n # Evaluate the Deferred and re-bind the name n items.append("Hello") print(n) func([1, 2, 3]) # prints: 3 References ========== https://github.com/DavidMertz/peps/blob/master/pep-9999.rst Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

16 48

Adopting better naming conventions (food for thought)
by abed...＠gmail.com 25 Jun '22

25 Jun '22

I was reading some of Kevlin Henney's views on naming conventions <https://kevlinhenney.medium.com/exceptional-naming-6e3c8f5bffac>, and I realized that, of the 65 classes that make up Python's built-in exceptions <https://docs.python.org/3/library/exceptions.html#exception-hierarchy>, only 8 (by my count) satisfy Henney's pretty basic concept of a good Exception name: *The name should represent whatever the problem is, and should do so directly and specifically. To add Exception to the end is either redundant — so remove it — or an indication of a poor name — so rename it.* Those are: SystemExit, KeyboardInterrupt, GeneratorExit, Exception, StopIteration, StopAsyncIteration, Warning, and BaseException (though Henney also discourages the use of "Base" in a base-class's name) I'm sure there are others throughout the standard library too. I always caution novice programmers to err on the side of pragmatism over dogmatic adherence to "sacred truths". To that end: 1) I realize changing built-in exception names would break backwards compatibility and probably isn't worth doing. This post is more intended as food for thought going forward and to bring more attention to Kevlin Henney's ideas. 2) I think many of the built-in exceptions are actually fine, particularly ones that represent a broad class of more specific exceptions like OSError. I don't think OSError has a specific enough name to be raised on its own (perhaps it should be abstract or otherwise not directly instantiable?), but catching error categories can be pretty helpful even if they don't lend themselves to explicit names. I would suggest a more ideal naming scheme would be something like the following: - BaseException -> Exception # The text for GeneratorExit <https://docs.python.org/3/library/exceptions.html#GeneratorExit> seems to indicate that in Python, an Error is considered a subclass of an Exception, so this naming seems more appropriate though it contradicts other views on the distinction between Exception and Error <https://stackoverflow.com/questions/5813614/what-is-difference-between-erro…> . - SystemExit - *KeyboardInterrupt -> UserInterrupt *#This is supposed to capture a user-initiated exit, whether it came from a keyboard or a magic wand is generally much less relevant - GeneratorExit - Exception -> Error - *StopIteration* - *StopAsyncIteration* - *ArithmeticError* - FloatingPointError -> Not used, get rid of it? - OverflowError -> Overflow - ZeroDivisionError -> ZeroDivision or Undefined - *AssertionError -> Contradiction or AssertionViolation* - *AttributeError* - *+ AttributeNotFound or AttributeUnavailable* - *+ AttributeUnassignable* - *BufferError* - *<specific errors>* - *EOFError -> EndOfFile* - *ImportError * - ModuleNotFoundError -> ModuleNotFound - *LookupError * - *IndexError -> IndexOutOfBounds* - *KeyError -> KeyNotFound* - *MemoryError -> OutOfMemory * - *NameError -> InvalidName (Not needed?) * - UnboundLocalError -> UnboundLocal - *OSError * - BlockingIOError -> BlockingIO - ChildProcessError -> ChildProcessFailed - ConnectionError - BrokenPipeError -> BrokenPipe - ConnectionAbortedError -> ConnectionAborted - ConnectionRefusedError -> ConnectionRefused - ConnectionResetError -> ConnectionReset - FileExistsError -> FileAlreadyExists - FileNotFoundError -> FileNotFound - InterruptedError -> Interrupted - IsADirectoryError -> NotAFile - NotADirectoryError -> NotADirectory - PermissionError -> PermissionDenied - ProcessLookupError -> ProcessNotFound - TimeoutError -> Timeout - *ReferenceError -> ReferentCollected? * - *RuntimeError * - NotImplementedError -> NotImplemented - RecursionError -> MaxRecursionDepthExceeded or RecursionOverflow - *SyntaxError * - IndentationError - TabError -> InconsistentIndentation - *SystemError -> SystemFailure* - *TypeError -> IncompatibleType * - *ValueError * - UnicodeError - UnicodeDecodeError - UnicodeEncodeError - UnicodeTranslateError - *Warning* # Should this be under Error? - DeprecationWarning -> Deprecated - PendingDeprecationWarning -> Obsolete - RuntimeWarning - SyntaxWarning - UserWarning - FutureWarning - ImportWarning - UnicodeWarning - BytesWarning - EncodingWarning - ResourceWarning

2 1

Ellipsis (...) to be roughly synonymous with *<varname> in destructuring but without capture.
by Steve Jorgensen 23 Jun '22

23 Jun '22

This is based on previous discussions of possible ways of matching all remaining items during destructuring but without iterating of remaining final items. This is not exactly a direct replacement for that idea though, and skipping iteration of final items might or might not be part of the goal. In this proposal, the ellipsis (...) can be used in the expression on the left side of the equals sign in destructuring anywhere that `*<varname>` can appear and has approximately the same meaning. The difference is that when the ellipsis is used, the matched items are not stored in variables. This can be useful when the matched data might be very large. ..., last_one = <expression> a, ..., z = <expression> first_one, ... = <expression> Additionally, when the ellipsis comes last and the data is being retrieved by iterating, stop retrieving items since that might be expensive and we know that we will not use them. Alternative A: Still iterate over items when the ellipsis comes last (for side effects) but introduce a new `final_elipsis` object that is used to stop iteration. The negation of `ellipsis` (e.g. `-...`) could return `final_ellipsis` in that case. Alternative B: Still iterate over items when the ellipsis comes last (for side effects) and don't provide any new means of skipping iteration over final items. The programmer can use islice to achieve that.

2 3

Re: Add a line_offsets() method to str
by Inada Naoki 22 Jun '22

22 Jun '22

On Tue, Jun 21, 2022 at 3:49 AM Marco Sulla <Marco.Sulla.Python(a)gmail.com> wrote: > > On Sun, 19 Jun 2022 at 03:06, Inada Naoki <songofacandy(a)gmail.com> wrote: > > FWIW, I had proposed str.iterlines() to fix incompatibility between > > IO.readlines() and str.splitlines(). > > It's a good idea IMHO. In your mind, str.iterlines() will find only > \n, \r and \r\n, as IO.readlines() do? I ask this because > str.splitlines() finds all line boundary chars. > I don't remember the old discussion, but I think so. Another idea is adding some option to `splitlines()` that uses only universal newlines. (e.g. str.splitlines(keepends=False, *, ascii=False)) Then we can add `str.iterlines` that have the same arguments to `str.splitlines`. -- Inada Naoki <songofacandy(a)gmail.com>

1 0

PEP 671 (late-bound arg defaults), next round of discussion!
by Chris Angelico 21 Jun '22

21 Jun '22

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere. *PEP 671: Syntax for late-bound function argument defaults* Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it? 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? (It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.) 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? 4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.) 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best! The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go. ChrisA

36 342

Null wildcard in de-structuring to ignore remainder and stop iterating
by Steve Jorgensen 21 Jun '22

21 Jun '22

A contrived use case: with open('document.txt', 'r') as io: (line1, line2, *) = io It is possible to kind of achieve the same result using `*_` except that would actually read all the lines from the file, even if we only want the first 2. …so I am suggesting that we use the bare `*` here to mean that we don't care whether there are additional items in the sequence, _and_ we want to stop iterating.

9 26

Bare wildcard in de-structuring to ignore remainder and stop iterating (restart)
by Steve Jorgensen 20 Jun '22

20 Jun '22

Restarting this with an improved title "Bare" vs "Raw", and I will try not to digress so much in the new thread. My suggestion is to allow a bare asterisk at the end of a desctructuring expression to indicate that additional elements are to be ignored if present and not iterated over if the rhs is being evaluated by iterating. (first, second, *) = items This provides a way of using destructuring from something that will be processed by iterating and for which the number of items might be very large and/or accessing of successive items is expensive. As Paul Moore pointed out in the original thread, itertools.islice can be used to limit the number of items iterated over. That's a nice solution, but it required knowing or thinking of the solution, an additional import, and repetition of the count of items to be destrucured at the outermost nesting level on the lhs. What are people's impressions of this idea. Is it valuable enough to pursue writing a PEP? If so, then what should I do in writing the PEP to make sure that it's somewhat close to something that can potentially be accepted? Perhaps, there is a guide for doing that?

10 29

Add a line_offsets() method to str
by Jonathan Slenders 20 Jun '22

20 Jun '22

Hi everyone, Today was the 3rd time I came across a situation where it was needed to retrieve all the positions of the line endings (or beginnings) in a very long python string as efficiently as possible. First time, it was needed in prompt_toolkit, where I spent a crazy amount of time looking for the most performant solution. Second time was in a commercial project where performance was very critical too. Third time is for the Rich/Textual project from Will McGugan. (See: https://twitter.com/willmcgugan/status/1537782771137011715 ) The problem is that the `str` type doesn't expose any API to efficiently find all \n positions. Every Python implementation is either calling `.index()` in a loop and collecting the results or running a regex over the string and collecting all positions. For long strings, depending on the implementation, this results in a lot of overhead due to either: - calling Python functions (or any other Python instruction) for every \n character in the input. The amount of executed Python instructions is O(n) here. - Copying string data into new strings. The fastest solution I've been using for some time, does this (simplified): `accumulate(chain([0], map(len, text.splitlines(True))))`. The performance is great here, because the amount of Python instructions is O(1). Everything is chained in C-code thanks to itertools. Because of that, it can outperform the regex solution with a factor of ~2.5. (Regex isn't slow, but iterating over the results is.) The bad things about this solution is however: - Very cumbersome syntax. - We call `splitlines()` which internally allocates a huge amount of strings, only to use their lengths. That is still much more overhead then a simple for-loop in C would be. Performance matters here, because for these kind of problems, the list of integers that gets produced is typically used as an index to quickly find character offsets in the original string, depending on which line is displayed/processed. The bisect library helps too to quickly convert any index position of that string into a line number. The point is, that for big inputs, the amount of Python instructions executed is not O(n), but O(1). Of course, some of the C code remains O(n). So, my ask here. Would it make sense to add a `line_offsets()` method to `str`? Or even `character_offsets(character)` if we want to do that for any character? Or `indexes(...)/indices(...)` if we would allow substrings of arbitrary lengths? Thanks, Jonathan

12 19

Add tz argument to date.today()
by Lucas Wiman 20 Jun '22

20 Jun '22

Since "today" depends on the time zone, it should be an optional argument to date.today(). The interface should be the same as datetime.now(tz=None), with date.today() returning the date in the system time zone. Rationale: It is common for processes to run in different timezones than the relevant end user. The most common case is probably a server running in UTC, which needs to do date calculations in timezones elsewhere. As a contrived example, you might do something like: tomorrow = date.today() + timedelta(days=1) return render_template(..., ship_by_date=tomorrow) But then if fulfillment is operating e.g. in US/Pacific time, the ship by date suddenly shows two days hence after 5 or 6pm when UTC midnight happens. In my particular use case, we get some collection of business rules about when a particular record is considered "stale" or prioritized, and naively using date.today() can lead to off-by-one errors. The way to "fix" the code above is to reference "now": tomorrow = datetime.now(tz=...).date() + timedelta(days=1) return render_template(..., ship_by_date=tomorrow) While this is not terrible, it seems like the API between now() and today() should be consistent: in reality, they need the same set of inputs. Calling date.today(...) is more explicit and specific than calling datetime.now(...).date(). Best wishes, Lucas

3 2

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas