Mailman 3 June 2022 - Python-ideas

Null wildcard in de-structuring to ignore remainder and stop iterating
by Steve Jorgensen 21 Jun '22

21 Jun '22

A contrived use case: with open('document.txt', 'r') as io: (line1, line2, *) = io It is possible to kind of achieve the same result using `*_` except that would actually read all the lines from the file, even if we only want the first 2. …so I am suggesting that we use the bare `*` here to mean that we don't care whether there are additional items in the sequence, _and_ we want to stop iterating.

9 26

Bare wildcard in de-structuring to ignore remainder and stop iterating (restart)
by Steve Jorgensen 20 Jun '22

20 Jun '22

Restarting this with an improved title "Bare" vs "Raw", and I will try not to digress so much in the new thread. My suggestion is to allow a bare asterisk at the end of a desctructuring expression to indicate that additional elements are to be ignored if present and not iterated over if the rhs is being evaluated by iterating. (first, second, *) = items This provides a way of using destructuring from something that will be processed by iterating and for which the number of items might be very large and/or accessing of successive items is expensive. As Paul Moore pointed out in the original thread, itertools.islice can be used to limit the number of items iterated over. That's a nice solution, but it required knowing or thinking of the solution, an additional import, and repetition of the count of items to be destrucured at the outermost nesting level on the lhs. What are people's impressions of this idea. Is it valuable enough to pursue writing a PEP? If so, then what should I do in writing the PEP to make sure that it's somewhat close to something that can potentially be accepted? Perhaps, there is a guide for doing that?

10 29

Add a line_offsets() method to str
by Jonathan Slenders 20 Jun '22

20 Jun '22

Hi everyone, Today was the 3rd time I came across a situation where it was needed to retrieve all the positions of the line endings (or beginnings) in a very long python string as efficiently as possible. First time, it was needed in prompt_toolkit, where I spent a crazy amount of time looking for the most performant solution. Second time was in a commercial project where performance was very critical too. Third time is for the Rich/Textual project from Will McGugan. (See: https://twitter.com/willmcgugan/status/1537782771137011715 ) The problem is that the `str` type doesn't expose any API to efficiently find all \n positions. Every Python implementation is either calling `.index()` in a loop and collecting the results or running a regex over the string and collecting all positions. For long strings, depending on the implementation, this results in a lot of overhead due to either: - calling Python functions (or any other Python instruction) for every \n character in the input. The amount of executed Python instructions is O(n) here. - Copying string data into new strings. The fastest solution I've been using for some time, does this (simplified): `accumulate(chain([0], map(len, text.splitlines(True))))`. The performance is great here, because the amount of Python instructions is O(1). Everything is chained in C-code thanks to itertools. Because of that, it can outperform the regex solution with a factor of ~2.5. (Regex isn't slow, but iterating over the results is.) The bad things about this solution is however: - Very cumbersome syntax. - We call `splitlines()` which internally allocates a huge amount of strings, only to use their lengths. That is still much more overhead then a simple for-loop in C would be. Performance matters here, because for these kind of problems, the list of integers that gets produced is typically used as an index to quickly find character offsets in the original string, depending on which line is displayed/processed. The bisect library helps too to quickly convert any index position of that string into a line number. The point is, that for big inputs, the amount of Python instructions executed is not O(n), but O(1). Of course, some of the C code remains O(n). So, my ask here. Would it make sense to add a `line_offsets()` method to `str`? Or even `character_offsets(character)` if we want to do that for any character? Or `indexes(...)/indices(...)` if we would allow substrings of arbitrary lengths? Thanks, Jonathan

12 19

Add tz argument to date.today()
by Lucas Wiman 19 Jun '22

19 Jun '22

Since "today" depends on the time zone, it should be an optional argument to date.today(). The interface should be the same as datetime.now(tz=None), with date.today() returning the date in the system time zone. Rationale: It is common for processes to run in different timezones than the relevant end user. The most common case is probably a server running in UTC, which needs to do date calculations in timezones elsewhere. As a contrived example, you might do something like: tomorrow = date.today() + timedelta(days=1) return render_template(..., ship_by_date=tomorrow) But then if fulfillment is operating e.g. in US/Pacific time, the ship by date suddenly shows two days hence after 5 or 6pm when UTC midnight happens. In my particular use case, we get some collection of business rules about when a particular record is considered "stale" or prioritized, and naively using date.today() can lead to off-by-one errors. The way to "fix" the code above is to reference "now": tomorrow = datetime.now(tz=...).date() + timedelta(days=1) return render_template(..., ship_by_date=tomorrow) While this is not terrible, it seems like the API between now() and today() should be consistent: in reality, they need the same set of inputs. Calling date.today(...) is more explicit and specific than calling datetime.now(...).date(). Best wishes, Lucas

3 2

Type checking for **kwargs based on its use
by Mauricio Villegas 17 Jun '22

17 Jun '22

Look at the following example: ``` def some_function(f1: int): assert isinstance(f1, int) def other_function(f0: str, **kwargs): assert isinstance(f0, str) some_function(**kwargs) other_function(f0='a', f1='b') ``` I would expect a static type checker to warn that f1='b' is wrong because the type should be int. There shouldn't be a need to add a type to **kwargs since what is accepted can be deduced from how it is used. Note that some_function could be defined in another module and be used in multiple places, thus the "don't repeat yourself" and "separation of concerns" principles apply. Better to have the type for f1 be only in the definition of some_function and not in the **kwargs. I created a github issue in pyright (https://github.com/microsoft/pyright/issues/3583) and the response was that PEP 484 forbids this. I have seen that there are discussions about TypedDict for more precise typing of **kwargs. However, requiring to use TypedDict for the cases in which it is possible to derive what **kwargs accepts based on its use, seems over-complicating things, which is yet another principle "keep it simple, stupid". I would think that it wouldn't be too problematic to have a PEP stating that static type checkers when **kwargs has not type annotation are allowed to analyze the source code to identify what **kwargs accepts. I would even say that when **kwargs has a type, checkers are allowed to analyze how it is used. Also that the type annotation for **kwargs be not necessary if it is possible to deduce what it accepts. Anyway, I am just giving out this idea to see what happens. As a side note, I have been developing jsonargparse which can be used for automatically creating complex parsers based on signatures (https://jsonargparse.readthedocs.io/en/stable/#classes-methods-and-functions). I do plan to support identifying what **kwargs accepts based on its use. See all the cases currently supported https://jsonargparse.readthedocs.io/en/latest/#ast-resolver.

3 2

TaskGroup and create_task method.
by Abdulla Al Kathiri 13 Jun '22

13 Jun '22

Hello all, I am trying the TaskGroup class in the python 3.11 beta version. And I like it a lot. It makes asynchronous programming intuitive enough even for the less experienced like myself. I can see myself teaching this to my colleges when they run linux parallel subprocesses. I will try to convince our IT department to download 3.11 when it’s officially out mostly because of new improved asyncio support. The speed gain will be another good justification. However, excuse my ignorance, I have a question... Why can’t I use TaskGroup.create_task method (or asyncio.create_task function) with a Future object (eg. output of loop.run_in_executor function) since it’s an awaitable (i.e., I can do “await future”)? In other words, why can’t create_task accept all awaitable objects? I can wrap the loop.run_in_executor in an async function and call that function inside create_task with no issues. It would be nice to do the following: async def func(num: int) -> int: await asyncio.sleep(5) return num def blocking_function(*args: float) -> float: time.sleep(5) sum(args) async def func() -> tuple[int, float]: loop = asyncio.get_event_loop() args = (1, 2, 3) async with asyncio.TaskGroup() as tg: task1 = tg.create_task(func(5)) task2 = tg.create_task(loop.run_in_executor(None, blocking_function, *args)) results = await task1, await task2 return results Also, is there a way I can put a timeout for the task group so I can handle it using except* TimeOutError? Also, if I handled/extracted the exception, can I uncancel some of the tasks (that were canceled due to the exception handling) that already took long time so I can resume it in another GroupTask? Thanks, Abdulla

1 0

A “big picture” question.
by James Johnson 11 Jun '22

11 Jun '22

The unasked question never gets answered. I don’t know if you can practically use the following suggestion, but it may be clarifying anyway. When an amateur develops code incorrectly, s/he sometimes ends up with a code object that doesn’t run because of intermediate compiler optimizations. As an example, you could ask a high school class to individually build a 26x26 grid, with the first line having abcdef…xyz; the second line having zabcd…wxy, and so on. This often results in code that has been corrected, but will not run correctly because of pre-mature compiler optimizations. Another task that might illustrate this artifact of program development might be to have amateurs write a function that raises a^b mod x. When programming in a C++ environment, this is monitored by keeping track of “commit charges.” I’m imagining the following solution. Instead of implementing compiler optimizations at every execution, have python run completely interpretive like old Basic. Then introduce a command line switch, that tells python you are ready to employ compiler optimizations. There exists a -O switch; there is also a “compileall,” command, both of which introduce optimizations. It isn’t clear what each does that is different, or how to turn all optimizations off for purposes of testing (usually bad) logic. Would python benefit as a language by introducing a clearer deliberation as to when the “commit charge,” occurs? I ask this only to suggest clarity of thought… it might be impractical to disambiguate them all, after “killing yourselves,” making the language run like a bat out of hell already. Thanks for all you do, James Johnson

4 3

Re: A “big picture” question.
by Steven D'Aprano 11 Jun '22

11 Jun '22

On Fri, Jun 10, 2022 at 09:59:36PM -0500, James Johnson wrote: > I guess I was jumping to conclusions. Thank you for taking the time to look > at my email. > > I apologize if I wasted your time. No stress -- opening issues up for discussion is not a waste of time. This would be a good time to mention that there have been previous requests to have more control of what optimizations the Python byte-code compiler performs, mostly for the benefit of profiling applications. While the compiler doesn't do many, or any, large complex optimizations like a C compiler may do, it does do some keyhole optimizations. Sometimes those keyhole optimizations interfere with the ability of programs to analyse Python code and report on code coverage. While the keyhole optimization doesn't change the semantics of the code, it does change the structure of it, and makes it harder to analyse whether or not each clause in a statement is covered by tests. So other people have also requested the ability to tell the compiler to turn off all optimizations. Another factor is that as we speak, Mark Shannon is doing a lot of work on optimization for the CPython byte-code compiler, including adding JIT compilation techniques. (PyPy has had this ability for many years.) So it is possible that future compiler optimizations may start to move into the same areas that C/C++ compiler optimizations take, possibly even changing the meaning of code. It would be good to plan ahead, and start considering more fine grained optimization control, rather than the underpowered -O and -OO flags we have now. -- Steve

2 1

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger 10 Jun '22

10 Jun '22

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

default as a keyword argument for dict.get and dict.pop
by martineznicolas41541＠gmail.com 08 Jun '22

08 Jun '22

Hello! Do you know if there has been discussions around why is the default argument is positional only in the dict methods get and pop? I think ``` d.get(key, default=3) ``` way more readable than ``` d.get(key, 3) ``` specially since max and min builtin functions use default as a keyword argument.

6 6