Mailman 3 October 2020 - Python-ideas

Inconsistency in splitting strings
by Antal Gábor 26 Oct '20

26 Oct '20

Hey there, This is my first letter here, I'll try my best to explain the problem. First of all, I ran into this "problem" several times. I find str.split() a bit confusing. First of all, this is a corner case, but can happen like every day. print("".split()) # returns [] print("".split(" ")) # returns [''] print("".split("\t")) # returns [''] print("".split("\n")) # returns [''] print("".splitlines()) # returns [] So using split with or without a separator matters a lot, even when we use the same whitespace character split() uses. I think it is quite annoying. My idea is to return a list with an empty string in all cases mentioned above. Best, Gabor

5 4

TextIOWrapper support for null-terminated lines
by Brian Allen Vanderburg II 26 Oct '20

26 Oct '20

I have a use case where I'm writing a small filter in Python which should be able to take output from the find command to filter and optionally pass it to other commands (like xargs), but also can be passed specific filenames for the input and/or output. Find can output it's filenames in null-terminated lines since it is possible to have newlines in a filename(yuck). There are workarounds that I can use to get the same thing done, essentially using TextIOWrapper in '' mode and scanning for any null terminators in the lines/splitting manually, but it would be nice if the TextIOWrapper supported null characters as line terminators for such use cases if possible. Something line this: # configure output stream if(arg.input): input_stream = open(arg.input, "rt", newline="\000" if arg.null_lines else None) else: input_stream = sys.stdin input_stream.reconfigure(newline="\000" if arg.null_lines else None) # configure input stream Because my use case also has the ability to specify an encoding for the input/output, my workaround was originally to use codec.getreader/getwriter which turned out to be rather slow (the output of the pipe to the filter was in bursts), so since I'm currently using 3.5 and can't use reconfigure, currently involves detaching sys.stdin and reattaching to a new TextIOWrapper, setting buffering to 0 and newline to '' in null-terminated mode to get all the characters untranslated and manually scanning/splitting on null characters, though I'm sure there are better way to do this. Thanks, Brian Vanderburg II

5 6

execute function on iterator items without changing or consuming iterator
by anderssonpublic＠gmail.com 25 Oct '20

25 Oct '20

Python has many tools for iteration, such as map and filter, that change an iterator into another iterator without consuming the iterator. These make it simple to deal with many items in a list without consuming excessive memory. Occasionally it is useful to be able to tap into iterator items and execute a function (such as a side effect to validate elements or print them) without making any changes to the overall iterator. This is similar to the idea of a tap in rxjs: https://rxjs-dev.firebaseapp.com/api/operators/tap. The proposed interface would be: def generate_items() -> list[int]: some_iter = range(10) some_iter = tap(assert_int, some_iter) some_iter = tap(print, some_iter) return list(some_iter) This would be useful to (for example): 1. Debug chained iterators without a debugger (at the moment you would have to convert the list and then print the whole list or include a print statement in one of the chained functions) 2. Check that items in an iterator conform to assumptions and raising exceptions if they do not 3. Improve type hints in editors since after the assert is executed all items conform to the assert in the tap (for example, they are an integer) The implementation would be quite simple (at least in python): def tap(func: typing.Callable[[T], typing.Any], iter: typing.Iterable[T]) -> typing.iterable[T]: for item in iter: func(item) yield item Thoughts?

5 4

Extrapolating PEP 634: The walrus was waiting for patmat all along
by Valentin Berlier 24 Oct '20

24 Oct '20

Pattern-matching is great. I think PEP 634 is on the right track, but it would be a waste to only use pattern-matching for choosing a branch in a match statement. Let’s look at Rust: if let [x, y] = my_array { ... } Rust "if let" constructs are an alternative to full-blown match statements that make it less verbose to match a single pattern. We have a similar problem. The syntax proposed by PEP 634 is pretty verbose for matching a single pattern: match my_list: case [x, y]: ... Two keywords and two indentation levels. We can do better: if [x, y] := my_list: ... Yes, your first intuition is right. This looks similar to the Rust version but would work completely differently. But hear me out. Let's look past my terrible example and focus on the idea. 1. The walrus operator was purposefully designed to create bindings and not to perform assignments to arbitrary lvalues. 2. Matching a pattern only introduces bindings as well, no assignments. 3. The current behavior of the walrus operator is equivalent to matching the right-side operand to an "irrefutable" capture pattern. Allowing the walrus operator to do pattern-matching would simply make the returned value conditional. If the pattern doesn't match, the walrus operator returns None. print(x := 42) # 42 print(1 := 42) # None The current PEG parser would backtrack when encountering the walrus operator to interpret the left-side as a pattern. Finally, more examples: # Buy every pizza on the menu sum( price for food in menu if ({"type": "pizza", "price": price} := food) ) # Download all images from document {url: download(url) for tag in html if (Img(src=url) := tag)} # Monitor service health while Response(status=200, json={"stats": stats}) := health_check(): print(stats) time.sleep(5) I'm convinced that making the walrus operator a pattern-matching operator would turn it into the perfect companion for PEP 634. What do you think? References: - PEP 634: https://www.python.org/dev/peps/pep-0634/ - Rust "if let": https://doc.rust-lang.org/book/ch06-03-if-let.html -- Valentin

3 4

f-string: empty expression should be allowed
by Hans Ginzel 23 Oct '20

23 Oct '20

Hello, consider this snippet please cursor.execute(f"INSERT INTO {table} VALUES (1, '{}');") SyntaxError: f-string: empty expression not allowed It is (absolutely) correct to insert empty json into database table field. Empty expression in f-string should * (silently) expand as '{}' (opening and closing braces), * generate a (compile time) warning if requested, e.g. with -W. Thank you in advance Hans

9 14

f-strings as assignment targets
by Dennis Sweeney 23 Oct '20

23 Oct '20

TL;DR: I propose the following behavior: >>> s = "She turned me into a newt." >>> f"She turned me into a {animal}." = s >>> animal 'newt' >>> f"A {animal}?" = s Traceback (most recent call last): File "<pyshell#2>", line 1, in <module> f"A {animal}?" = s ValueError: f-string assignment target does not match 'She turned me into a newt.' >>> f"{hh:d}:{mm:d}:{ss:d}" = "11:59:59" >>> hh, mm, ss (11, 59, 59) === Rationale === Part of the reason I like f-strings so much is that they reduce the cognitive overhead of reading code: they allow you to see *what* is being inserted into a string in a way that also effortlessly shows *where* in the string the value is being inserted. There is no need to "paint-by-numbers" and remember which variable is {0} and which is {1} in an unnecessary extra layer of indirection. F-strings allow string formatting that is not only intelligible, but *locally* intelligible. What I propose is the inverse feature, where you can assign a string to an f-string, and the interpreter will maintain an invariant kept in many other cases: >>> a[n] = 17 >>> a[n] == 17 True >>> obj.x = "foo" >>> obj.x == "foo" True # Proposed: >>> f"It is {hh}:{mm} {am_or_pm}" = "It is 11:45 PM" >>> f"It is {hh}:{mm} {am_or_pm}" == "It is 11:45 PM" True >>> hh '11' This could be thought of as analogous to the c language's scanf function, something I've always felt was just slightly lacking in Python. I think such a feature would more clearly allow readers of Python code to answer the question "What kinds of strings are allowed here?". It would add certainty to programs that accept strings, confirming early that the data you have is the data you want. The code reads like a specification that beginners can understand in a blink. === Existing way of achieving this === As of now, you could achieve the behavior with regular expressions: >>> import re >>> pattern = re.compile(r'It is (.+):(.+) (.+)') >>> match = pattern.fullmatch("It is 11:45 PM") >>> hh, mm, am_or_pm = match.groups() >>> hh '11' But this suffers from the same paint-by-numbers, extra-indirection issue that old-style string formatting runs into, an issue that f-strings improve upon. You could also do a strange mishmash of built-in str operations, like >>> s = "It is 11:45 PM" >>> empty, rest = s.split("It is ") >>> assert empty == "" >>> hh, rest = rest.split(":") >>> mm, am_or_pm = s.split(" ") >>> hh '11' But this is 5 different lines to express one simple idea. How many different times have you written a micro-parser like this? === Specification (open to bikeshedding) === In general, the goal would be to pursue the assignment-becomes-equal invariant above. By default, assignment targets within f-strings would be matched as strings. However, adding in a format specifier would allow the matches to be evaluated as different data types, e.g. f'{foo:d}' = "1" would make foo become the integer 1. If a more complex format specifier was added that did not match anything that the f-string could produce as an expression, then we'd still raise a ValueError: >>> f"{x:.02f}" = "0.12345" Traceback (most recent call last): File "<pyshell#2>", line 1, in <module> f"{x:.02f}" = "0.12345" ValueError: f-string assignment target does not match '0.12345' If we're feeling adventurous, one could turn the !r repr flag in a match into an eval() of the matched string. The f-string would match with the same eager semantics as regular expressions, backtracking when a match is not made on the first attempt. Let me know what you think!

22 110

Call a function at invocation
by Michael Smith 23 Oct '20

23 Oct '20

There are many ways to invoke a function from the commandline. You can use setuptools' console_scripts entrypoint. You can use a decorator from click. And of course you can always do the classic if __name__ == "__main__": main() to call whatever main() is. But there are inconveniences with these approaches. For setuptools, you have to actually run setup, somehow. For click, you have to install click. And for __name__, you are either locked into a single function name, or you have to write some arg parsing to determine what name to use. I propose it would be nice to be able to call a function from python, using syntax like python -m module:thunk The simplest proposal, I think, is if the function must accept no arguments -- or at least no required ones. This could be as straightforward as just being shorthand for python -c 'import module; module.thunk()' and remove a small amount of code that is repeated very frequently. I picked the colon syntax because that is what several other tools that enable calling functions from the commandline seem to do, but if your only objection is the specific syntax I picked, please propose a different one. What do you think?

7 12

Re: f-strings as assignment targets
by Chris Angelico 21 Oct '20

21 Oct '20

On Wed, Oct 21, 2020 at 1:13 AM Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > > On 20/10/2020 13:05, Chris Angelico wrote: > > On Tue, Oct 20, 2020 at 10:37 PM Rob Cliffe via Python-ideas > > <python-ideas(a)python.org> wrote: > >> In short, "assigning" to f-strings is not and cannot be a simple reversal of having them in expressions. Rather, it is opening a big can of worms. > >> > > It's not a reversal of them being in expressions any more than > > assigning to a list display is a reversal of constructing a list. It's > > a parallel operation, a counterpart. > > > > It would be Python's way of offering an sscanf-like operation, which > > is something that I've frequently wished for. Regular expressions > > aren't ideal for all situations, and there are plenty of times when a > > simple set of parsing rules could be very nicely encoded into a > > compact form like this. C's sscanf and sprintf aren't perfect > > counterparts, but they're incredibly valuable. Python has percent > > formatting for sprintf, but no form of sscanf. > > > Er, well, why not just add a sscanf to Python? A couple of reasons, but the main one is that you can't have "output variables" (in C, sscanf takes pointers, so you pass it the address of your variables), which means that all you'd get is a fancy string split and then you put all the assignment on the left. That leads to an all-or-nothing result for the simple form, and no easy way to do anything else. Consider: a, b, c = sscanf("%d %d %d", "123 432") Boom! Instead of having a clean way to represent partial parsing. It'd still be better than regex partial parsing, where you get absolute silence and no way to debug it, but it's not nearly as good as could be done. ChrisA

5 15

Consolidated ideas
by Madhu Mohan Nelemane 21 Oct '20

21 Oct '20

Hi, I have been programming in python for about 6 years now. However, I am a newbie to this mailing list. My observation in the last couple of days is that this is literally an explosion of ideas and in various degrees. I am wondering if there is a common place or site where the ideas that went beyond proposal and got heads up for implementation have been listed. Are they consolidated somewhere to be discussed further ? Or is it something I have to scrape through mailing list archives ? Thanks and Regards, Madhu Mohan

2 1

Optimize out setting unused underscored local variables
by Serhiy Storchaka 21 Oct '20

21 Oct '20

It is common to use _ as a placeholder for variable whose value is not used. For example: for _ in range(n) head, _, tail = name.partition(':') first, *_, last = items I though about optimizing out unnecessary assignments. Actually I wrote a patch half year ago and tested it. It did not add much to performance, and did not reduce the bytecode, so I ccoled down to him and put it off. Later PEP 622 was declared the use of _ in pattern matching, so I was waiting for what it would come to. And now PEP 640 is created to solve the same problem. My patch was too conservative. It was limited to local variables and underscored names. For global variables we can't determine if the variable is not used. And eliminating unused non-underscored variables breaks too many tests (mainly for debugger, tracing, etc). I was not sure whether it should be limited to underscored names or just '_' (I seen also uses of '__' as a drop out variable in wild). Maybe we can extend this to global '_'. Global '_' is used as a holder for the last result in REPL and as an alias to gettext (this is the reason of PEP 640), but none of them is actually set in the assignment statement. You can use `globals()['_'] = ...` or `globals().update({'_': ...})` or `sys.modules[__name__]._ = ...` to set global '_'. I do not want to create tens of alternate PEPs with minor variations, and this issue is not worth a PEP. What is your opinion about this? Is it worth to include such optimization? For what kind of variables should it be applied? Should it include global '_'? Should it be merely an optimization (maybe controlled by the -O option) or change in the language?

6 10