Mailman 3 April 2021 - Python-ideas

Re: Non-blocking concurrent.futures ThreadPoolExecutor
by Brunoais April 10, 2021

April 10, 2021

(I know this has some time but I think it makes sense in context) That is much easier said than done in real life when the whole project is doing by a team and you are stuck with whatever you all do in the group. Even then, I prefer making my own programs with a mix of daemon and non-daemon threads to do the word. Daemon threads do work that can be interrupted at any time for any reason. Non-daemon threads are more well behaved and focus on doing their short job with hard IO. In my case, the ThreadPoolExecutor does most of the work for me for these cases. Unfortunately, I think I'll have to implement my own for daemon threads.

1 0

Greek alphabet in built-in`string` module
by mitchell.negus.57＠gmail.com April 9, 2021

April 9, 2021

I was wondering if it might be reasonable to consider adding a set of Greek letters to the constants in the `string` module. In my mind, the format would follow the existing convention—constants for both `greek_lowercase` and `greek_uppercase`, as well as a combined constant `greek_letters`. Judging by how these constants are defined in the module, this seems like it might be an almost trivially easy addition (with minimal future maintenance required), and could provide a neat little functionality. Where Python 3 already uses unicode strings, I don't believe this would cause any backward compatibility issues. The only one drawback I can see might be polluting the namespace if it is not a commonly used feature. That said, the Greek alphabet is used so commonly in math/science that I'd hazard a guess it might be valuable for some, especially as unicode becomes increasingly prevalent. Obviously developers could build their own similar constant just by creating a string with those characters, but this would save a step, whether that's adding a dependency or copying/pasting a handful of characters. I also anticipate that the argument may come up that this opens a can of worms, and then why not include even more symbols. I think that is a valid concern, though I think the Greek alphabet is somewhat unique in it's prevalence (due to it's use in math/science), on top of the fact that it is both limited and permanent (in a way that things like logograms and emojis may not be). I'm certainly not an expert on the Python source code though, so please correct me if there's an obvious reason not to add this or if this has been debated before :)

4 4

Allow Annotated annotation in dataclass
by brianmarroq＠gmail.com April 5, 2021

April 5, 2021

typing.Annotated could be used to build dataclasses. Using Annotated will allow libraries to add functionality to a dataclass without having to change dataclass creation or behavior. The example below shows how a dataclass could be implemented. It continues the example of struct2 shown in pep593. From a dataclass point of view, the Sample and AnnotatedSample would be equivalent. ```python @dataclass class Sample: a: int b: int c: int = field(default=5) d: int = 10 e: int = field(default=10) @packed @dataclass class AnnotatedSample: a: Annotated[int, ctype("I")] b: int c: Annotated[int, field(default=5), ctype("I")] d: Annotated[int, ctype("h")] = 10 e: Annotated[int, ctype("h")] = field(default=10) out_bytes = struct2.pack(AnnotatedSample()) ``` When parsing the Annotated parameters, the dataclass decorator will only look at the type parameter and ```field``` parameter if present. If not Annotated, it falls back to existing behavior. Let me know what you think. Thanks!

3 2

Change the exception type and message raised when _curses is not found
by Shreyan Avigyan April 5, 2021

April 5, 2021

When importing the curses module, be it on Windows or Darwin or UNIX-based OS or any other platform, if the _curses module is not found then just a ModuleNotFoundError is raised. But this error is not very informational in case of _curses module. Since the curses module is packaged with the Python interpreter itself at first it may seem, to beginners especially, that the Python interpreter was not installed correctly and then they would go searching for an answer for about 4-5 days. We know that curses library is not installed on windows by default and may or may not be present on other operating systems. Most UNIX system have ncurses or other curses library installed by default. Python errors have a reputation of being very informational. I would like to submit a PR to modify the curses module a little bit by declaring a BaseException class and raising that Exception with the message "_curses module not found. Make sure a curses library is installed" or some kind of message like that. But before I do that I would like to take advice from experienced developers about somethings. Is this change in the exception, raised when _curses module is not found, acceptable by the Python Community? If it is then should a draft PEP be submitted or should a PR be directly submitted to https://github.com/python/cpython?

5 9

Reverse polish notation
by John April 3, 2021

April 3, 2021

It occurs to me it's a syntax error everywhere to put two potentially-calculable terms in a row with nothing between them, e.g.: ```python a = b c mylist = [x 6 for x in y if "a" in x] ``` Unless I'm missing something, this makes it relatively easy (from a syntax perspective; writing a parser r hard) to detect such: ```python a = b c 6 ** + 5 / mylist = [x 6 + for x in y if "a" in x] ``` This is a bit less ambiguous and simpler to write (and easier to read, caveat you have to know *how* to read it) when dealing with complicated expressions with a lot of parenthesis—a use case I automatically want to dismiss despite always running into this immediately. ```python # tl counts from sleep_us() print(" Iteration: %s\t Correct: %s of 10 (%s/sec)" % (i,c,1000/(tl/(1000*i)) if tl > 0 else 0)) # RPN print(" Iteration: %s\t Correct: %s of 10 (%s/sec)" % (i,c, 1000 tl 1000 i * / / if tl > 0 else 0)) ``` I took the following steps in writing this: * 1000 / * 1000 tl / / * 1000 tl 1000 i * / / The first is the incomplete operation 1000 / <something> The second adds the divisor between the term (1000) and the operator (/), which is again going to be tl / <something>. The third adds the divisor there, 1000 * i, between tl and its operator. Reading it back is easy enough: scan until finding an operator and take the two terms to the left, e.g. u=(1000*i), then v=tl/u, then w=1000/v. You could also read "1000 tl 1000 i * / / gettime() +" by further doing x=w+gettime() ... it's read directly in sequence. From my perspective, one of these is harder to read than the other, and no amount of carefully-placed white spaces is going to replace a piece of scrap paper to untangle the order of operations. You can break it down to intermediate steps, but that can make it difficult to follow a complex calculation. Downsides: * Have to actually implement it in the parser * If almost nobody uses it, still have to maintain it until the end of time for compatibility * While it takes all of 30 seconds to learn, it's not obvious what in the heck you're looking at if you've never seen it * While it's easier to revise complex calculations and at times more readable, infix exists for a reason. RPN is built from the outside in for nested computations (although often that also involves going back and forth to add extra parenthesis while writing infix, rather than simply typing it out left to right in one go) Things to not do: * Mix-and-match single expressions e.g. 6 2 (3 * 4) + *, 6 * 2 3 4 * + While the first looks innocent enough, without the parentheses it becomes 6*(2*3 + 4). The second is just unreadable. * Reverse polish function calls e.g. 3 4 + str() no, just no * Prohibit infix in a call e.g. 3 4 getdata(n + 2) + * This isn't ambiguous, nor is it more difficult to read or parse Thoughts?

12 31

Access to source code when using exec()
by Alex Hall April 3, 2021

April 3, 2021

When using `exec(string)`, the standard Python REPL, or `python -c` in a terminal, the executed source code is not stored anywhere and cannot be retrieved (except in the `-c` case by looking at `sys.argv`). This means that: - The source line doesn't show up in tracebacks, e.g: >>> exec("1/0") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> ZeroDivisionError: division by zero - Users cannot retrieve the source code using `inspect`, although many want to: - https://stackoverflow.com/questions/29850508/python-how-to-inspect-code-of-… - https://stackoverflow.com/questions/12072252/cant-get-source-code-for-a-met… - https://stackoverflow.com/questions/62160529/how-to-get-source-inside-exec-… - https://stackoverflow.com/questions/59244323/how-can-you-get-the-source-cod… - https://stackoverflow.com/questions/50822656/source-code-for-an-evaled-dyna… - https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-… - Libraries such as traceback formatters and debuggers which depend on source code can't work. This often manifests when people try out a library for the first time in the REPL and it doesn't seem to work. This leads to many similar looking bug reports which sometimes take a while to resolve. Probably many more people think these libraries don't work without ever opening an issue. - https://github.com/cknd/stackprinter/issues/34 - https://github.com/Qix-/better-exceptions/issues/17 - https://github.com/Qix-/better-exceptions/issues/68 - https://github.com/aroberge/friendly-traceback/issues/23 - https://github.com/gruns/icecream/issues/17 - https://github.com/cool-RR/PySnooper/issues/135 - https://github.com/cool-RR/PySnooper/issues/164 - https://github.com/alexmojaki/sorcery/issues/3 - https://github.com/alexmojaki/sorcery/issues/5 - https://github.com/alexmojaki/sorcery/issues/6 I suggest that: 1. `exec(string)` assigns a unique generated filename to the compiled code object and stores the source code in the `linecache` module under that filename, similar to what the IPython REPL does. I imagine there might be complaints about memory usage, security implications of revealing source code, compatibility, etc. If that's an issue this could be off by default with a flag to turn it on. 2. The REPL, `-c`, and other similar mechanisms (IDLE?) either follow the same strategy as `exec` or defer to it. 3. The standard exception hook falls back to `linecache` for filenames that don't exist. I don't know why it doesn't already. Here is a question on that topic: https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-…

2 1

str.substring_after and str.substring_before as in Kotlin
by janfrederik.konopka＠gmail.com April 3, 2021

April 3, 2021

Dear pythonistas, I would really like having a function like String.substringAfter(sep) and String.substringBefore(sep) as in Kotlin. fun String.substringAfter( delimiter: String, missingDelimiterValue: String = this ): String substringAfter/substringBefore takes a delimiter, and keeps everything after the first occurrence of the delimiter. This could be useful for simple HTML parsing for example. It also take a second argument to use as value when the seperator is not found (defaults to the whole input String). ---- A lot of code in the wild currently uses str.split(sep) or manual indexing, as exemplified by the two top answers here: https://stackoverflow.com/q/12572362/2111778 - str.split(sep) runs into issues if the separator occurs repeatedly: substringBefore = lambda s, sep: s.split(sep)[0] # Don't use this. substringAfter = lambda s, sep: s.split(sep)[1] # Don't use this. substringAfter = lambda s, sep: s.split(sep)[-1] # "Fixes" IndexError, but still don't use this. This will bite the user if he doesn't know about the second argument which limits the number of splits: substringBefore = lambda s, sep: s.split(sep, 1)[0] substringAfter = lambda s, sep: s.split(sep, 1)[1] # IndexError if nonexistent substringAfter = lambda s, sep: s.split(sep, 1)[-1] # original string if nonexistent I have regrettably even written code like this before: substringAfter = lambda s, sep: sep.join(s.split(sep)[1:]) - Another approach uses indexing: substringAfter = lambda s, sep: s[s.index(sep) + len(sep):] This works okay as a separate function but cannot be inlined very well due to 'sep' and 's' both being used twice: after = s[s.index('some separator string') + len('some separator string'):] Plus it's too much cognitive load for a simple "substring after" operation. - Regexes can be used, but these are too powerful and can introduce subtle bugs. Java has this problem with their String#split method which takes a regex String. So while in Python '8.8..'.split('.') == ['8', '8', '', ''], in Java you have to doubly escape the dot using "8.8..".split("\\."), EXCEPT that will result in a String[] of length 2 rather than 4, so you ACTUALLY have to do "8.8..".split("\\.", -1). So regexes have huge potential for confusing users. Another example with a substringAfter/substringBefore use case: >>> import re >>> re.findall('$.*$', 'whitespace bad (haha) (not really)') ['(haha) (not really)'] It is not obvious how to fix this: >>> re.findall("$.*?$", 'whitespace bad (haha) (not really)') ['(haha)', '(not really)'] - The best alternative currently is str.partition(sep) but such code is not very readable, plus most users do not know about it, as proven by the StackOverflow link. Note that if not found this defaults to the original str for substringAfter, and an empty string for substringBefore: substringAfter = lambda s, sep: s.partition(sep)[2] substringBefore = lambda s, sep: s.partition(sep)[0] ---- If added to the <str> class, a typical use case would be something like this: bracketed = 'whitespace bad (haha) (not really)'.substringAfter('(').substringBefore(')') assert bracketed == 'haha' I find this code highly readable. Currently the best alternatives would be: bracketed = 'whitespace bad (haha) (not really)'.partition('(')[2].partition(')')[0] bracketed = 'whitespace bad (haha) (not really)'.split('(', 1)[-1].split(')', 1)[0] bracketed = 'whitespace bad (haha) (not really)'.split('(', 1)[1].split(')', 1)[0] All of these are not very readable, the latter ones even has 4 seemingly-random integer constants. Even while writing this I got them wrong multiple times. Plus they differ in behavior: The first one returns an empty string if the separators are not found, the second one returns the original string instead, the third one throws an IndexError (but only if the first separator is missing, not if the second one is, yikes). Monkey-patching the <str> class to add str.substringAfter(sep) on the user side is also not possible as it is a C type. I think this would fit well in Python (apart from the camelCase ;)) because it would complement removeprefix/removesuffix which are being added in 3.9 already. Plus I do use substringAfter/substringBefore in Kotlin all the time. One might even think about a substringBetween to be honest: bracketed = 'whitespace bad (haha) (not really)'.substringBetween('(', ')') assert bracketed == 'haha' As an alternative, I think str.partition(sep) could be changed to return a NamedTuple rather than a simple tuple. This should be interoperable with pre-existing code. This could enable calls as follows: substringAfter = lambda s, sep: s.partition(sep).after substringBefore = lambda s, sep: s.partition(sep).before bracketed = 'whitespace bad (haha) (not really)'.partition('(').after.partition(')').before What do you think? Greetings Jan

2 1

Clarification to PEP 394
by Adam Hendry April 3, 2021

April 3, 2021

Dear pythonistas, I was wondering if we could add a clarification to PEP 394? I'm trying to get some visibility on this. There's a bullet in the PEP that states: *For scripts that are only expected to be run in an activated virtual environment, shebang lines can be written as #!/usr/bin/env python, as this instructs the script to respect the active virtual environment.* Could we clarify the following? - env is a system binary in /usr/bin available in most Linux distros that searches $PATH for strings containing the provided argument and returns the first instance it finds. In the above syntax, env will search for the first instance of python in $PATH and return it. - The shebang #! /usr/bin/env python works equally well in Linux when a script is to be run from within an activated virtual environment or when a default python installation is to be used and it is uncertain whether the python interpreter exists in /bin , /usr/bin , /usr/local/bin, or another custom path. - Since python may be an alias for python2 or python3 (typically, it is an alias for python2), the developer may choose to be specific and use the shebang #!/usr/bin/env python3 where desired. The Windows py launcher also accepts this shebang and searches the application directory, current directory, system directories, and PATH for “python.exe”. However, note that using python3 instead of python will not work as expected because it will instead use the preferred version of python 3.x that's installed and not look through directories. Although PEP 394 is geared towards Linux users, those unfamiliar with Linux don't know what env is or what it does. I've posted this issue here <https://discuss.python.org/t/pep-394-clarify-bullet-for-usr-bin-env-python/…> and give some more information. Guido said I should post this on python-dev, but I'm still waiting for access. Does anyone have any thoughts? Thank you, Adam Hendry

2 1