Mailman 3 February 2020 - Python-ideas

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar 07 Mar '20

07 Mar '20

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to allow > > item = object[1, *some_tuple, 2] > item = object[1, *some_tuple, :] > > Was this overlooked in the original proposal, or deliberately rejected? > > Eric >

2 2

CSV Dictwriter - Handling escape characters
by allu.matikainen＠hotmail.com 06 Mar '20

06 Mar '20

Hi, I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters. At the moment, DictWriter writes \n:s as new lines (shown in notepad or Excel) and I would like it to just write "\n" characters in my csv file. I asked a question about this on Stack Overflow but have had no good answers yet. My question also has an example of the data and its form. https://stackoverflow.com/questions/60073809/python-how-to-write-literal-n-…

6 5

Incremental step on road to improving situation around iterable strings
by Steve Jorgensen 03 Mar '20

03 Mar '20

From reading many of the suggestions in this list (including some of my own, of course) there seem to be a lot of us in the Python coder community who find the current behavior of strings being iterable collections of single-character strings to be somewhat problematic. At the same time, trying to change that would obviously have vast consequences, so not something that can simply be changed, even with at a major version boundary. I propose that we start by making a small, safe step in the right direction so we can maybe reach the bigger goal in a far future release (e.g. 5.x or 6.x). The step I propose is to add a `chars` method that returns a sequence (read-only view) of single-character strings that behaves exactly the same as `str` currently does when treated as a collection. Encourage developers to use that for iterating over characters in a string instead of iterating over the string directly. In maybe Python 4 or 5, directly using an `str` instance as a collection could become a warning. BTW, while adding `chars`, it might also be nice to have `ords` which would be a view of the string's character sequence as `ord` integer values.

23 65

Re: Incremental step on road to improving situation around iterable strings
by Bruce Leban 02 Mar '20

02 Mar '20

--- Bruce On Mon, Feb 24, 2020 at 1:19 PM Alex Hall <alex.mojaki(a)gmail.com> wrote: > I think maybe you forgot to reply to the list as well? > > I did mean to reply to the list, so adding it back in. > How many scripts do you have where your own code directly iterated over a > string? How hard do you think it would be to update that code? > Actually quite a bit. I write a lot of code that manipulates words to create puzzles. In every one of those I use strings as iterables. For what it's worth, in one of these programs, the problem I encountered was that strings were not sufficiently performant to solve a very complex problem. I modeled the strings as integers, and built the subset of needed string-like operations on them -- including iteration. If the warning comes from a dependency that hasn't updated, you can simply > add the appropriate code to filter that category of warnings. I think you > have the option of two lines of Python code or one environment variable. > That's not a good answer IMHO. Warnings should matter. Telling me I can filter them out means it shouldn't be there in the first place. And I have to filter them out because warnings that don't matter obscure the ones that do. > > As for your sample, this has been requested twice by others, here was my > response: > > That represents a misunderstanding of my position. I think I'm an outlier >> among the advocates in this thread, but I do not believe that implementing >> any of the ideas in this proposal would significantly affect code that >> lives in the long term. Some code would become slightly better, some >> slightly worse. My concern surrounds the user experience when debugging >> code that accidentally iterates over a string. So it's impossible for me to >> show you code that becomes significantly better because that's not what I'm >> arguing about, and it's unfair to say that quoting people who have >> struggled with these bugs is not evidence for the problem. >> > The claim was made by someone that certain code is "problematic" because the strings are iterable. Asking to see before and after code is certainly relevant, so we can see exactly what the "problem" is and how it gets better. Maybe the code gets *worse* and debugging gets easier. That tradeoff is relevant. And more to the point, is there anything that can't be done by a better linter instead of mucking up the language? > Similarly for jdveiga, I cannot give you "real workable surprising code" >> because I'm talking about code that isn't workable as a result of being >> surprising. I have given examples of real non-working surprising code, and >> I can give more, and if that's not indicative of a real problem then I'm >> very confused and would appreciate more explanation. > > > Now my question is, what is the worst damage that this change could cause? > Because I haven't heard an argument that goes deeper than aesthetics. > The worst damage? The change generates spurious warnings from millions of lines of correct code. It annoys people to the extent they no longer think Python is a stable platform to use to build code. Python dies. Sure that worst case isn't likely but you asked. The bar for changing such a core feature of Python is much higher than "the worst case isn't going to kill the language." In my opinion, declaring code deprecated when there is zero likelihood that it will ever be removed is counter-productive at best. --- Bruce

2 1

Means of comparing slices for intersection or containment containment and computing intersections or unions
by Steve Jorgensen 01 Mar '20

01 Mar '20

I am purposefully proposing this for slices as opposed to ranges because it is about the bounds of the slices, not the items in synthetic sequences. Also, slices can refer to any type of value, not just integers. Presumably, these operations would raise exceptions when used with slices that have `step` values other than `None`. Alternatively, those could hypothetically be valid in restricted cases such as when all properties are either int or Fraction types. Probably better to have them be simply unsupported though. ``` a in b # True if <a> is fully contained within <b> a.intersects(b) # True if any value could be within both <a> and <b> a & b # Intersection of <a> and <b> or None if no intersection a | b # Union of <a> and <b> or Exception if neither contiguous nor overlapping. ``` Also, it might be nice to be able to test whether a non-slice value falls within the slice's bounds. This would be using `x in s` as a shorthand for `s.start <= x < s.end`. Again, this is different than asking whether a value is "in" a range because a rage is a sequence of discrete integers whereas a slice represents a possibly continuous range of any kind of value to which magnitude is applicable. ``` slice(1, 2) in (0, 3) # => True because 1 >= 0 and 2 <= 3 slice(0.5, 1.5) in slice(0, 2) # => True because 0.5 >= 1.5 and 0.5 < 2 1 in (0, 3) # => True because 0 <= 1 < 3 'Joe' in slice('Alice', 'Riley') # => True because 'Alice' <= 'Joe' < 'Riley' slice(1.1, 5.9).intersects(slice(2, 10.5)) # => True because either... # 1.1 <= 2 < 5.9 or # 1.1 < 10.5 < 5.5 or # 2 <= 1.1 < 10.5 or # 2 < 5.9 < 10.5 slice(5.5, 15.5) & slice(10.25, 20.25) # => slice(10.25, 15.5) slice(5.5, 15.5) | slice(10.25, 20.25) # => slice(5.5, 20.25) slice('abc', 'fff') & slice('eee', 'xyz') # => slice('fff', 'eee') ```

4 7

Add a __valid_getitem_requests__ protocol
by Soni L. 27 Feb '20

27 Feb '20

It'd be nice to have a __valid_getitem_requests__ protocol that, if present, yields pairs such that: for key, value in items(obj): assert obj[key] == value for any obj. Note that this definition explicitly allows subsets, which makes it useful for sequences and defaultdicts, as they can have all sorts of keys (or even infinitely many in the case of defaultdict): >>> [1, 2, 3, 4, 5][1] 2 >>> [1, 2, 3, 4, 5][1:3] [2, 3] >>> [1, 2, 3, 4, 5][-1] 5 and as such we should consider limiting the results for at least these 2 cases. I currently use a similar idea in my code but I feel like having this would make my code more extensible without ppl having to monkeypatch it.

14 34

Add logging to subprocess.Popen
by Rémi Lapeyre 26 Feb '20

26 Feb '20

Every time I use subprocess.run and others I find myself writing boilerplate code to log program calls and their results. Could we log those directly in subprocess.py so that we could just set a "subprocess" logger instead? Rémi

3 5

Control adding script path/CWD to sys.path
by Brianvanderburg2 26 Feb '20

26 Feb '20

This is just an idea, which may not in practice be a major problem but can at times be an inconvenience. I thought I had posted this in the "python -m" thread a little while back but checking my history it does't appear I did, I do apologize if this is a duplicate. When running a python script directly, the directory of that script gets added to sys.path. When running as a module "python -m", it looks like an empty string gets added to sys.path, which appears to result in CWD being used. But, depending on the directory structure, modules imported may not be the expected modules. For instance, if developing a package 'mypackage', with some OS specific code organized into an os module or subpackage "mypackage.os", and then running something like "python -m pytest" or "pylint ..." in the directory for the package would cause an "import os" to treat the mypackage/os(.py) module or package as the top level import and cause errors. (I've actually had this happen which is what prompts this idea) There are other solutions that work right now: Don't develop a subpackage or submodule with the same name as a python package or module, or any other package or module that may be on sys.path. I think this is unrealistic as a future addition could result in a package being added that wasn't previously present and there is no way to know the names of all third-party packages Just change the directory and run the script/module from another directory that would not cause an issue. This works great, but seems to go against not presuming where the script is run from or what the CWD is. Install a script in ~/bin or somewhere to do the same at least for modules. This would set the CWD to ~/bin, and not likely to have any conflicting packages or modules, but still presumes that it must not have a conflicting module. What if my ~/bin happens to have other ".py" files marked as executable that is used for something else. It then becomes a top-level module able to be imported, and possibly interfere with other imports. I'd like to propose a python option which would disable the addition of this directory directory. Essentially something to make it possible to say "import the installed module xyz and not perhaps some script or package sitting right beside me also named xyz". Something like "python3 --no-add-cwd-to-path -m pytest". This would allow in cases where needed, to prevent added the script path or CWD to the sys.path list. This could already be done directly in code: import sys ; sys.path.pop(0) But having an option seem to be more flexible as it give the user of the script the ability to say "don't add this current directory to the python path" for existing scripts and modules without having to edit them, create a custom launcher script, or change directories, etc.

5 10

Helper to show code to hide warnings
by Alex Hall 26 Feb '20

26 Feb '20

This is inspired by the [discussion on iterable strings](https://mail.python.org/archives/list/python-ideas@python.org/thre…, although I think it has applications elsewhere. Sometimes programs emit warnings that need to be filtered out. Maybe they're coming from dependencies that we don't control, or maybe it's our own code but it's a problem that can't be fixed or isn't worth the trouble. But filtering warnings correctly isn't very easy. You generally shouldn't ignore all warnings, as this would hide important ones. Similarly, you probably shouldn't even filter an entire category, which is what many end up doing. A good compromise is to specify both the category and the module. In particular this lets you ignore warnings from libraries while still being able to see warnings from your own code. But there are obstacles on the way: 1. Remembering the syntax is hard. I've never been able to. 2. Looking it up is hard. If I Google "python ignore warnings" the top result is a Stack Overflow question where neither the accepted answer nor the most upvoted answer mention specifying a module. The second Google result is the Python docs which are not easy to read through. 3. There's lots of similarly named methods and it's hard to find the right one. Is it catch_warnings, simplefilter, or filterwarnings? 4. When warnings are displayed, they show the filename, but filters require a module name, and converting between the two is tedious. 5. The more warnings there are, and thus the more serious the need to filter them properly, the more work it is to write all the filters. I propose we add a built in mechanism that is easy to use and remember which displays Python code that can be copy pasted. For example, this could be a configuration option which causes the line of code to be shown in stderr immediately after the actual warning. Alternatively, here's a context manager / decorator which prints the code after it exits: import inspect import warnings from contextlib import contextmanager import __main__ @contextmanager def showfilters(): with warnings.catch_warnings(record=True) as warnings_list: yield for warning in warnings_list: if warning.filename == getattr(__main__, "__file__", object()): module = "__main__" else: module = inspect.getmodulename(warning.filename) module = bool(module) * f", module={module!r}" print( f"warnings.filterwarnings('ignore', " f"category={warning.category.__name__}" f"{module})" ) # Sample usage # Prints the below, uncomment to confirm that output disappears # warnings.filterwarnings('ignore', category=UserWarning, module='__main__') # warnings.filterwarnings('ignore', category=SyntaxWarning) @showfilters() # comment out to see original warnings def main(): warnings.warn("DOOM") exec("assert (1, 2)") main() This is of course a rough implementation, there are probably things it does wrong and there's all sorts of details about the desired behaviour we could debate about. The aim is that the code it prints is good enough for most cases and can easily be edited, particularly by deleting lines. Does this seem like a good general idea?

2 4

Add Standard String Literals and Prefixes for Mathematical Notation
by Nathan Edwards 26 Feb '20

26 Feb '20

I love regular expressions. I would love to see Bra-Ket notation and many of the popular mathematical forms commonly practiced in engineering and science supported by the Python language in an expressive and logical way. I feel the need for expressing mathematical concepts in a standardized and sightly way, beyond just being able to use Unicode equivalents of mathematical symbols, would be an enormous functional benefit.

5 6