Python-ideas January 2021

python-ideas@python.org

70 participants
48 discussions

by Alex Hall

When using `exec(string)`, the standard Python REPL, or `python -c` in a terminal, the executed source code is not stored anywhere and cannot be retrieved (except in the `-c` case by looking at `sys.argv`). This means that: - The source line doesn't show up in tracebacks, e.g: >>> exec("1/0") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> ZeroDivisionError: division by zero - Users cannot retrieve the source code using `inspect`, although many want to: - https://stackoverflow.com/questions/29850508/python-how-to-inspect-code-of-… - https://stackoverflow.com/questions/12072252/cant-get-source-code-for-a-met… - https://stackoverflow.com/questions/62160529/how-to-get-source-inside-exec-… - https://stackoverflow.com/questions/59244323/how-can-you-get-the-source-cod… - https://stackoverflow.com/questions/50822656/source-code-for-an-evaled-dyna… - https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-… - Libraries such as traceback formatters and debuggers which depend on source code can't work. This often manifests when people try out a library for the first time in the REPL and it doesn't seem to work. This leads to many similar looking bug reports which sometimes take a while to resolve. Probably many more people think these libraries don't work without ever opening an issue. - https://github.com/cknd/stackprinter/issues/34 - https://github.com/Qix-/better-exceptions/issues/17 - https://github.com/Qix-/better-exceptions/issues/68 - https://github.com/aroberge/friendly-traceback/issues/23 - https://github.com/gruns/icecream/issues/17 - https://github.com/cool-RR/PySnooper/issues/135 - https://github.com/cool-RR/PySnooper/issues/164 - https://github.com/alexmojaki/sorcery/issues/3 - https://github.com/alexmojaki/sorcery/issues/5 - https://github.com/alexmojaki/sorcery/issues/6 I suggest that: 1. `exec(string)` assigns a unique generated filename to the compiled code object and stores the source code in the `linecache` module under that filename, similar to what the IPython REPL does. I imagine there might be complaints about memory usage, security implications of revealing source code, compatibility, etc. If that's an issue this could be off by default with a flag to turn it on. 2. The REPL, `-c`, and other similar mechanisms (IDLE?) either follow the same strategy as `exec` or defer to it. 3. The standard exception hook falls back to `linecache` for filenames that don't exist. I don't know why it doesn't already. Here is a question on that topic: https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-…

3 months, 3 weeks

Make fnmatch.filter accept a tuple of patterns

by Andre Delfino

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

4 months

Allow syntax "func(arg=x if condition)"

by Peter O'Connor

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

4 months, 1 week

Dataclasses, keyword args, and inheritance

by George Leslie-Waksman

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman

4 months, 2 weeks

PEP 9999 (provisional): support for indexing with keyword arguments

by Stefano Borini

PEP for support for indexing with keyword arguments is now submitted as PR. https://github.com/python/peps/pull/1612 Thanks to everybody involved in the development of the PEP and the interesting discussion. All your contributions have been received and often added to the document. If the PEP is approved, I would like to attempt an implementation, but I am not particularly skilled in the python internals. If a core developer is willing to teach me the ropes (especially in the new parser, I barely understood the syntax of the old one, but have no idea about the new one) and review my code I can give it a try. I would not mind refreshing my C a bit. -- Kind regards, Stefano Borini

4 months, 3 weeks

A shortcut to load a JSON file into a dict : json.loadf

by The Nomadic Coder

Hi All, This is the first time I'm posting to this mailing group, so forgive me if I'm making any mistakes. So one of the most common ways to load json, is via a file. This is used extensively in data science and the lines. We often write something like :- with open(filename.json, "r") as f: my_dict = json.load(f) or my_dict = json.load(open("filename.json", "r")) Since this is sooooo common, why doesn't python have something like :- json.loadf("filename.json") Is there an obvious issue by defining this in the cpython? I don't whipping up a PR if it gains traction.

4 months, 3 weeks

Conditional with statements

by Jonathan Crall

I'm not sure if this has been asked / suggested before. I'm wondering if there is any interest in conditional or loop-based `with` statements. I think it could be done without a syntax change. ### Napkin proposal Context managers could define `__if__` or `__for__`, and if those dunder methods were defined, then the `with` statement would either behave like a conditional or a loop. If `__if__` was defined then ``` with Ctx(): print('hi') ``` would only print `hi` if `__if__` returned True. This doesn't require a syntax change. The `__for__` variant would likely need a minor syntax change. ``` with item in Ctx(): print(item) ``` The `__for__` method is a generator that generates arguments of a loop. The item will be printed as many times as there are items generated by `__for__`. ### Use Cases This would simplify usage of my Timerit and ubelt library. The timerit module defines ``timerit.Timerit``, which is an object that is iterable. It has an ``__iter__`` method that generates ``timerit.TimerTimer`` objects, which are context managers. >>> import math >>> from timerit import Timerit >>> for timer in Timerit(num=200, verbose=2): >>> with timer: >>> math.factorial(10000) The timer context manager measures how much time the body of it takes by "tic"-ing ``__enter__`` and "toc"-ing on ``__exit__``. The underlying object has access to the context manager, so it is able to read its measurement. These measurements are stored and then we compute some statistics on them. Notably the minimum, mean, and standard-deviation of grouped (batched) running times. Unfortunately the syntax is one line and one indent bulker than I would prefer. However, a more consice version of the synax is available. >>> import math >>> from timerit import Timerit >>> for _ in Timerit(num=200, verbose=2): >>> math.factorial(10000) In this case the measurement is made in the `__iter__` method ``Timerit`` object itself, which I believe contains slightly more overhead than the with-statement version. (I should test to determine if this is the case). In the case where it does make a difference, a cool syntax might look like: >>> import math >>> from timerit import Timerit >>> with timer in Timerit(num=200, verbose=2): >>> math.factorial(10000) The other case is that my ``ubelt.Cacher`` library. Currently it requires 4 lines of boilerplate syntax. >>> import ubelt as ub >>> # Defines a cache name and dependencies, note the use of `ub.hash_data`. >>> cacher = ub.Cacher('name', cfgstr=ub.hash_data('dependencies')) # boilerplate:1 >>> # Calling tryload will return your data on a hit and None on a miss >>> data = cacher.tryload() # boilerplate:2 >>> # Check if you need to recompute your data >>> if data is None: # boilerplate:3 >>> # Your code to recompute data goes here (this is not boilerplate). >>> data = 'mydata' >>> # Cache the computation result (pickle is used by default) >>> cacher.save(data) # boilerplate:4 But a conditional ``with`` syntax would reduce boilerplate to 3 lines. >>> import ubelt as ub >>> with ub.Cacher('name', cfgstr=ub.hash_data('dependencies')) as cacher: >>> data = 'mydata' >>> cacher.save(data) >>> data = cacher.data I'm sure there are a lot of viable syntax variations, but does the idea of a conditional or loop aware "with" statement seem like a reasonable language proposal? -- -Dr. Jon Crall (him)

5 months, 1 week

Add extend_const action to argparse

by python＠roganartu.com

Forgive me if this idea has been discussed before, I searched the mailing lists, the CPython repo, and the issue tracker and was unable to find anything. I have found myself a few times in a position where I have a repeated argument that uses the `append` action, along with some convenience arguments that append a specific const to that same dest (eg: `--filter-x` being made equivalent to `--filter x` via `append_const`). This is particularly useful in cli apps that expose some kind of powerful-but-verbose filtering capability, while also providing shorter aliases for common invocations. I'm sure there are other use cases, but this is the one I'm most familiar with. The natural extension to this filtering idea are convenience args that set two const values (eg: `--filter x --filter y` being equivalent to `--filter-x-y`), but there is no `extend_const` action to enable this. While this is possible (and rather straight forward) to add via a custom action, I feel like this should be a built-in action instead. `append` has `append_const`, it seems intuitive and reasonable to expect `extend` to have `extend_const` too (my anecdotal experience the first time I came across this need was that I simply tried using `extend_const` without checking the docs, assuming it already existed). Please see this gist for a working example that may help explain the idea and intended use case more clearly: https://gist.github.com/roganartu/7c2ec129d868ecda95acfbd655ef0ab2

5 months, 3 weeks

Make UTF-8 mode more accessible for Windows users.

by Inada Naoki

The fact that Python does not use UTF-8 as the default encoding when opening text files is an obstacle for many Windows users, especially beginners in programming. If you search for UnicodeDecodeError, you will see that many Windows users have encountered the problem. This list is only part of many search results. * https://qiita.com/Yuu94/items/9ffdfcb2c26d6b33792e * https://www.mikan-partners.com/archives/3212 * https://teratail.com/questions/268749 * https://github.com/neovim/pynvim/issues/443 * https://www.coder.work/article/1284080 * https://teratail.com/questions/271375 * https://qiita.com/shiroutosan/items/51358b24b0c3defc0f58 * https://github.com/jsvine/pdfplumber/issues/304 * https://ja.stackoverflow.com/questions/69281/73612 * https://trend-tracer.com/pip-error/ Looking at the errors, the following are the most common cases. * UnicodeDecodeError is raised when trying to open a text file written in UTF-8, such as JSON. * UnicodeEncodeError is raised when trying to save text data retrieved from the web, etc. * User run `pip install` and `setup.py` reads README.md or LICENSE file written in UTF-8 without `encoding="UTF-8"` Users can use UTF-8 mode to solve these problems. I wrote a section for UTF-8 mode in the "3. Using Python on Windows" document. https://docs.python.org/3/using/windows.html#utf-8-mode However, UTF-8 mode is still not very well known. How can we make UTF-8 mode more user-friendly? Right now, UTF-8 mode can be enabled using the `-Xutf8` option or the `PYTHONUTF8` environment variable. This is a hurdle for beginners. In particular, Jupyter users may not use the command line at all. Is it possible to enable UTF-8 mode in a configuration file like `pyvenv.cfg`? * User can enable UTF-8 mode per-install, and per-venv. * But difficult to write the setting file when Python is installed for system (not for user), or Windows Store Python * User can still enable UTF-8 mode in venv. But many beginners don't need venv. Is it possible to make it easier to configure? * Put a checkbox in the installer? * Provide a small tool to allow configuration after installation? * python3 -m utf8mode enable|disable? * Accessible only for CLI user * Add "Enable UTF-8 mode" and "Disable UTF-8 mode" to Start menu? Any ideas are welcome. -- Inada Naoki <songofacandy(a)gmail.com>

5 months, 3 weeks

List .index() Method

by Francis O'Hara Aidoo

Hello, I'm Francis, a beginner programmer from Ghana, West Africa. I was wondering why the list .index() method doesn't return negative indices as well as well as positive indices. Although ambiguity will make it difficult to implement, in certain cases, it would be useful if it could return the negative index. For instance, if one creates an if statement that checks whether an element is the last item in a list as follows: listy = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] if listy.index(10) == -1: print("Monty Python") I understand that the same effect can be achieved with the index notation - as in if listy[-1] == 10: print("Monty Python") - but the way that came naturally to me was to use the .index method rather than index notation, and it took a very long time for me to figure out why my code was not working(mostly because I'm a beginner). So do what you will, I guess.:)

5 months, 4 weeks

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas January 2021