Mailman 3 February 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum 01 Dec '22

01 Dec '22

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Bringing the print statement back
by Guido van Rossum 02 Mar '22

02 Mar '22

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow 17 Feb '22

17 Feb '22

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor 19 Jan '22

19 Jan '22

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

Barrier Object in asyncio lib
by Yves Duprat 10 May '21

10 May '21

Hi,the list, I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help. Yves

6 13

Top Level Await in Python like in Deno
by redradist＠gmail.com 15 Apr '21

15 Apr '21

Hi all, Deno is JavaScript runtime that has very nice feature like Top Level Await, I think it would be also nice to have such feature in Python, it will make using async/await more convenient What do you think ? Share your ideas lets discuss ...

5 8

Access to source code when using exec()
by Alex Hall 03 Apr '21

03 Apr '21

When using `exec(string)`, the standard Python REPL, or `python -c` in a terminal, the executed source code is not stored anywhere and cannot be retrieved (except in the `-c` case by looking at `sys.argv`). This means that: - The source line doesn't show up in tracebacks, e.g: >>> exec("1/0") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> ZeroDivisionError: division by zero - Users cannot retrieve the source code using `inspect`, although many want to: - https://stackoverflow.com/questions/29850508/python-how-to-inspect-code-of-… - https://stackoverflow.com/questions/12072252/cant-get-source-code-for-a-met… - https://stackoverflow.com/questions/62160529/how-to-get-source-inside-exec-… - https://stackoverflow.com/questions/59244323/how-can-you-get-the-source-cod… - https://stackoverflow.com/questions/50822656/source-code-for-an-evaled-dyna… - https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-… - Libraries such as traceback formatters and debuggers which depend on source code can't work. This often manifests when people try out a library for the first time in the REPL and it doesn't seem to work. This leads to many similar looking bug reports which sometimes take a while to resolve. Probably many more people think these libraries don't work without ever opening an issue. - https://github.com/cknd/stackprinter/issues/34 - https://github.com/Qix-/better-exceptions/issues/17 - https://github.com/Qix-/better-exceptions/issues/68 - https://github.com/aroberge/friendly-traceback/issues/23 - https://github.com/gruns/icecream/issues/17 - https://github.com/cool-RR/PySnooper/issues/135 - https://github.com/cool-RR/PySnooper/issues/164 - https://github.com/alexmojaki/sorcery/issues/3 - https://github.com/alexmojaki/sorcery/issues/5 - https://github.com/alexmojaki/sorcery/issues/6 I suggest that: 1. `exec(string)` assigns a unique generated filename to the compiled code object and stores the source code in the `linecache` module under that filename, similar to what the IPython REPL does. I imagine there might be complaints about memory usage, security implications of revealing source code, compatibility, etc. If that's an issue this could be off by default with a flag to turn it on. 2. The REPL, `-c`, and other similar mechanisms (IDLE?) either follow the same strategy as `exec` or defer to it. 3. The standard exception hook falls back to `linecache` for filenames that don't exist. I don't know why it doesn't already. Here is a question on that topic: https://stackoverflow.com/questions/50515651/why-does-the-python-linecache-…

2 1

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino 26 Mar '21

26 Mar '21

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman 11 Mar '21

11 Mar '21

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman

12 35

Add lazy import statement to protect legitimate circular imports from failing
by Matt del Valle 11 Mar '21

11 Mar '21

I have seen a lot of interest in this topic, and I think the sheer number of different lazy import implementations (generally using proxy objects for modules and then loading them when an attribute is accessed) should present pretty good evidence that this is a very desired feature, particularly for writers of library code seeking to optimize start-up times and similar. Unfortunately, all of these implementations have serious limitations in that often static code analysis tools cannot understand them, resulting in autocompletions and useful diagnostic errors being lost. Also, they are fundamentally incapable of dealing with 'from x import y' syntax, because this is *always *loaded eagerly by the Python interpreter. With the increasingly widespread adoption of type hints I think it is time to revive the proposal for lazy imports as an official language feature. It is very common to type hint the return type or argument type of a function as a type which cannot be imported at the top of the file inside the module namespace because it would trigger a circular import, and must therefore be imported inside the function. In these instances it is currently recommended to use the 'TYPE_CHECKING' value in the 'typing' module which is always 'False' at runtime, in order to be able to provide the return or argument type hint: some_module.py > from typing import TYPE_CHECKING > > if TYPE_CHECKING: > from module_that_depends_on_me import MyClass > > > def some_func() -> MyClass: > from module_that_depends_on_me import MyClass > > return MyClass() However, this is unnecessarily verbose and confusing to most users who are not familiar with this usage pattern. IDEs will also be unable to warn you that MyClass is undefined if you try to use it inside 'some_func' without importing it within 'some_func' first, because it appears to be defined even though it isn't. It also is fundamentally incapable of handling this other extremely common usage pattern: some_library/ __init__.py > __all__ = ["SomeClass", "AnotherClass"] > > from .extremely_expensive_module import SomeClass > from .another_extremely_expensive_module import AnotherClass some_module.py > from some_library import SomeClass Where a library author wants to make several classes in their library's public API available at package level in the __init__.py, but doesn't want to eagerly load all of them when many users will only ever use one in a given program. In this situation if I try to import SomeClass from some_library/__init__.py I will *always* trigger AnotherClass to be loaded as well, totally unnecessarily. My proposed syntax is pretty much the same as was suggested by Nicolas Cellier in 2017 here: https://mail.python.org/pipermail/python-ideas/2017-February/044894.html, but with the additional stipulation that 'from x import y' format imports should *also* be supported. The two new lazy import statements should therefore be: > lazy import some_module > from some_module lazy import SomeClass For the two examples given above the equivalent code should be. some_module.py > from module_that_depends_on_me lazy import MyClass # does not raise ImportError when module_that_depends_on_me tries to import me, because this import is deferred > > > def some_func() -> MyClass: > return MyClass() # MyClass is not imported until this line, in a function that isn't called until after module level imports have already been resolved some_library /__init__.py > __all__ = ["SomeClass", "AnotherClass"] > > from .extremely_expensive_module lazy import SomeClass # this import is deferred > from .another_extremely_expensive_module lazy import AnotherClass # this import is deferred some_module.py > from some_library import AnotherClass # this causes SomeClass to be imported from some_library/__init__.py which in turn causes the import statement on line 4 of some_library/__init__.py to finally happen for real. The import of AnotherClass on line 3 *still* hasn't actually happened, and may never happen if AnotherClass isn't imported (or lazy imported and then used) during the lifetime of this program. These are just some examples that I could think of off the top of my head. I am certain there are more. This would be insanely useful. I really hope this sparks some discussion :)

3 8

Arrow functions polyfill
by Paul Sokolovsky 11 Mar '21

11 Mar '21

Hello, On Fri, 12 Feb 2021 18:26:53 +1100 Chris Angelico <rosuav(a)gmail.com> wrote: > On Fri, Feb 12, 2021 Paul Sokolovsky <pmiscml(a)gmail.com> wrote: > > ... And on the 2nd thought, that won't work. The reason it works in > > JS is that it doesn't have tuples. In Python, "(a, b) => (1, 2)" > > means "compare a tuple for greater-or-equal". > > Should be safe actually - "=>" is not a valid comparison operator. To punish myself for making such stupid mistakes, I volunteered to implement PoC of that. And by long honorable tradition, improvements to Python get implemented in Python first. So with my "imphook" thingy https://pypi.org/project/imphook/, and with the hook module at the end, following works as expected: =========== $ cat example_arrow_func.py f = (a, b) => a + b print(f(1, 2)) res = ((a, b) => a + b)(3, 4) print(res) print(list(map((x) => x * 2, [1, 2, 3, 4]))) # Confirm there's no crashing on bare tuple at the tail of file. (1, 2) $ python3 -m imphook -i mod_arrow_func -m example_arrow_func 3 7 [2, 4, 6, 8] =========== The implementation was written a bit cowboyishly in 15 mins, maybe, just maybe, you can still crash it. (For example, it clearly doesn't support newlines in arrow param list): ===== mod_arrow_func.py ====== import sys import tokenize import imphook class TokBuf: def __init__(self): self.tokens = [] def append(self, t): self.tokens.append(t) def clear(self): self.tokens.clear() def empty(self): return not self.tokens def spool(self): yield from self.tokens self.clear() def xform(token_stream): tokbuf = TokBuf() for t in token_stream: if t[1] == "(": # We're interested only in the deepest parens. if not tokbuf.empty(): yield from tokbuf.spool() tokbuf.append(t) elif t[1] == ")": nt1 = next(token_stream) nt2 = next(token_stream) if nt1[1] == "=" and nt2[1] == ">": yield (tokenize.NAME, "lambda") yield from tokbuf.tokens[1:] tokbuf.clear() yield (tokenize.OP, ":") else: yield from tokbuf.tokens tokbuf.clear() yield t yield nt1 yield nt2 elif not tokbuf.empty(): tokbuf.append(t) else: yield t def hook(modname, filename): with open(filename, "rb") as f: # Fairly speaking, tokenizing just to convert back to string form # isn't too efficient, but CPython doesn't offer us a way to parse # token stream so far, so we have no choice. source = tokenize.untokenize(xform(tokenize.tokenize(f.readline))) mod = type(imphook)(modname) exec(source, vars(mod)) return mod imphook.add_import_hook(hook, (".py",)) =========== -- Best regards, Paul mailto:pmiscml@gmail.com

23 62

PEP 9999 (provisional): support for indexing with keyword arguments
by Stefano Borini 10 Mar '21

10 Mar '21

PEP for support for indexing with keyword arguments is now submitted as PR. https://github.com/python/peps/pull/1612 Thanks to everybody involved in the development of the PEP and the interesting discussion. All your contributions have been received and often added to the document. If the PEP is approved, I would like to attempt an implementation, but I am not particularly skilled in the python internals. If a core developer is willing to teach me the ropes (especially in the new parser, I barely understood the syntax of the old one, but have no idea about the new one) and review my code I can give it a try. I would not mind refreshing my C a bit. -- Kind regards, Stefano Borini

20 100

A shortcut to load a JSON file into a dict : json.loadf
by The Nomadic Coder 10 Mar '21

10 Mar '21

Hi All, This is the first time I'm posting to this mailing group, so forgive me if I'm making any mistakes. So one of the most common ways to load json, is via a file. This is used extensively in data science and the lines. We often write something like :- with open(filename.json, "r") as f: my_dict = json.load(f) or my_dict = json.load(open("filename.json", "r")) Since this is sooooo common, why doesn't python have something like :- json.loadf("filename.json") Is there an obvious issue by defining this in the cpython? I don't whipping up a PR if it gains traction.

22 67

SimpleNamespace vs object
by Sven R. Kunze 02 Mar '21

02 Mar '21

Starting this as a new thread for the interested reader: On 17.02.21 11:18, Chris Angelico wrote: > On Wed, Feb 17, 2021 at 9:11 PM Sven R. Kunze <srkunze(a)mail.de> wrote: >> Still think that "object()" should be writable since this seems like an >> arbitrary restriction (+SimpleNamespace is no builtin and at least I >> would use object() for fast PoCs or dirty hackery). But I guess there's >> been discussion around this already. >> [...] > Using SimpleNamespace is the best way to do this, and maybe there's a > good argument for making it a builtin (or at least giving it a shorter > name), but changing object would be problematic. I would be skeptical that either those ideas would help a lot. My reasoning is the following: Given the usual terminology in computer science and programming courses, we think in "objects". "Namespaces" on the other side are quite static, usual representing surroundings or environments which in turn can contain "objects". Sadly, the terminology-wise ideal solution is blocked by technical reasons. The thinking in the concepting use-case goes like "I need a blank object which I can do random things with" (yes, including attaching new attributes to it, maybe other stuff as well). The thinking "hey I need a (simple) namespace which I can attach attributes to" requires quite some thoughts especially when being in a more "concepting"/"trying out ideas" mood. In this phase, one does not simply know what one needs ("could be sentinel only", "could be data carrying", "could be signalling", "could something else") and things move quickly. So, just as an example when renaming/aliasing "types.SimpleNamespace" to "builtins.thing", my expectation would be that people could be horribly confused seeing "object" and "thing" being a builtin when used in a project simultaneously. This always leads to questions like "what to use now?" As a conclusion I would say let's leave it like it is since it's a minor issue. Best Sven

19 44

Have virtual environments led to neglect of the actual environment?
by Random832 01 Mar '21

01 Mar '21

I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking: Christian Heimes wrote: > Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install. I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly? I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason. How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?

13 20

argparse: mutually inclusive arguments
by Paul Korir 26 Feb '21

26 Feb '21

Hi All, I've been using the argparse library for a long time and one use case that repeatedly shows us is the need to have two arguments appear together i.e either both appear or none of them appear. I'll refer to these as *mutually inclusive* in contrast to the existing *mutual exclusive* feature present. I might be wrong but there seems to be no way to enforce this. Of course it is possible to have two arguments that are *required* but it's not always the case that they will be needed. A more relevant example would be a situation in which you have a set of mutually exclusive arguments, one of which is a group of arguments. For example, consider a scenario in which a user is required to enter multiple related files that share the same file root (excluding the extension). The program would then infer the various files (co-files) by searching for the extensions. The first option would have the user only need to provide the file name root. However, the user may also provide the paths to the various co-files. In this case, the co-files must occur together but must occur mutually-exclusively from the file root option. Another example would be a case in which activating one set of options implies the need for other options to be simultaneously activated and for which defaults may not be easily discerned. Quite often the author has to perform pythonic gymnastics to catch such invalid options, which might be better handled by the argparse library. In evaluating the need for this I've come across the following responses: https://stackoverflow.com/questions/15935092/creating-mutually-inclusive-po… https://bugs.python.org/issue11588 https://bugs.python.org/issue23298 https://stackoverflow.com/questions/19414060/argparse-required-argument-y-i… https://www.debugcn.com/en/article/57242307.html https://picocli.info/#_mutually_dependent_options I perceive there is a compelling use-case to be addressed here despite an earlier request being turned down. With kind regards, *Paul*

5 5

Deprecate/change the behaviour of ~bool
by Soni L. 24 Feb '21

24 Feb '21

Currently ~False is -1 and ~True is -2. Would be nicer if ~bool was the same as not bool. Hopefully nobody actually relies on this but nevertheless, it would be a backwards-incompatible change so the whole deprecation warnings and whatnot would be required. In particular, this is nice for xnor operator: a ^~ b. This currently works on ints, but not on bools, while most other operators, including xor, do successfully work on bools.

8 15

Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?
by Random832 24 Feb '21

24 Feb '21

I accidentally removed Python-Ideas from the recipient list while posting this reply. On Wed, Feb 24, 2021, at 09:16, Random832 wrote: > On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote: > > Separate directories don't prevent clashes and system breakage. But they > > provide an easy way to *recover* from a broken system. > > I think it could be turned into a way to prevent them by A) having > site-packages always take precedence over dist-packages [i believe this > is already the case] in normal usage and B) providing an option to the > interpreter, used by system scripts, to exclude site-packages entirely > from the path. > > Basically, site-packages would effectively be layered on top of "Lib + > dist-packages" in a similar way to how a venv is layered on top of the > main python installation - the inverse of the suggestion someone else > in the thread made for the system python to be a venv. This wouldn't > *exactly* be a venv because it wouldn't imply the other things that > entering a venv does such as "python" [and script names such as pip] > being an alias for the correct version of python, but it would provide > the same kind of one-way isolation, whereby the "system environment" > can influence the "normal environment" and not vice-versa, in the same > way that packages installed in the main installation affect a venv > [unless system-site-packages is disabled] but the venv obviously has no > effect on the main installation.

1 0

Re: [Python-Dev] Re: Have virtual environments led to neglect of the actual environment?
by Henk-Jaap Wagenaar 24 Feb '21

24 Feb '21

On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou <antoine(a)python.org> wrote: > On Tue, 23 Feb 2021 20:29:52 -0500 > Jonathan Goble <jcgoble3(a)gmail.com> wrote: > > > > I can't speak for distributors or maintainers [1], but I can speak for > > myself as a user. I run Debian testing (currently bullseye as that is > > preparing for release) as my daily OS on my personal laptop, used for > > personal matters and school assignments (I'm a university computer > > science student in my senior year). > > > > I don't use the system Python for anything of my own, whether it's a > > school assignment or a personal project, precisely because I don't > > want to risk screwing something up. Rather, I maintain a clone/fork of > > the official CPython GitHub repo, and periodically build from source > > and `make altinstall` into `~/.local/`. > > For the record, instead of building Python by hand, you could use a > distribution such as Anaconda or the community-maintained conda-forge. > > Regards > > Antoine. > > I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.

2 1

Alternate lambda syntax
by J. Pic 23 Feb '21

23 Feb '21

Hi all, Lambdas can be defined as such: w = lambda: [12] x = lambda y: len(y) I'd like to propose the following: w = (): [12] x = (y): len(y) Or even another contraction for when there are no arguments: w =: [12] This would also be consistent with the other proposal on anonymous functions for defaults: https://mail.python.org/pipermail/python-list/2021-February/900795.html -- ∞

26 66

Multi-line lambda as last argument
by Aleksey Ploskov 23 Feb '21

23 Feb '21

Hello, Multi-line lambda is element of many languages, but they are not needed in the form in which they are made in many languages. А good use would be to use the lambda as the last argument of the function. Syntax: > def func(lambda1): > lambda1() > > func(): > print(1) > Can be used for create dsl. For example html: > html: > head() > > body(): > p(): > "Hello world" > a(href="python.org"): > "Python link" > Or for create new syntax constructions: > def unless(condition, _lambda): > if not(condition): > _lambda() > > unless(0): > print('yes') >

1 0

Re: Inadequate error reporting during function call setup stage
by Peter Otten 23 Feb '21

23 Feb '21

On 21/02/2021 23:06, Terry Reedy wrote: > On 2/21/2021 12:04 PM, Paul Sokolovsky wrote: > >> Traceback (most recent call last): >> File "pseudoc_tool.py", line 91, in <module> >> first_class_function_value(func, **pass_params) >> TypeError: print() got an unexpected keyword argument 'noann' > > This is not typical behavior in current Python (3.8+). The way I understand it's not about print(), it's about disambiguating multiple functions with the same name. Example: PS > type .\ambiguous_names.py import random def do_stuff(): pass f = do_stuff def do_stuff(a, b): pass g = do_stuff random.choice([f, g])(42) PS > py .\ambiguous_names.py Traceback (most recent call last): File "...\ambiguous_names.py", line 13, in <module> random.choice([f, g])(42) TypeError: do_stuff() missing 1 required positional argument: 'b' The traceback gives no clue which of the two do_stuff() functions caused the error, you have to check both implementations. If that is a comman problem one might consider including module name and co_firstlineno in the message, or at least adding the relevant do_stuff() function to the exception's args.

6 18

Inadequate error reporting during function call setup stage
by Paul Sokolovsky 21 Feb '21

21 Feb '21

Hello, Here's example: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() got an unexpected keyword argument 'noann' Ok, which "print" do you mean, dear CPython? I have a dozen of print functions (mostly methods) around. So I now need to reduce to printf debugging to find which one is actually meant. And what if that happens in 3rd-party library installed in /usr/lib? On a production system? The most interesting fact is that despite functions has always been first-class values and 30-years language history, such issues are still there. I'm fixing this right away in my Pycopy dialect, because while it's minimalist and I tremble over each half-byte added, but it's Python first of all, and Python means good error reporting. Well, actually I don't fix it right away, but work around, put putting source loc in TypeError's error message. And here's an idea of what the real fix should be: The fact that CPython (and Pycopy) sets up function's frame "outside" the function execution is an implementation detail. The model of execution should be: there's a CALL_FUNCTION/CALL_FUNCTION_KW opcode, and after it executes, the context switches to the target function, and that's *when* argument checking happens. That would give the desired behavior that the last entry in traceback is the actual function which was called and which failed parameter checks. Again, it's an implementation detail if some project doesn't do it like that, and performs param checking before actually pointing VM at the new bytecode function (from which location traceback will be automatically constructed). If it does that, it just still need to append an additional traceback entry pointing to the beginning of the target function. I now implemented that too, and now everything makes sense: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> File "../xforms.py", line 25, in print TypeError: unexpected keyword argument 'noann' -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

Syntax for duplicate types in seq[...]
by wyz23x2＠163.com 20 Feb '21

20 Feb '21

Imagine specifying multiple same types. Awful. ``` from typing import Tuple def time() -> Tuple[int, int, int, int, int, int, int, str]: """ Return (year, month, day, hour, minute, second, millisecond, timezone name). ... ``` So I suggest a new syntax, ``Seq[type * times]``, to solve this problem: ``` from typing import Tuple def time() -> Tuple[int * 7, str]: ... ```

3 2

Conditional with statements
by Jonathan Crall 16 Feb '21

16 Feb '21

I'm not sure if this has been asked / suggested before. I'm wondering if there is any interest in conditional or loop-based `with` statements. I think it could be done without a syntax change. ### Napkin proposal Context managers could define `__if__` or `__for__`, and if those dunder methods were defined, then the `with` statement would either behave like a conditional or a loop. If `__if__` was defined then ``` with Ctx(): print('hi') ``` would only print `hi` if `__if__` returned True. This doesn't require a syntax change. The `__for__` variant would likely need a minor syntax change. ``` with item in Ctx(): print(item) ``` The `__for__` method is a generator that generates arguments of a loop. The item will be printed as many times as there are items generated by `__for__`. ### Use Cases This would simplify usage of my Timerit and ubelt library. The timerit module defines ``timerit.Timerit``, which is an object that is iterable. It has an ``__iter__`` method that generates ``timerit.TimerTimer`` objects, which are context managers. >>> import math >>> from timerit import Timerit >>> for timer in Timerit(num=200, verbose=2): >>> with timer: >>> math.factorial(10000) The timer context manager measures how much time the body of it takes by "tic"-ing ``__enter__`` and "toc"-ing on ``__exit__``. The underlying object has access to the context manager, so it is able to read its measurement. These measurements are stored and then we compute some statistics on them. Notably the minimum, mean, and standard-deviation of grouped (batched) running times. Unfortunately the syntax is one line and one indent bulker than I would prefer. However, a more consice version of the synax is available. >>> import math >>> from timerit import Timerit >>> for _ in Timerit(num=200, verbose=2): >>> math.factorial(10000) In this case the measurement is made in the `__iter__` method ``Timerit`` object itself, which I believe contains slightly more overhead than the with-statement version. (I should test to determine if this is the case). In the case where it does make a difference, a cool syntax might look like: >>> import math >>> from timerit import Timerit >>> with timer in Timerit(num=200, verbose=2): >>> math.factorial(10000) The other case is that my ``ubelt.Cacher`` library. Currently it requires 4 lines of boilerplate syntax. >>> import ubelt as ub >>> # Defines a cache name and dependencies, note the use of `ub.hash_data`. >>> cacher = ub.Cacher('name', cfgstr=ub.hash_data('dependencies')) # boilerplate:1 >>> # Calling tryload will return your data on a hit and None on a miss >>> data = cacher.tryload() # boilerplate:2 >>> # Check if you need to recompute your data >>> if data is None: # boilerplate:3 >>> # Your code to recompute data goes here (this is not boilerplate). >>> data = 'mydata' >>> # Cache the computation result (pickle is used by default) >>> cacher.save(data) # boilerplate:4 But a conditional ``with`` syntax would reduce boilerplate to 3 lines. >>> import ubelt as ub >>> with ub.Cacher('name', cfgstr=ub.hash_data('dependencies')) as cacher: >>> data = 'mydata' >>> cacher.save(data) >>> data = cacher.data I'm sure there are a lot of viable syntax variations, but does the idea of a conditional or loop aware "with" statement seem like a reasonable language proposal? -- -Dr. Jon Crall (him)

13 33

Adding PathLiteral class and existing object instance to pathlib
by mwmajewsk 15 Feb '21

15 Feb '21

Hi All, The pathlib library is really helpful when working with paths, but creating an instance of Path class could be easier. There was a discussion about solving this with p-strings here https://mail.python.org/archives/list/python-ideas@python.org/thread/MNKRQF… But that would alter python syntax. Easier, but just as nice solution could be adding the following snippet to pathlib: class PathLiteral(): def __truediv__(self, other): return Path(other) path_literal = PathLiteral() This would allow the following snippet to work: >>> from pathlib import path_literal as p >>> p/"foo" PosixPath('foo') This idea is inspired by this tweet https://twitter.com/bitecode_dev/status/1217429169396731906 The core of the idea is that it does not require syntax changes, only need a small instance of an object, that can simplify and encourage use of pathlib. When someone needs to rewrite a lots of string-paths to pathlib's Path you only need to add that import line, and prepend strings with "p/". Cheers, MM

3 4

Pattern-Matching for functions
by Abdulla Al Kathiri 14 Feb '21

14 Feb '21

In some functional programming languages, they can pattern match a function based on its parameters. Similar to function overloading. Will Python Pattern Matching do something similar to functions? If calling a function doesn't match any of the cases and catch-all case is not present, then some sort of new or exisiting Exception needs to be raised. Type annotations can be used (for the parameters, they may be redundant, but for the return, they can be useful). The Return annotation comes right after the guard. See theortical example below. class Point: def __init__(self, pos): self.pos = pos def write(self, msg): return f"writing {msg}" match def f: case (float(x) | int(x), float(y) | int(y)) if x > 0 -> int: return round(x + y) # f(3.4, 7) => 10 case (int(x) | float(x)): return round(2 * x) # f(8) => 16 case (0): return 0 # f(0) => 0 case (0, 0): return "Origin" # f(0, 0) => "Origin" case ([int(x) | float(x), int(y) | float(y)], int(z) | float(z)): return x + y + z # f((5, 2.5), 2.5) => 10 case ( (Point(pos=(int(x), int(y)))) as point, *_, message=str(msg), **_, ) if x > 0 and y > 0: txt = point.write(msg) res = x + y return res, txt # point = Point([1, 9]) # f(point, message="Hello") => 10, "writing Hello" case (*_, **_): return "Catch_All" ## which can be shortened to... case _: return "Catch_All" #f(1, 3, 4, 5) => “Catch_All"

1 0

Pattern Matching -- Removing Name Binding from Partial Matching
by Abdulla Al Kathiri 14 Feb '21

14 Feb '21

I will explain it in the following few lines of code.. name = "George" year = 2021 d = {"name": "Mike", "year": 1919} match d: case {"name": name, "year": 1917}: print("match 1 found”) # I want to remove binding "name" here from partial matching case {"year": year, "name": "Michael”}: print("match 2 found”) # I want to remove binding "year" here from partial matching. # Basically removing all name bindings after every partial/failed matching case _: print("match not found!”) print(f"{name = }, {year = }”) ### Output ###: match not found! name = 'Mike', year = 1919 But I want :var: 'name' to stay being the global name “George" and :var: 'year' being the global year 2021 if an exact matching is not found. Maybe it is done the way it is for speed optimization (overhead reduction or something), but what if I don't care about speed and I care about having no un-intentional side-effects? Can we do something like the following to solve this issue? match d, partial_binding=False: case … : ... case … : ... with partial_binding=True by default. Any other idea in how to prevent name binding due to partial matching from happening? Any previous discussions on this? Thanks, Abdulla

3 5

Dead simple import hooks - proposed solution
by Paul Sokolovsky 13 Feb '21

13 Feb '21

Hello, Some time ago I posted a query regarding how the simplest possible (while still practical to use) import hooking might look like. Since then, I've coded up my try: https://github.com/pfalcon/python-imphook . I'd be interested in hearing any criticism, in particular in regard to the claims of minimalism and being "dead simple". Below is the copy of README for your reference (documentation is still WIP). A few initial examples are available in the git repo above. imphook - Simple and clear import hooks for Python ================================================== The ``imphook`` module allows to easily define per file type import hooks, i.e. overload or extend import processing for particular file types, without affecting processing of other file types, and at the same time, while ensuring that new processing integrates as seamlessly as possible with normal Python import rules. Besides the Python-level API to install import hooks, the module also provides command-line interface to run an existing Python script or module with one or more import hooks preloaded (i.e. without modifying existing script source code). Some but not all things you can easily do using ``imphook`` (most of these require additional modules to do the heavy lifting, ``imphook`` just allows to plug it seamlessly into the Python import system): * Override importing of (all or some) .py files, to support new syntax or semantics in them. * Import files written using a DSL (domain-specific language) as if they were Python modules. E.g., config or data files. * Import modules written in other language(s), assuming you have an interpreter(s) for them. * Import binary files, e.g. Java or LLVM bytecode. ``imphook`` works both with new, lightweight legacy-free Python API, as promoted by the `Pycopy <https://github.com/pfalcon/pycopy>`_ Python dialect (the original source of the "easy import hooks" idea), and CPython (the older, reference Python implementation), and with other Python implementations which are CPython-compatible. Quick Start ----------- Make sure that you already installed ``imphook`` using:: pip3 install -U imphook Below is a complete example of an import hook module to load ``key = value`` style config files:: import imphook def hook(modname, filename): with open(filename) as f: # Create a module object which will be the result of import. mod = type(imphook)(modname) for l in f: k, v = [x.strip() for x in l.split("=", 1)] setattr(mod, k, v) return mod imphook.add_import_hook(hook, (".conf",)) Save this as the ``mod_conf.py`` file, and add the two following files to test it: ``example_settings.conf``:: var1 = 123 var2 = hello ``example_conf.py``:: import example_settings as settings print(settings.var1) print(settings.var2) Now run:: python3 -m imphook -i mod_conf example_conf.py As you can see, the ``example_conf.py`` is able to import ``example_settings.conf`` as if it were a normal Python module. Besides copy-pasting the above and other examples, you can also clone the Git repository of ``imphook``, which contains various ready-to-use examples:: git clone https://github.com/pfalcon/python-imphook API to install hooks and hook structure --------------------------------------- The API of the module consists of one function: `imphook.add_import_hook(hook, ext_tuple)`. *hook* is a name of hook function. *ext_tuple* is a tuple of file extensions the hook function should handle (the leading dot should be included). More often than not, you will want to handle just one extension, so don't forget to use the usual Python syntax with a trailing comma for 1-element tuple, e.g.: ``(".ext",)``. Python modules may not contain a dot (``"."``) in their names (they are used to separate subpackages), so the extension you register may contain multiple dots, e.g. ``".foo.bar"``, with filename ``my_module.foo.bar`` matching it. It is possible to call `imphook.add_import_hook(hook, ext_tuple)` multiple times to install multiple hooks. The hooks are installed in the stack-like fashion, the last installed will be called first. It is possible to install multiple hooks for the same file extension, and earlier installed hooks may still be called in this case, because a hook function may skip processing a particular file, and let other hooks to take a chance, with default processing happening if no hook handled the import. The signature and template of the actual hook function is:: def my_hook(modname, filename): # Return None if you don't want to handle `filename`. # Otherwise, load `filename`, create a Python module object, # with name `modname`, populate it based on the loaded file # contents, and return it. The *modname* parameter is a full module name of the module to import, in the usual dot-separated notation, e.g. ``my_module`` or ``pkg.subp.mod``. For relative imports originated from within a package, this name is already resolved to full absolute name. The *modname* should be used to create a module object with the given name. The *filename* parameter is a full pathname (with extension) of the file which hook should import. This filename is known to exist, so you may proceed to open it directly. You may skip processing this file by returning ``None`` from the hook, then other hooks may be tried, and default processing happens otherwise (e.g. ``.py`` files are loaded as usual, or ImportError raised for non-standard extensions). For package imports, the value of *filename* ends with ``/__init__.py``, and that is the way to distinguish module vs package imports. If the hook proceeds with the file, it should load it by whatever means suitable for the file type. File types which are not natively supported by Python would require installing and using other extension modules (beyond ``imphook``). After loading the file, the hook should create an empty Python module object which will be the result of the import. There are a few ways to do that: * The baseline is to call a module type as a constructor. To get a module type, just apply the usual ``type()`` function to an existing (imported) module. You'll definitely have ``imphook`` itself imported, which leads us to:: mod = type(imphook)(modname) The parameter to constructor is the name of module to create, as passed to the hook. * If the above looks too magic for you, you can import symbolic name for module type from the ``types`` module:: from types import ModuleType mod = ModuleType(modname) * Finally, you may use the ``imp`` module, which may be as well the clearest (to the newbie) way of doing it:: import imp mod = imp.new_module(modname) But mind that the ``imp`` module is considered deprecated. Of the choices above, the first is the most efficient - no need to import additional modules, and it's just one line. And once you saw and were explained what it does, it shouldn't be a problem to remember and recognize it later. Once the module object is created as discussed above, you should populate it. A way to do that is by using ``setattr()`` builtin to set a particular attribute of a module to a particular value. Attributes usually represent variables with data values, but may be also functions and classes. Finally, you just return the populated module object. In case you want to perform custom transformation on the Python source, the process is usually somewhat different, where you transform a representation of the source, and then execute it in the context of a new module, which causes it to be populated. -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

Gauging appetite for a divergent alternative to pattern matching
by Ambient Nuance 11 Feb '21

11 Feb '21

Pattern matching re-surfaced on my radar today, along with the impression of a conclusion being just around the corner. Although this has seemed an ongoing state at times, a certain finality would come from a steering council decision. At the same time, it seems the polarisation that has developed around this feature has solidified more than dissipated. So I am prompted to throw my hat in the ring of language design. However, I must acknowledge that it is very late in the development cycle of this feature. Given the sheer volume of previous discussion, I am uncertain if there is an appetite for further deliberation at this point. This post is an invitation to help clarify such uncertainty. What I can bring to the table is a syntax design that is structurally close to PEP 634, but diverges significantly in detail and priorities. As to the content of this design, it seems best for the moment to give only a brief outline of significant differences. Laying out specific syntactic proposals seems a sure way to ignite a discussion which some/many may not wish to have. - on a general level, it is a superset of both pattern matching and type hints. - maybe this interferes with static typing efforts? Maybe it's a more flexible common ground? - prioritises consistency with current syntax and 'style' of Python. - to do so, introduces a new dunder method (single argument, boolean return). - design began from interrogating arbitrary Python objects, rather than tuple unpacking. - more 'object' oriented, but requires explicit/verbose notation of 'store' variables. - centres around a new 'template' class, which any object can be 'queried' against. - these can be created, customised and re-used outside of 'match' statements. Since I didn't set myself much of a conceptual ceiling, several optional extensions came about. - a guard template to control whether the 'match' block is entered (e.g. integers only, then different ranges). - opt-in fall-through on a per-branch basis. - an additional dunder method to improve brevity of repeated templates. - a new operator to utilise the new dunder method outside of 'match' contexts. - a separate but related 'assign' block statement (shortcut for assignment to a common variable). - outside of a 'match' context, using multiple template dispatch for functions/methods. Whatever the outcomes of this post and PEP 643, I will continue with my development of prototype emulations for current Python syntax. Discussion in this thread will help me decide on the degree of pace and detail to put into coding and communication efforts, e.g. PyPI and blog vs. full-on PEP. Below is a description of my own context in relation to pattern matching. It is included primarily to elaborate on the motivation and timing of this post. --- Shortly after the original publication of PEP 622, I began a loosely-bound train of thought on syntactic sugar in Python. My particular focus was on the clarity and consistency of future possibilities. Cue many iterations and multiple restarts from scratch. I ended up with syntax designs ranging from generalised container merging to graph data structures. In order to more thoroughly iron out shortcomings, I have been on-and-off developing emulations using current Python syntax. Meanwhile, pattern matching has continued on its own steam in the tuple-unpacking direction. After reading a variety of discussions, my takeaway impression was that in most cases the design was not particularly open to substantive changes. This seemed true at both a minute and broad scale. So I stepped away from these discussions to take the time to flesh out my own thoughts. For myself, the most exciting idea within PEP 622 is switch-based control flow driven by custom predicates. In the current proposal, such predicates are optimised for a combination of structural unpacking and class inheritance. A kind of inline abbreviated-expression dispatch. The escape hatch that I found to diversify predicate-space was to delegate all 'matching' details to a new class of objects. This enables the control of predicate logic outside of a 'match' statement, sidestepping the need to compress such details into an inline format. On a final note... As others do, I have my own likes and dislikes about the current form of pattern matching. Regardless of these opinions, I want to make clear that it is not my intention with this post to make a divisive affront to previous and ongoing efforts. If others wish to construe it as such, or take it to such a place, please instead spend your efforts building a reflective and much-less-binary dialogue around language design, rather than language designers.

3 2

Make filter() and map() return function if iterable is not passed
by Henry Harutyunyan 11 Feb '21

11 Feb '21

One of my students gave me an interesting question about why `filter` and `map` do not return a reusable function in case the iterable is not passed. With the current design ``` In [1]: a = filter(lambda x: x > 60) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-1-ad0178d4fce0> in <module> ----> 1 a = filter(lambda x: x > 60) TypeError: filter expected 2 arguments, got 1 ``` this is fine as long as you need to use that filter for once. But if you want to reuse it you either need to create the iterator every time specifying the filter function, or save the function in a var and create the filter with that. I'm wondering why can't we create custom filters and mappers that are reusable. The result will look something like this: ``` >>> pass_filter = filter(lambda x: x > 60) >>> foo = [42, 56, 67, 87] >>> for i in pass_filter(foo): >>> print(i) 67 86 >>> bar = {45, 94, 65, 3} >>> for i in pass_filter(bar): >>> print(i) 65 94 ``` Are there any drawbacks or limitations to this? Is is worth working on?

3 2

Re: Make UTF-8 mode more accessible for Windows users.
by Eryk Sun 11 Feb '21

11 Feb '21

On 2/6/21, Christopher Barker <pythonchb(a)gmail.com> wrote: > On Sat, Feb 6, 2021 at 11:47 AM Eryk Sun <eryksun(a)gmail.com> wrote: > >> Relative to the installation, "python.cfg" should only be found in the >> same directory as the base executable, not its parent directory. > > OK, my mistake — I thought that was already the case with pyvenv.cfg. > Though I don’t get why it matters. Chiefly, I don't want to overload "pyvenv.cfg" with new behavior that's unrelated to virtual environments. I also dislike the way this file is found. If the parent directory is "C:\Program Files", then I'm not worried about finding "C:\Program Files\pyvenv.cfg" when the interpreter tries to open it. But this pattern is not safe in general when installed to an arbitrary directory, or with a portable distribution. The presence of a "._pth" file (Windows only) beside the DLL or executable bypasses the search for "pyvenv.cfg", among other things. The embedded distribution includes a ._pth that locks it down. This is another reason to use a different file to configure defaults for -X settings such as "utf8", a file that's guaranteed to always be read. >> Add an option in the installed "python.cfg" to set the name of the >> organization and application. > > That would work for, e.g. pyinstaller (which I hope already ignores these > kinds if configuration. > > But not for, e.g. web applications that expect to use virtual environments > to isolate themselves. The idea to use the profile data directories %ProgramData% and %LocalAppData% was for symmetry with how this could be supported in POSIX, which doesn't use the application directory as Windows does. The application "python.cfg" (in the directory of the executable, including a virtual environment) can support a setting to isolate it from system and user "python.cfg" files.

5 31

Re: Generic NamedTuples
by Peter Ludemann 10 Feb '21

10 Feb '21

I use dataclass (with frozen=True) instead of NamedTuple - and I can do mixins. https://docs.python.org/3/library/dataclasses.html

2 2

Re: Very strange problem in python
by nate lust 10 Feb '21

10 Feb '21

This is not really a support mailing list, it is intended for ideas about modifying the python language or its libraries. However after a quick look, the line tt = arr2 is not creating a copy of array2. It creates a new name bound to the same underlying object, an alias if you will. You will need to use some sort of copy function (such as np.copy) if you want an independent array. If you are interested in learning more, I suggest you look into how python implements variables. The FAQ ( https://docs.python.org/3/faq/programming.html#why-did-changing-list-y-also…) has some info on this. There are other sources (Fluent Python book comes to mind) that go into this in more detail. -- Nate Lust, PhD. Astrophysics Dept. Princeton University

1 0

Very strange problem in python
by Priya Singh 10 Feb '21

10 Feb '21

def closset_mtach(check, arr, itr): id_min = [] for ii in range(itr): id_min_tmp = np.argmin( abs(arr - check) ) id_min.append(id_min_tmp) arr[id_min_tmp] = float('-inf') id_min = np.array(id_min) return id_min def get_CS_id(arr1, arr2, tol, itr): tt = arr2 for ii in range(5,6): id_com = np.where( abs(arr1[ii] - tt) < tol )[0] if np.size(id_com) >= itr: mm = closset_mtach(arr1[ii], tt[id_com], itr) id_ttt = np.array(id_com)[mm] #print(id_ttt) print(arr2[id_ttt]) tt[id_ttt] = 0.0 ########-9999.9 ###float('-inf') print(arr2[id_ttt]) This is the two routines I am using to compare the two float arrays and getting the matching in them up to some tolerance. The very strange thing which is happening here is you see variable arr2 at ID id_ttt before changing the variable tt at the same ID (i.e. id_ttt]) is showing some value but once I am printing its value after assigning tt[id_ttt] = 0.0, arr2[id_ttt] is also showing 0 value however, I have not touched arr2 anywhere as far as changing its values are concerned. Can any one please help me.

1 0

warning when importing a file from directoty
by Jonatan 09 Feb '21

09 Feb '21

I see many people who have their project something like this: ``` directory/ some_module.py main.py ``` and in main.py they do ``` from some_module import some_function ``` while they have a library named `some_module` that is installed from pip, and they can't figure out why Python says that there is no such function. I would recommend adding a warning like "ImportWarning: The same module exists in this folder and site-packages a well." so people won't be confused why this function doesn't exist, it happens a lot for beginners.

2 2

class(obj) could return obj.__class__
by Hans Ginzel 08 Feb '21

08 Feb '21

Please, consider class(obj) to return obj.__class__ consistenly with dir(), vars(), repr(), str(),… >>> class c: pass >>> o = c() >>> o.__class__ <class '__main__.c'> >>> class(o) File "<stdin>", line 1 class(o) ^ SyntaxError: invalid syntax H.

3 2

Generic NamedTuples
by Ben Avrahami 08 Feb '21

08 Feb '21

In both python 3.7 and 3.8, typing.NamedTuple had some strange behaviour regarding multiple inheritance. This behaviour was solved in 3.9 (bpo-36517), but this fix also barred me from a feature very close to my heart: the generic named tuple ``` T = TypeVar('T') class LLNode(NamedTuple, Generic[T]): value: T next: Optional[LLNode[T]] ``` bpo-36517 was committed because any additional features of other classes would be missing from the NamedTuple class at runtime. But Generic has no features at runtime, and thus this snippet actually works as expected in python 3.7/8. But in 3.9, it raises a TypeError, as bpo-36517 completely barred any other bases in a typing.NameTuple subclass. I propose that instead of wholesale barring multiple inheritance in NamedTuple, NamedTuple will allow some pre-defined bases alongside it. It will only allow do-nothing mixins with no runtime features (I can only think of `Generic` Aliases atm, but perhaps others exist). As before, these bases will be actually omitted from the class's bases at runtime. Any thoughts?

2 1

What's up with assignment expression and tuples?
by Paul Sokolovsky 08 Feb '21

08 Feb '21

Hello, Everyone knows how hard to find a compelling usecase for the assignment expression operator (":=", colloquially "walrus operator"). https://www.python.org/dev/peps/pep-0572/ examples never felt compelling and we all remember the split it caused. I finally found a usecase where *not* using assignment expression is *much* worse than using it. I'm working on SSA (Static Single Assignment, https://en.wikipedia.org/wiki/Static_single_assignment_form) conversion of Python programs, where there's a need to "join" dataflow of values from different control flow paths using a special function (Phi function). This "joining" itself creates a new variable, and of course, the original variable was used in an expression. We've got assignment in expression, assignment expression operator to the rescue! With it, a simple loop like: ---- a = 0 while a < 5: a += 1 ---- becomes: ---- a0 = 0 while (a1 := phi(a0, a2)) < 5: a2 = a1 + 1 ---- So far, so good. But semantics of Phi function is parallel assignment. No problem with Python either, "a, b = b, c" is exactly parallel assignment. So, let's try example with 2 variables: ---- a = 0 b = 10 while a < 5: a += 1 b += 1 ---- becomes: ---- a0 = 0 b0 = 10 while ((a1, b1) := phi([a0, a2], [b0, b2]))[0] < 5: a2 = a1 + 1 b2 = b1 + 1 ---- But oops: > SyntaxError: cannot use assignment expressions with tuple To reproduce in the REPL: ---- >>> ((a, b) := (1, 2)) File "<stdin>", line 1 SyntaxError: cannot use assignment expressions with tuple ---- Why this accidental syntactic gap? Why assignment statement can do parallel assignment with a tuple on LHS, and assignment operator suddenly can't? Why the adhoc naming and conceptual shift on the AST level, when PEP572 explicitly talks about *assignment operator*, but corresponding node on the AST level is called NamedExpr? Why look at assignment expression as "name of expression" instead of assignment expression per se? It's of course not a problem to recast: NamedExpr(expr target, expr value) to NamedExpr(expr* target, expr value) in the ASDL (and it works out of the box), the point is that it should have been ExprAssign from the start (folloing the AugAssign and AnnAssign tradition). -- Best regards, Paul mailto:pmiscml@gmail.com

2 4

Re: [Python-Dev] Re: What's up with assignment expression and tuples?
by Paul Sokolovsky 07 Feb '21

07 Feb '21

Hello, Thanks for the reply. On Fri, 5 Feb 2021 13:32:25 -0500 Terry Reedy <tjreedy(a)udel.edu> wrote: > On 2/5/2021 2:51 AM, Paul Sokolovsky wrote: > > > https://www.python.org/dev/peps/pep-0572/ > ... > > >>>> ((a, b) := (1, 2)) > > File "<stdin>", line 1 > > SyntaxError: cannot use assignment expressions with tuple > > ---- > > > > Why this accidental syntactic gap? > > As should be clear from reading "Differences between assignment > expressions and assignment statements", this 'gap' in entirely > intentional, not accidental. *All* elaborations of 'name := > expression' are listed and rejected as outside the scope of the > proposal, which was to keep one reference to the expression value for > later use. At least some of these elaborations were suggested and > rejected during the voluminous discussion. And looking back now, that seems like intentionally added accidental gap in the language (https://en.wikipedia.org/wiki/Accidental_gap). Similar to artificially limiting decorator syntax, which was already un-limited. But seems, there're no "lessons learned", and there's now need to wait a decade again before fixing that? > The principal a.e. use in conditional expressions is testing for > non-nullness. Your > > > while ((a1, b1) := phi([a0, a2], [b0, b2]))[0] < 5: > > a2 = a1 + 1 > > b2 = b1 + 1 > > is an unusual and very specific use. Well, many people were thinking (and I bet still think) that ":=" itself is very unusual case. But if it's in, why not make it consistent with the assignment statement and unleash the full power of it? > You want to have your tuple > (for subscripting for testing) and eat it too (by unpacking). That's good characterization, thanks. And Python syntax alone would allow to do that, if not extra-syntactical limitations put on top of it. > One can instead separate unpacking from testing a couple of ways > > while (tup := phi([a0, a2], [b0, b2]))[0] < 5: > a2, b2 = tup > a2 = a1 + 1 > b2 = b1 + 1 > > > while True: > a1, b1 = phi([a0, a2], [b0, b2]) > if a1 >= 5: break > a2 = a1 + 1 > b2 = b1 + 1 Right, but I started my original email with "I finally found a usecase where *not* using assignment expression is *much* worse than using it." Both conversions above apply additional disturbances to the original program, beyond pure SSA conversion, which is quite a disturbance on its own. I was really excited that Python 3.7+ would be *the* language which would allow to express SSA conversion faithfully on the source form of the high-level language (usually SSA is applied to low-level assembly-like intermediate representation). But oops, accidental gap, and despite all the Python advanceness, still need to apply workarounds as with other mundane languages. [] -- Best regards, Paul mailto:pmiscml@gmail.com

4 9

Add extend_const action to argparse
by python＠roganartu.com 07 Feb '21

07 Feb '21

Forgive me if this idea has been discussed before, I searched the mailing lists, the CPython repo, and the issue tracker and was unable to find anything. I have found myself a few times in a position where I have a repeated argument that uses the `append` action, along with some convenience arguments that append a specific const to that same dest (eg: `--filter-x` being made equivalent to `--filter x` via `append_const`). This is particularly useful in cli apps that expose some kind of powerful-but-verbose filtering capability, while also providing shorter aliases for common invocations. I'm sure there are other use cases, but this is the one I'm most familiar with. The natural extension to this filtering idea are convenience args that set two const values (eg: `--filter x --filter y` being equivalent to `--filter-x-y`), but there is no `extend_const` action to enable this. While this is possible (and rather straight forward) to add via a custom action, I feel like this should be a built-in action instead. `append` has `append_const`, it seems intuitive and reasonable to expect `extend` to have `extend_const` too (my anecdotal experience the first time I came across this need was that I simply tried using `extend_const` without checking the docs, assuming it already existed). Please see this gist for a working example that may help explain the idea and intended use case more clearly: https://gist.github.com/roganartu/7c2ec129d868ecda95acfbd655ef0ab2

3 2

Type hinting **kwargs
by Paul Bryan 07 Feb '21

07 Feb '21

I'm encountering a situation where it would be far cleaner to use **kwargs (multiple functions taking the same set of parameters); I'd like to have type hints, and avoid repeating multiple function definitions, explicitly enumerating all parameters for each function definition. Therefore, I'd like to have some meaningful type hint for **kwargs. First, I thought TypedDict would be a great fit, until I found this passage in PEP 589: > These features were left out from this PEP, but they are potential > extensions to be added in the future: > ... > TypedDict can't be used for specifying the type of a **kwargs > argument. This would allow restricting the allowed keyword arguments > and their types. According to PEP 484, using a TypedDict type as the > type of **kwargs means that the TypedDict is valid as the value of > arbitrary keyword arguments, but it doesn't restrict which keyword > arguments should be allowed. The syntax **kwargs: Expand[T] has been > proposed for this [11]. > ... > [11] https://github.com/python/mypy/issues/4441 In the mypy issue discussion, there seems to be two issues: 1. Ambiguity of whether the type hint applies to the **kwargs dict vs. each value in the dict. I think this could be a red herring; kwargs presents as a dict and I think TypedDict can unambiguously represent the structure of that dict. I don't see how it should be different than any other dict parameter being defined as a function parameter. 2. A question of whether one needs to assume that for a TypedDict, the type validator should assume total=False. I think what keys are required should be explicitly determined by the TypedDict (required keys inferred from total). I suspect the issue could be that mypy may not be able to determine this during static type analysis? In summary, **kwargs is a dict; TypedDict can specify its structure; static type checker limitations should not necessarily restrict its use in this way. Therefore, I propose a short PEP be drafted to codify the validity of using TypedDict to specify the type of a **kwargs argument. Paul

2 1

PEP for experimental/provisional features
by sascha.schlemmer＠me.com 07 Feb '21

07 Feb '21

Reading the latest discussion about PEP 622/634-636 on the Python Dev mailing list, I've noticed there appears to be confusion about what `__future__` imports are supposed to be used for. Multiple people in that thread (and probably elsewhere in one of the many many threads about these PEPs) have suggested including pattern matching as a "provisional" or "experimental" feature that is enabled by an import from `__future__`. I don't want to discuss whether `__future__` is the correct place for this however (and I also don't want to discuss PEP 622/634-636 specifically). Instead I'd like to propose a `__provisional__` namespace (or whatever color you favor for your bike shed) for new language features/ standard library modules that might undergo breaking changes over a couple of subsequent python releases. This is similar to how many applications start in a version V0.1 and undergo changes until they mature and reach V1.0. From the user perspective this would allow more "adventurous" users to play with the latest features (with the clear warning attached that they might have to adapt their own code over subsequent python releases) while anyone else can just ignore the new feature for now. From the perspective of the language designers the advantage I see is that they can get a first version of a feature out to real users to collect feedback on whether the feature is used as they imagined it being used and change/iterate the design if need be. It is simply unrealistic to get something like `asyncio` or PEP 622/634-636 perfect and feature complete on the first try or otherwise wait indefinitely until something is ready. Some thoughts what `__provisional__` SHOULD and SHOULD NOT be used for: - `__provisional__` SHOULD be useable at the users own discretion (e.g. no use of provisional features in the standard library etc. ) - `__provisional__` SHOULD be a place for features with a vast design space that need to "ripen" as use cases are explored (`asyncio` would have been the poster child candidate for `__provisional__`) - features SHOULD only be placed in `__provisional__` if there is a clear commitment to the feature (the general feature is accepted by the SC before being included in `__provisional__`). `__provisional__` SHOULD NOT be a place where contentious PEPs are thrown into willy-nilly and postpone the decision of whether to accept a PEP or not. Otherwise users would not only have to deal with changes to a feature (they obviously showed interest in using) but also with complete removal/deprecation. - `__provisional__` SHOULD only be used for features that are additive to the language and don't change existing code (which is what imho. what `__future__` is for). - `__provisional__` SHOULD have a clearly defined process (timelines etc.) Alternative ideas that have been mentioned (in threads about PEP 622/634-636 and elsewhere) and why I personally don't think they are sufficient: - "just throw together a library that parses strings and uses `eval`, put it on pypi to see how that goes": For features that require new syntax this just doesn't work imho. Using `pep622_lib.match("put your match statement here")` is a very poor substitute for trying this out due to the influence design choices such as keywords, indentation etc. have on usability. Examples for this are syntax highlighting, autocomplete/autoindent/autoformat and linting which you just don't get this way. - various degrees of: "solve the issue of contention over new languages features once and for all by introducing macros/preprocessors to the language": Against this I have just my personal plea "please don't turn python into lisp/c++".

2 2

Make UTF-8 mode more accessible for Windows users.
by Inada Naoki 06 Feb '21

06 Feb '21

The fact that Python does not use UTF-8 as the default encoding when opening text files is an obstacle for many Windows users, especially beginners in programming. If you search for UnicodeDecodeError, you will see that many Windows users have encountered the problem. This list is only part of many search results. * https://qiita.com/Yuu94/items/9ffdfcb2c26d6b33792e * https://www.mikan-partners.com/archives/3212 * https://teratail.com/questions/268749 * https://github.com/neovim/pynvim/issues/443 * https://www.coder.work/article/1284080 * https://teratail.com/questions/271375 * https://qiita.com/shiroutosan/items/51358b24b0c3defc0f58 * https://github.com/jsvine/pdfplumber/issues/304 * https://ja.stackoverflow.com/questions/69281/73612 * https://trend-tracer.com/pip-error/ Looking at the errors, the following are the most common cases. * UnicodeDecodeError is raised when trying to open a text file written in UTF-8, such as JSON. * UnicodeEncodeError is raised when trying to save text data retrieved from the web, etc. * User run `pip install` and `setup.py` reads README.md or LICENSE file written in UTF-8 without `encoding="UTF-8"` Users can use UTF-8 mode to solve these problems. I wrote a section for UTF-8 mode in the "3. Using Python on Windows" document. https://docs.python.org/3/using/windows.html#utf-8-mode However, UTF-8 mode is still not very well known. How can we make UTF-8 mode more user-friendly? Right now, UTF-8 mode can be enabled using the `-Xutf8` option or the `PYTHONUTF8` environment variable. This is a hurdle for beginners. In particular, Jupyter users may not use the command line at all. Is it possible to enable UTF-8 mode in a configuration file like `pyvenv.cfg`? * User can enable UTF-8 mode per-install, and per-venv. * But difficult to write the setting file when Python is installed for system (not for user), or Windows Store Python * User can still enable UTF-8 mode in venv. But many beginners don't need venv. Is it possible to make it easier to configure? * Put a checkbox in the installer? * Provide a small tool to allow configuration after installation? * python3 -m utf8mode enable|disable? * Accessible only for CLI user * Add "Enable UTF-8 mode" and "Disable UTF-8 mode" to Start menu? Any ideas are welcome. -- Inada Naoki <songofacandy(a)gmail.com>

6 31

Re: [Python-Dev] Re: What's up with assignment expression and tuples?
by Paul Sokolovsky 06 Feb '21

06 Feb '21

Hello, On Sat, 6 Feb 2021 10:46:54 +0200 Serhiy Storchaka <storchaka(a)gmail.com> wrote: > 05.02.21 09:51, Paul Sokolovsky пише: > > a0 = 0 > > b0 = 10 > > while ((a1, b1) := phi([a0, a2], [b0, b2]))[0] < 5: > > a2 = a1 + 1 > > b2 = b1 + 1 > > Such code quickly becomes unreadable. Especially if in real code > function has additional arguments and names are longer that 2-3 > characters. But at least it fully corresponds to the original statements and control flow of the program, that's the point! > The following code is not much larger but more clear and extensible: > > a0 = 0 > b0 = 10 > while True: > a1, b1 := phi([a0, a2], [b0, b2])) > if b1 >= 5: > break > a2 = a1 + 1 > b2 = b1 + 1 As I mentioned in other replies, that's what I've been doing. But that does *NOT* correspond to the original program, or SSA conversion of it. Instead, it's SSA + random munging. And when debugging SSA, you may argue what's worse: to look at 3-stories phi's, or to look at the code structure which doesn't correspond to the input code. But that's not the point. I now in https://bugs.python.org/issue43143 gave more down to earth example: For as long as you agree that following is acceptable: func(a := val) , then I'd say it really doesn't make much sense to argue that following should not be acceptable: min((b, a) := (a, b)) -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

Add a couple of options to open()'s mode parameter to deal with common text encodings
by Ben Rudiak-Gould 04 Feb '21

04 Feb '21

There's a long ongoing thread with the subject "Make UTF-8 mode more accessible for Windows users." There are two obvious problems with UTF-8 mode. First, it applies to entire installations, or at least entire running scripts, including all imported libraries no matter who wrote them, etc., making it a blunt instrument. Second, the problem on Windows isn't that Python happens to use the wrong default encoding, it's that multiple encodings coexist, and you really do have to think each time you en/decode something about which encoding you ought to use. UTF-8 mode doesn't solve that, it just changes the default. It seems as though most of those commenting in the other thread don't actually use Python on Windows. I do, and I can say it's a royal pain to have to write open(path, encoding='utf-8') all the time. If you could write open(path, 'r', 'utf-8'), that would be slightly better, but the third parameter is buffering, not encoding, and open(path, 'r', -1, 'utf-8') is not very readable. UTF-8 mode is somehow worse, because you now have to decide between writing open(path), and having your script be incompatible with non-UTF-8 Windows installations, or writing open(path, encoding='utf-8'), making your script more compatible but making UTF-8 mode pointless. There's a constant temptation to sacrifice portability for convenience - a temptation that Unix users are familiar with, since they omit encoding='utf-8' all the time. My proposal is to add a couple of single-character options to open()'s mode parameter. 'b' and 't' already exist, and the encoding parameter essentially selects subcategories of 't', but it's annoyingly verbose and so people often omit it. If '8' was equivalent to specifying encoding='UTF-8', and 'L' was equivalent to specifying encoding=(the real locale encoding, ignoring UTF-8 mode), that would go a long way toward making open more convenient in the common cases on Windows, and I bet it would encourage at least some of those developing on Unixy platforms to write more portable code also. For other encodings, you can still use 't' (or '') and the encoding parameter. Note that I am not suggesting that 'L' be equivalent to PEP 597's encoding='locale', because that's specified to behave the same as encoding=None, except that it suppresses the warning. I think that's a terrible idea, because it means that open's behavior still depends on the global UTF-8 mode even if you specify the encoding explicitly. This is really a criticism of PEP 597 and not a part of this proposal as such. I think UTF-8 mode was a bad idea (just like a global "binary mode" that interpreted every mode='r' as mode='rb' would have been), and it should be ignored wherever possible. In particular, encoding='locale' should ignore UTF-8 mode. Then 'L' could and should mean encoding='locale'. Obviously the names '8' and 'L' are debatable. 'L' could be argued to be unnecessary if there's a simple way to achieve the same thing with the encoding parameter (which currently there isn't).

6 7