Mailman 3 November 2020 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Bringing the print statement back
by Guido van Rossum March 2, 2022

March 2, 2022

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using … [View More]dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…> [View Less]

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

PEP 9999 (provisional): support for indexing with keyword arguments
by Stefano Borini March 10, 2021

March 10, 2021

PEP for support for indexing with keyword arguments is now submitted as PR. https://github.com/python/peps/pull/1612 Thanks to everybody involved in the development of the PEP and the interesting discussion. All your contributions have been received and often added to the document. If the PEP is approved, I would like to attempt an implementation, but I am not particularly skilled in the python internals. If a core developer is willing to teach me the ropes (especially in the new parser, I … [View More]

20 100

A shortcut to load a JSON file into a dict : json.loadf
by The Nomadic Coder March 10, 2021

March 10, 2021

Hi All, This is the first time I'm posting to this mailing group, so forgive me if I'm making any mistakes. So one of the most common ways to load json, is via a file. This is used extensively in data science and the lines. We often write something like :- with open(filename.json, "r") as f: my_dict = json.load(f) or my_dict = json.load(open("filename.json", "r")) Since this is sooooo common, why doesn't python have something like :- json.loadf("filename.json") Is there an obvious … [View More]

22 67

Add extend_const action to argparse
by python＠roganartu.com Feb. 8, 2021

Feb. 8, 2021

Forgive me if this idea has been discussed before, I searched the mailing lists, the CPython repo, and the issue tracker and was unable to find anything. I have found myself a few times in a position where I have a repeated argument that uses the `append` action, along with some convenience arguments that append a specific const to that same dest (eg: `--filter-x` being made equivalent to `--filter x` via `append_const`). This is particularly useful in cli apps that expose some kind of … [View More]

3 2

Re: [Python-Dev] reversed enumerate
by Steven D'Aprano Jan. 22, 2021

Jan. 22, 2021

Hi Ilya, I'm not sure that this mailing list (Python-Dev) is the right place for this discussion, I think that Python-Ideas (CCed) is the correct place. For the benefit of Python-Ideas, I have left your entire post below, to establish context. [Ilya] > I needed reversed(enumerate(x: list)) in my code, and have discovered > that it wound't work. This is disappointing because operation is well > defined. It isn't really well-defined, since enumerate can operate on infinite … [View More]iterators, and you cannot reverse an infinite stream. Consider: def values(): while True: yield random.random() a, b = reversed(enumerate(values()) What should the first pair of (a, b) be? However, having said that, I think that your idea is not unreasonable. `enumerate(it)` in the most general case isn't reversable, but if `it` is reversable and sized, there's no reason why `enumerate(it)` shouldn't be too. My personal opinion is that this is a fairly obvious and straightforward enhancement, one which (hopefully!) shouldn't require much, if any, debate. I don't think we need a new class for this, I think enhancing enumerate to be reversable if its underlying iterator is reversable makes good sense. But if you can show some concrete use-cases, especially one or two from the standard library, that would help your case. Or some other languages which offer this functionality as standard. On the other hand, I think that there is a fairly lightweight work around. Define a helper function: def countdown(n): while True: yield n n -= 1 then call it like this: # reversed(enumerate(seq)) zip(countdown(len(seq)-1), reversed(seq))) So it isn't terribly hard to work around this. But I agree that it would be nice if enumerate encapsulated this for the caller. One potentially serious question: what should `enumerate.__reversed__` do when given a starting value? reversed(enumerate('abc', 1)) Should that yield...? # treat the start value as a start value (1, 'c'), (0, 'b'), (-1, 'a') # treat the start value as an end value (3, 'c'), (2, 'b'), (1, 'a') Something else? My preference would be to treat the starting value as an ending value. Steven On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote: > Hi, > > I needed reversed(enumerate(x: list)) in my code, and have discovered that > it wound't work. This is disappointing because operation is well defined. > It is also well defined for str type, range, and - in principle, but not > yet in practice - on dictionary iterators - keys(), values(), items() as > dictionaries are ordered now. > It would also be well defined on any user type implementing __iter__, > __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors. > > That's plenty of usecases, therefore I guess it would be quite useful to > avoid hacky / inefficient solutions like described here: > https://code.activestate.com/lists/python-list/706205/. > > If deemed useful, I would be interested in implementing this, maybe > together with __reversed__ on dict keys, values, items. > > Best Regards, > -- > Ilya Kamen > > ----------- > p.s. > > *Sketch* of what I am proposing: > > class reversible_enumerate: > > def __init__(self, iterable): > self.iterable = iterable > self.ctr = 0 > > def __iter__(self): > for e in self.iterable: > yield self.ctr, e > self.ctr += 1 > > def __reversed__(self): > try: > ri = reversed(self.iterable) > except Exception as e: > raise Exception( > "enumerate can only be reversed if iterable to > enumerate can be reversed and has defined length." > ) from e > > try: > l = len(self.iterable) > except Exception as e: > raise Exception( > "enumerate can only be reversed if iterable to > enumerate can be reversed and has defined length." > ) from e > > indexes = range(l-1, -1, -1) > for i, e in zip(indexes, ri): > yield i, e > > for i, c in reversed(reversible_enumerate("Hello World")): > print(i, c) > > for i, c in reversed(reversible_enumerate([11, 22, 33])): > > print(i, c) > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD… > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

7 14

pathlib enhancements
by Todd Jan. 18, 2021

Jan. 18, 2021

I know enhancements to pathlib gets brought up occasionally, but it doesn't look like anyone has been willing to take the initiative and see things through to completion. I am willing to keep the ball rolling here and even implement these myself. I have some suggestions and I would like to discuss them. I don't think any of them are significant enough to require a pep. These can be split it into independent threads if anyone prefers. 1. copy The big one people keep bringing up that I … [View More]strongly agree on is a "copy" method. This is really the only common file manipulation task that currently isn't possible. You can make files, read them, move them, delete them, create directories, even do less common operations like change owners or create symlinks or hard links. A common objection is that pathlib doesn't work on multiple paths. But that isn't the case. There are a ton of methods that do that, including: * symlink_to * link_to * rename * replace * glob * rglob * iterdir * is_relative_to * relative_to * samefile I think this is really the only common file operation that someone would need to switch to a different module to do, and it seems pretty strange to me to be able to make symbolic or hard links to a file but not straight up copy one. 2. recursive remove This could be a "recursive" option to "rmdir" or a "rmtree" method (I prefer the option). The main reason for this is symmetry. It is possible to create a tree of folders (using "mkdir(parents=True)"), but once you do that you cannot remove it again in a straightforward way. 3. newLine for write_text This is the only relevant option that "Path.open" has but "Path.write_text" doesn't, and is a serious omission when dealing with multiple operating systems. 4. uid and gid You can get the owner and group name of a file (with the "owner" and "group" methods), but there is no easy way to get the corresponding number. 5. Stem with no suffixes The stem property only takes off the last suffix, but even in the example given ('my/library.tar.gz') it isn't really useful because the suffix has two parts ('.tar' and '.gz'). I suggest another property, probably called "rootstem" or "basestem", that takes off all the suffixes, using the same logic as the "suffixes" property. This is another symmetry issue: it is possible to extract all the suffixes, but not remove them. 6. with_suffixes Equivalent to with_suffix, but replacing all suffixes. Again, this is a symmetry issue. It is hard to manipulate all the suffixes right now, as the example show. You can add them or extract them, but not change them without doing several steps. 7. exist_ok for is_* methods Currently all the is_* methods (such as is_file) return False if the file doesn't exist or if it is a broken symlink. This can be dangerous, since it is not trivially easy to tell if you are dealing with the wrong type of file vs. a missing file. And it isn't obvious behavior just from the method name. I suggest adding an "exist_ok" argument to all of these, with the default being "True" for backwards-compatibility. This argument name is already in use elsewhere in pathlib. If this is False and the file is not present, a "FileNotFoundError" is raised. [View Less]

10 21

Add command-line option to unittest for enabling post-mortem debugging
by Dominik Vilsmeier Jan. 9, 2021

Jan. 9, 2021

Consider the following example: import unittest def foo(): for x in [1, 2, 'oops', 4]: print(x + 100) class TestFoo(unittest.TestCase): def test_foo(self): self.assertIs(foo(), None) if __name__ == '__main__': unittest.main() If we were calling `foo` directly we could enter post-mortem debugging via `python -m pdb test.py`. However since `foo` is wrapped in a test case, `unittest` eats the exception and thus prevents post-… [View More]

1 1

Python JIT Compilation Thoughts
by redradist＠gmail.com Jan. 6, 2021

Jan. 6, 2021

Hi all, I do not know maybe it was already discussed ... but the toolchain like LLVM is very mature and it can provide the simpler JIT compilation to machine code functionality and it will improve performance of the Python a lot !!

9 15

Add decorator_with_params function to functools module
by Yurii Karabas Dec. 24, 2020

Dec. 24, 2020

In the current python codebases, there are a lot of decorators with parameters, and their code in most cases contains a lot of code boilerplates. The purpose of this feature is to add `decorator_with_params` (I think the name can be changed to smth better) function to `functools` module. This function will allow using a decorated function as a simple decorator and decorator with parameter. I believe the real example will bring more context, for instance, `typing._tp_cache` function can be … [View More]rewritten from: ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` To ``` @decorator_with_params def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ``` And `_tp_cache` can be used as: ``` @_tp_cache(typed=True) def __getitem__(self, parameters): return self._getitem(self, parameters) ... @_tp_cache def __getitem__(self, parameters): return self._getitem(self, parameters) ``` Proposed implementation is quite simple (I used it in a lot of projects): ``` def decorator_with_params(deco): @wraps(deco) def decorator(func=None, /, *args, **kwargs): if func is not None: return deco(func, *args, **kwargs) def wrapper(f): return deco(f, *args, **kwargs) return wrapper return decorator ``` I believe it will be a great enhancement for the Python standard library and will save such important indentations. I have already opened an issue https://bugs.python.org/issue42455 (Sorry about that I didn't know that smth like python-ideas exists). [View Less]

3 5

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Add the brotli & zstandard compression algorithms as modules
by Omer Katz Dec. 14, 2020

Dec. 14, 2020

Hello everyone, We have many compression algorithms as standard modules, but we lack two important new algorithms: brotli and zstandard. Brotli is an important compression algorithm both for HTTP2 and for compressing/decompressing HTTP responses. ZStandard is another algorithm for compression that can easily beat gzip with a better compression ratio and less compression time. Both are now stable enough and are past the 1.0 mark. Let's include them in the standard library and, of course, make … [View More]

12 21

Experimental syntax proposal
by André Roberge Dec. 5, 2020

Dec. 5, 2020

# Experimental Syntax Proposal I would like to propose that Python adopts a modified process before introducing significant changes of its syntax. ## Preamble Given the following file: ```python # coding: experimental-syntax from experimental-syntax import fraction_literal from experimental-syntax import decimal_literal assert 1 /3F == Fraction(1, 3) assert 0.33D == Decimal('0.33') print("simple_test.py ran successfully.") ``` This is what happens when I run it, with the standard Python … [View More]interpreter: ``` $ python simple_test.py simple_test.py ran successfully. ``` In what follows, I use this as a concrete example for one of the three possible options mentioned. ## The problem Python evolves in many ways including with the addition of new modules in the standard library and with the introduction of new syntax. Before considering the addition of a module in the standard library, it is often suggested to have a version on Pypi so that users can experiment with it, which can lead to significant improvements. However, when it comes to proposed syntax changes, this is currently not possible to do, at least not in any standard way that would immediately be easily recognized as such by the wider community. ## Proposed solutions For those that agree that this is something that could be improved upon, I see at least three possible solutions. 1. Adoption of a simple convention to identify the use of non-standard syntax. This could take the form of a comment with a specific form introduced near the top of the file. This comment could be used as a search term, to identify modules or projects that make use of some custom syntax. 2. Addition of a special encoding module in the Python standard library that can be used to implement nonstandard syntax through source transformation. The short program written at the top provides an example of what this might look like. Note the comment at the top defining a special codec, which could also 3. Addition of a standard import hook in the standard library which could be used, like the special encoding example above, to implement syntactic changes. By doing searches in standard locations (Github, Gitlab, etc.) on, one could identify how many modules are making use of a given experimental syntax, giving an idea of the desirability of incorporating into Python. Imagine how different the discussion about the walrus operator would have been if people had had the opportunity to try it out on their own for a few months prior to a decision being made about adding it to Python. ## New Syntax: two approaches There are currently at least two ways in which one can write a Python program that can use some non standard syntax: * By using an import hook. This is the most powerful approach as it allows one to do changes either at the source level (prior to parsing), or at the AST level, or both. * By using a custom encoding. This only allows transformations at the source level. Both approaches currently require a two-step process. To use an import hook, one has first to load a function that sets up this import hook before importing the module that contains the experimental syntax. This makes it impossible to simply write ``` python module_with_new_syntax.py ``` and have it being executed. Import hooks, like the name implies, only apply to modules being imported - not to the first module executed by Python. With a custom encoding, it is however possible to first register a custom codec either via a site.py or usercustomize.py file. Once this is done, the above way of running a module with new syntax is possible, provided that the appropriate encoding declaration is included in the script. This is what I did with the example shown above. ### Current limitation of these two approaches Using an import hook for enabling new syntax does not work for code entered in the standard Python REPL, which limits the possibility of experimentation. While I have read that specifying the environment variable `PYTHONIOENCODING` enables the Python REPL to use a custom encoding, I have not been able to make it work on Windows and confirm that it is indeed possible to do so. However, I have been able to use a simple customized REPL that can make use of the above. Perhaps the standard REPL could be modified in a similar way. ## New suggested process Assuming one of the options mentioned above is adopted, before changes to the syntax are introduced in a Python version, they would be made available to be used either as an encoding variant or an import hook giving enough time for interested pythonistas to experiment with the new syntax, writing actual code and possibly trying out various alternatives. ## Proof of concept As a proof of concept shown at the beginning of this post, I have created two tiny modules that introduce new syntax that had been discussed many times on this list: 1. a new syntax for decimal literals 2. a new syntax for rationals Both modules have been uploaded separately to Pypi; I wanted to simulate what could happen if a proposal such as this one were adopted. To make use of these, you need to use the `experimental-syntax` codec found in my `ideas` package. To run the example as shown above, you first need to register the codec, which can be done by using either the `site.py` or `usercustomize.py` approach. I chose the latter, by setting the environment variable `PYTHONPATH` to be a path where the following `usercustomize.py` file is found. ```python from ideas import experimental_syntax_encoding print(f" --> {__file__} was executed") ``` Doing this on Windows, I found that it did not seem to work when using a virtual environment (I added the print statement to confirm that it was loaded). Here's a sample session using the code available currently on pypi. ``` C:\Users\andre\github\ideas>python simple_test.py --> C:\Users\andre\github\ideas\usercustomize.py was executed simple_test.py ran successfully. C:\Users\andre\github\ideas>python --> C:\Users\andre\github\ideas\usercustomize.py was executed Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:30:28) [MSC v.1926 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from ideas import experimental_syntax_encoding >>> from ideas import console >>> console.start() Configuration values for the console: transform_source: <function transform_source at 0x00A8AE38> -------------------------------------------------- Ideas Console version 0.0.19. [Python version: 3.8.4] ~>> import simple_test simple_test.py ran successfully. ~>> from experimental-syntax import decimal_literal ~>> 3.46D Decimal('3.46') ~>> from experimental-syntax import fraction_literal ~>> 2/3F Fraction(2, 3) ~>> ``` Installation: python -m pip install ideas # will also install token-utils python -m pip install decimal-literal python -m pip install fraction-literal More information about "ideas" can be found at https://aroberge.github.io/ideas/docs/html/ It has not (yet) been updated to include anything about the `experimental_syntax_encoding` module, which I just worked on today. ## Final thoughts This proposal is really about the idea of adopting a standard process of some sort that enables users to experiment with any proposed new syntax, rather than the specific "silly" examples I have chosen to illustrate it. André Roberge P.S. If anyone knows how to make a usercustomize file run in a virtual environment on Windows, please let me know. [View Less]

4 7

Making the for statement work better with nested functions
by Greg Ewing Dec. 5, 2020

Dec. 5, 2020

Starting a new thread for this as suggested by Guido. On 18/11/20 7:20 pm, Guido van Rossum wrote: > On Tue, Nov 17, 2020 at 22:12 Greg Ewing <greg.ewing(a)canterbury.ac.nz > <mailto:greg.ewing@canterbury.ac.nz>> wrote: > > If there's anything I would change, it would be to have the for > statement create a new binding on each iteration, so that capturing > it with a def or lambda inside the loop works as expected. I even > came up with a way … [View More]to do that while still allowing the last-bound > value to be seen afterwards, but the idea didn't gain any traction. > > I wonder if we should try that idea again? The problem may be backwards > compatibility — there’s always someone who depends on such a thing (even > if by accident, it would be a pain to debug). Maybe we could invent new > syntax to signal this behavior, though that might defeat the purpose. I think it can be made close enough to backwards compatible, although I'm not greatly opposed to new syntax if it's thought desirable. Essentially the idea is this: If the loop body contains a def or lambda that captures the loop variable, then on each iteration, a new binding is created, instead of changing an existing binding. Note that this is *not* the same as introducing a new scope. All the bindings share the same name, and from the perspective of code outside the nested function, nothing changes -- the name is still bound to the most recent value, and can still be accessed after the loop has finished. In CPython, it would be implemented by creating a new cell for each iteration, instead of changing the value of an existing cell. For other implementations, I don't know -- it would depend on how those implementations currently deal with nested functions. Do we need new syntax? My feeling is, probably not. I believe that the vast majority of existing code, perhaps all of it, would be unaffected by this change. All the existing workarounds for the current behaviour would continue to work. The only things that would break would be code that somehow relies on capturing the same instance of the loop variable every time, and I find it hard to imagine code that needs to do that. If we do want new syntax, my suggestion would be for new x in some_iterator: ... The downside of requiring special syntax is that we would still regularly get people asking why their lambdas in for statements don't do what they expect. We would have a better answer for them, but the questions wouldn't go away. -- Greg [View Less]

17 76

Making "Any" a builtin
by Abdulla Al Kathiri Dec. 2, 2020

Dec. 2, 2020

Instead of importing “Any" from the typing module, we can annotate our functions with “Any" right away without the extra step. What do you think? We have the builtin function “any” which some Python users could mistakingly use, but static type checkers should catch that.

8 14

Re: Making the for statement work better with nested functions
by Paul Sokolovsky Nov. 30, 2020

Nov. 30, 2020

Hello, On Mon, 30 Nov 2020 12:26:53 +0000 Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > On 30/11/2020 08:01, Paul Sokolovsky wrote: > > Hello, > > > > On Sun, 29 Nov 2020 18:53:57 -0800 > > Christopher Barker <pythonchb(a)gmail.com> wrote: > > > >> This is a massively less ambitious idea, and doesn't solve the > >> original problem, but: > >> > >> I'd like to see a scope for names that are "obviously ;-)" … [View More]meant > >> to be short lived. At the moment that's the for loop "counters" > >> and those created by context managers: > >> > >> for i in something: > >> # use i as usual > >> more_code > >> # now i is not defined > > What other language(s) implement such a scoping discipline? I know > > of none. > > > > On the other hand, block-scoped variables are implemented in: > > > > * C > > * C++ > > * Java > > * Rust > > * Lua > > * JavaScript (not by default, as opt-in) > > * Scheme > > * Common Lisp (as opt-in) > > * ML > > * Ocaml > > * Haskell > > * very long list of other languages... > > > > > > The aim of the block scoping proposal is to make Python *not worse* > > than these other languages, instead of adding funny workarounds > > again. > > > > [] > > > > CHB's proposal may or may not be a good idea. But please treat it on > its merits, Absolutely. I appreciate all the discussion and alternative ideas people put into this. I, on my side, may have a "strong opinion" that ideas I utter are "right", but that doesn't mean I'm not mistaken. These matters really require good consideration from as wider as possible Python community. > instead of saying, as you seem to be, "No other language > does this, so it's bound to be a bad idea which will make Python > worse". That's why I try to choose words carefully ;-). I said that if Python does it like many other languages, it definitely won't be worse than them. I said nothing what happens if it doesn't do it like that ;-). All in all, it's just an argument, as anyone else's. But I do think about those matters, I have to admit. And I'd hate yet another advanced Python programmer to leave for Haskell, Go, Rust, Julia, because, from their PoV, Python has a culture of applying workarounds instead of best-practice solutions. > With that attitude, there would never be any innovation in > language design. I'm first to subscribe under that. Recent example: Pattern Matching, a common mantra is "we did it like that because other languages do it like that" (example: https://mail.python.org/archives/list/python-dev@python.org/message/EWX3GWN…) My response: But did you consider Python's situation? Both "minor" issues, like lack of constants, which requires ugly workarounds, and overall audience of Python? Was the option to use sigils to mark capture (i.e. assignment) targets *really well* considered? They said yes. Ok. Works for me, and I tried to do as much noise as possible to draw other people's attention to that. > Best wishes Rob Cliffe [] -- Best regards, Paul mailto:pmiscml@gmail.com [View Less]

1 0

adding a timeit.Timer context manager and decorator to time code/functions
by sdementen＠gmail.com Nov. 30, 2020

Nov. 30, 2020

When optimizing code, I often need to timeit a part of code or a function. I am using then the following class ``` import logging import time from functools import wraps from typing import Optional _logger = logging.getLogger("Timer") class Timer: def __init__(self, description: Optional[str] = None, logger: Optional[logging.Logger] = None): self.description = "<anonymous code block>" if description is None else description self.logger = _logger if logger is None … [View More]

6 9

[PoC] Implementing block-scope for the "for" statement variables
by Paul Sokolovsky Nov. 29, 2020

Nov. 29, 2020

Hello, This is continuation of the "Making the for statement work better with nested functions" thread (https://mail.python.org/archives/list/python-ideas@python.org/thread/3ZKYV6…). I post this as a separate thread to draw more attention to and promote this way of prototyping Python changes. The changes are implemented using Python3 port of Python2's "compiler" module: https://github.com/pfalcon/python-compiler . For more information on background, see the parallel thread: https://mail.… [View More]python.org/archives/list/python-ideas@python.org/message/W2FRE… python-compiler module requires CPython3.5. So, please be prepared to enjoy again the golden age of Python3 (comparing to iron (perhaps, oil?) age of now). If you don't have that handy, the repository has a script which will build it for you in a couple of minutes (on Linux). The changes are implemented in the following branch: https://github.com/pfalcon/python-compiler/commits/for-block-scope . The top 3 commits are that, shouldn't be too hard to review (and have some additional info in the commit messages). As what's available there is only AST-to-bytecode compiler, not source parser, I couldn't easily implement "for let"/"for const" syntax, so went to switch *all* for statements to the block scope for this prototype. I also prototyped only "block scope" part, not the "constness" part. To describe high-level idea of the changes: The conceptual idea is that we can rewrite each definition of block-scoped variable to have a unique name, and then rewrite usages accordingly in all inner scopes (including inner functions). Direct implementation of that was deemed "too naive", based on the grounds that (again naive) implementation of it would require whole extra pass on AST, and patching it. Note that it's not necessarily that (what? bad), and if it ever comes to implementing in the C compiler, it may be a good idea to actually go that way first and see how actually "bad" it is. It may be possible to combine that pass with another pass (variable scoping AST pass, literally). The benefit of this approach is that it's "obviously correct" and doesn't introduce other overheads. Anyway, my point was to show that even doing it "better" is "not rocket science". I did it half-way still, because there's obvious rising conceptual complexity and rising storage requirements for intermediate info (what made me think that maybe "naive" approach described above isn't that bad). Here's example of it in action: $ cat example_for1.py def fun(): x = 123 for x in range(5): print(x) print("old x:", x) fun() $ ./python3.5-nopeephole -m compiler --dis example_for1.py 1 0 LOAD_CONST 0 (<code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>) 3 LOAD_CONST 1 ('fun') 6 MAKE_FUNCTION 0 9 STORE_NAME 0 (fun) 7 12 LOAD_NAME 0 (fun) 15 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 18 POP_TOP 19 LOAD_CONST 2 (None) 22 RETURN_VALUE Disassembly of <code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>: 2 0 LOAD_CONST 1 (123) 3 STORE_FAST 0 (x) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_GLOBAL 0 (range) 12 LOAD_CONST 2 (5) 15 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_FAST 1 (x.1) 4 25 LOAD_GLOBAL 1 (print) 28 LOAD_FAST 1 (x.1) 31 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 34 POP_TOP 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_GLOBAL 1 (print) 42 LOAD_CONST 3 ('old x:') 45 LOAD_FAST 0 (x) 48 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 51 POP_TOP 52 LOAD_CONST 0 (None) 55 RETURN_VALUE 0 1 2 3 4 old x: 123 -- Best regards, Paul mailto:pmiscml@gmail.com [View Less]

2 2

Variadic generics PEP draft
by Matthew Rahtz Nov. 29, 2020

Nov. 29, 2020

Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew

5 4

async types?
by Ben Avrahami Nov. 29, 2020

Nov. 29, 2020

Hi all, this is a general feeler for if this idea has any traction: All too often I see the following pattern in asyncio 3rd-party libs, either in their own source code or in the inusage: ``` inst = SomeClass() await inst.initialize() ``` What happens here is that, since coroutines cannot be used inside __init__ methods, the __init__ method only stores the parameters, and another, asynchronous method actually initializes the object. This led many 3rd-party libs to use factory methods to … [View More]create their class instances, which is unideal for both users and developers. We see this pattern in nearly any async 3rd-party lib I came across. To solve this, I propose a new metaclass: `AsyncType`. The core difference between AsyncType and type is that when called, AsyncType will produce a coroutine that asynchronously calls __new__ and __init__, and returns the instance, allowing simply for `instance = await SomeClass()`. In classes of `AsyncType`, the __new__ and __init__ methods must be async methods (returning coroutines). As an additional syntactic sugar, we could also have an `async class(base)` declaration that implicitly sets the class's metaclass to be AsyncType (or raise a TypeError if that is impossible). This proposal is obviously incomplete, and there are many open questions: what about __del__? How would one easily make an async ABC (without a metaclass conflict)? How would an async class easily inherit from a sync class (suppose sync class A implements __new__, and async class B(A) implements __init__)? Perhaps `AsyncType` should only make the __init__ method async and leave __new__ synchronous? I'd just like to get ahead of the (not unjustified) argument that constructors should be lightweight: 1. Python's __init__ is not strictly a constructor. It is, as the name implies, an initializer, that makes the type usable. Constructing the type without initializing it makes no sense and has no usages, so why seperate the operations? 2. We can see numerous examples of (syncronous) connect-on-initialize classes in the standard library, including ftplib, smtplib, and pretty much most *lib modules in the standard library "Internet Protocols and Support" doc page. 3. The "initialize in a separate method" pattern produces some strange-looking code where the class holds on to its initialization params even though it doesn't really need them after the separate methods. The classes should now also consider and safe-guard against cases where the separate method is called twice, or not at all. What does the community think about this idea? [View Less]

3 2

Move semantics
by 3mir33＠gmail.com Nov. 27, 2020

Nov. 27, 2020

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

15 38

Getting rid of FOR loops and simplifying cicular conviolutions with Circular Indexing
by Mathew M. Noel Nov. 26, 2020

Nov. 26, 2020

If circular indexing is used then instead of using a double FOR loop to go through a list M times we can iterate from 0 to M*N (where N is the length of the list) !!! Almost all Machine Learning (ML) algorithms iterate for some predefined epochs over a large data-set. So a double FOR loop is extremely common in ML. Using circular indexing gets rid of this extra FOR loop. If we have to iterate 2 times you can iterate using range(-n,n) but in most cases you need to iterate over 10 or more … [View More]epochs in ML. Most scientific applications of Python involve an outer FOR loop which progressively refines an approximation with an inner FOR loop by going through a list of items. So circular indexing is useful. In the following I discuss increasingly compelling reasons for adopting a circular indexing scheme in Python. Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc). If circular indexing is used then we don't need the arbitrary rule that -1 is the index of the last element. Since -1 is the same as n-1 automatically in modular arithmetic. A trivial objection: "why not use list_name[i%n] whenever we need this feature?" By the same token we could do away with negative indices and use -1%n for example when we need to index with -1! Its unclear why that people have an irrational preference for indices that lie to the left of 0 while strongly rejecting the idea of indices that lie to the right of n-1! Python does not raise a "index out of bound" exception for negative indices like other programming languages. If this negative indexing is a "feature" (although it allows some fatal errors to slip) then indices above n-1 can also be considered a feature! Are there any deep mathematical reasons for adopting circular convention? Circular convolution is a most important operation in a wide variety of scientific disciplines since the Discrete Fourier Transform (DFT) of the circular convolution of two signals is the product of the transforms. Because of the universal applicability of Fourier ideas in science and the close mathematical relationship between finite length and periodic sequences circular indexing is extensively used in signal processing and mathematics. We can extend the idea of circular indexing to multidimensional arrays. A 2D array can be folded into a cylinder for indexing. Further this cylinder can be folded into a toroid to reduce a triple FOR loop to a single FOR loop. A deep mathematical justification for cylindrical indexing of 2D and in general nD arrays is offered by the fact that n-dimensional DFT reduces n-dimensional circular convolution to element-wise multiplication. [View Less]

7 11

A python_include() for cgi programming
by warp_drive enterprise Nov. 26, 2020

Nov. 26, 2020

I have recently have the opportunity to work on a Python web application. I've found it beneficial to reduce duplicate code with the following code lines import textwrap from pathlib import Path exec(textwrap.dedent(Path('myfile.py').read_text())) Perhaps such code could be the basis for a python_include() statement similar to how includes, requires work in PHP. Thank you for any consideration.

5 4

Circular Indexing
by Mathew M. Noel Nov. 26, 2020

Nov. 26, 2020

Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example:… [View More]

10 12

Circular Indexing and FOR loop minimization
by Mathew M. Noel Nov. 25, 2020

Nov. 25, 2020

As discussed earlier on the post on 'Circular Indexing', considering interpreting indices that lie outside the range 0 to n-1 modulo n for list indexing. Thus element n corresponds to element 0, n+1 to element 1 and so on. This has two consequences: 1.) Makes the Python convention of interpreting index -1 as the last element more logical since -1 is the same as n-1 when taken modulo n. If -1 is the first element then to be logically consistent we have to interpret index n as the element 0 as … [View More]

3 2

Packaging Apps (Was Adding PyInstaller to the stdlib)
by Abdur-Rahmaan Janhangeer Nov. 25, 2020

Nov. 25, 2020

Greetings list, > Then I would suggest starting a new thread, Long overdue i guess. My view on the topic is like Mr Moore with the exception that i'm for native capabilities to be in the stdlib. Since Zipapp does not seem to go the C-extension route, i propose a new tool that turn projects into independent executables This solves the issue of Gui etc etc etc etc etc etc Mr Moore's mail of Nov 24, 2020 has ton of ideas and a lot of potential Kind Regards, Abdur-Rahmaan Janhangeer about … [View More]

1 0

A pypi-based app store solution for platform specific installation
by Ricky Teachey Nov. 25, 2020

Nov. 25, 2020

I was reading the pyinstaller thread and had this idea but didn't want to hijack. Maybe a wild idea, and very possible totally impractical or hopelessly complex, but: could the existing pypi infrastructure be leveraged into a set of platform-specific app stores? Maybe maybe we could make it as simple as a single line in a pyproject.toml file. Imagine if all the end user does is install the store, and clicks the link to the project app. The store takes care of the rest. The developer marks … [View More]

5 6

Adding PyInstaller to the standard library
by Abdur-Rahmaan Janhangeer Nov. 25, 2020

Nov. 25, 2020

Greetings list, What do you think of adding PyInstaller as an official part of CPython? Among the different native exports options, PyInstaller holds a nice track of clean delivery. Instead of the project battling for its survival as any other project, if the community finds it useful enough, the devs can get some peace of mind. The stereotyped idea about languages like Python is that they don't by default produce executables compared to let's say Go. Adding the native ability to produce … [View More]

25 119

JSON loading
by Stestagg Nov. 24, 2020

Nov. 24, 2020

I was thinking about the "Load JSON file as single line" thread from a bit back, and had an idea for a neat solution, albeit one that has potential stylistic issues. Idea: Create two new methods on pathlib.Path objects: Path.load(loader, **kwargs) and Path.dump(dumper, obj, **kwargs) Here, `loader` should be an object that implements a method `load(fileobj: BinaryIO, *args, **kwargs) -> object` and `dumper` should be an object that implements `dump(obj: object, fileobj: BinaryIO, *… [View More]

1 0

Matching TypedDicts and other values in JSON
by David Foster Nov. 24, 2020

Nov. 24, 2020

I am excited about the potential of the new PEP 634-636 "match" statement to match JSON data received by Python web applications. Often this JSON data is in the form of structured dictionaries (TypedDicts) containing Lists and other primitives (str, float, bool, None). PEP 634-636 already contain the ability to match all of those underlying data types except for TypedDicts, so I'd like to explore what it might look like to match a TypedDict... Consider an example web application that wants … [View More]to provide a service to draw shapes, perhaps on a connected physical billboard. The service has a '/draw_shape' endpoint which takes a JSON object (a Shape TypedDict) describing a shape to draw: from bottle import HTTPResponse, request, route from typing import Literal, TypedDict, Union class Point2D(TypedDict): x: float y: float class Circle(TypedDict): type: Literal['circle'] center: Point2D # has a nested TypedDict! radius: float class Rect(TypedDict): type: Literal['rect'] x: float y: float width: float height: float Shape = Union[Circle, Rect] # a Tagged Union / Discriminated Union @route('/draw_shape') def draw_shape() -> None: match request.json: # a Shape? ... case _: return HTTPResponse(status=400) # Bad Request Now, what syntax could we have at the ... inside the "match" statement to effectively pull apart a Shape? The current version of PEP 634-636 would require duplicating all the keys and value types that are defined in Shape's underlying Circle and Rect types: match request.json: # a Shape? case {'type': 'circle', 'center': {'x': float(), 'y': float()}, \ radius: float()} as circle: draw_circle(circle) # type is inferred as Circle case {'type': 'rect', 'x': float(), 'y': float(), \ 'width': float(), 'height': float()} as rect: draw_rect(rect) # type is inferred as Rect case _: return HTTPResponse(status=400) # Bad Request Wouldn't it be nicer if we could use class patterns instead? match request.json: # a Shape? case Circle() as circle: draw_circle(circle) case Rect() as rect: draw_rect(rect) case _: return HTTPResponse(status=400) # Bad Request Now that syntax almost works except that Circle and Rect, being TypedDicts, do not support isinstance() checks. PEP 589 ("TypedDict") did not define how such isinstance() checks should work initially because it's somewhat complex to specify. From the PEP: > In particular, TypedDict type objects cannot be used in isinstance() tests > such as isinstance(d, Movie). The reason is that there is no existing > support for checking types of dictionary item values, since isinstance() > does not work with many PEP 484 types, including common ones like List[str]. > [...] > This is consistent with how isinstance() is not supported for List[str]. Well, what if we (or I) took the time to specify how isinstance() worked with TypedDict? Then the match syntax above with TypedDict as a class pattern would work! Refining the example above even further, it would be nice if we didn't have to enumerate all the different types of Shapes directly in the match-statement. What if we could match on a Shape directly? match request.json: # a Shape? case Shape() as shape: draw_shape(shape) case _: return HTTPResponse(status=400) # Bad Request Now for that syntax to work it must be possible for an isinstance() check to work on a Shape, which is defined to be a Union[Circle, Rect], and isinstance() checks also aren't currently defined for Union types. So it would be useful to define isinstance() for Union types as well. Of course that match-statement is now simple enough to just be rewriten as an if-statement: if isinstance(shape := request.json, Shape): draw_shape(shape) else: return HTTPResponse(status=400) # Bad Request Now *that* is a wonderfully short bit of parsing code that results in well-typed objects as output. 🎉 So to summarize, I believe it's possible to support really powerful matching on JSON objects if we just define how isinstance() should work with a handful of new types. In particular the above example would work if isinstance() was defined for: * Union[T1, T2], Optional[T] * T extends TypedDict * Literal['foo'] For arbitrary JSON beyond the example, we'd also want to support isinstance() for: * List[T] We already support isinstance() for the other JSON primitive types: * str * float * bool * type(None) So what do folks think? If I were to start writing a PEP to extend isinstance() to cover at least the above cases, would that be welcome? -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [View Less]

3 5

'Infinity' constant in Python
by Cade Brown Nov. 23, 2020

Nov. 23, 2020

I am positing that Python should contain a constant (similar to True, False, None), called Infinity. It would be equivalent to `float('inf')`, i.e. a floating point value representing a non-fininte value. It would be the positive constant; negative infinity could retrieved via `-Infinity` Or, to keep float representation the same, the name `inf` could be used, but that does not fit Python's normal choice for such identifiers (but indeed, this is what C uses which is the desired behavior of … [View More]

28 161

super() magic-methods.
by Jonatan Nov. 22, 2020

Nov. 22, 2020

```class A: def __eq__(self, other): return '__eq__' class B(A): def __eq__(self, other): print(super() == other) print(super().__eq__(other)) B() == ...``` OUTPUT: False __eq__ As you can see here, when you run the code, __uq__ does not get printed twice. somehow, The expression "super() <OPERATOR> other" does not turn into "super().__OPERATOR__(other)". this bug is for any operator that you apply on super(). I'd be happy to hear from you why it … [View More]

3 2

PEP 634-636: Mapping patterns and extra keys
by David Foster Nov. 22, 2020

Nov. 22, 2020

From PEP 636 (Structural Pattern Matching): > Mapping patterns: {"bandwidth": b, "latency": l} captures the "bandwidth" and "latency" values from a dict. Unlike sequence patterns, extra keys are ignored. It surprises me that ignoring extra keys would be the *default* behavior. This seems unsafe. Extra keys I would think would be best treated as suspicious by default. * Ignoring extra keys loses data silently. In the current proposal: point = {'x': 1, 'y': 2, 'z': 3) match … [View More]point: case {'x': x, 'y': y}: # MATCHES, losing z O_O pass case {'x': x, 'y': y, 'z': z}: # will never match O_O pass * Ignoring extra keys is inconsistent with the handling of sequences: We don't allow extra items when using a destructuring assignment to a sequence: p = [1, 2] [x, y] = p [x, y, z] = p # ERROR: ValueError: not enough values to unpack (expected 3, got 2) :) * Ignoring extra keys in mapping patterns is inconsistent with the current proposal for how sequence patterns match data: point = [1, 2, 3] match point: case [x, y]: # notices extra value and does NOT match :) pass case [x, y, z]: # matches :) pass * Ignoring extra keys is inconsistent with TypedDict's default "total" matching behavior: from typing import TypedDict class Point2D(TypedDict): x: int y: int p1: Point2D = {'x': 1, 'y': 2} p2: Point2D = {'x': 1, 'y': 2, 'z': 3) # ERROR: Extra key 'z' for TypedDict "Point2D" :) * It is *possible* to force an exact key match with a pattern guard but it's clumsy to do so. It should not be clumsy to parse strictly. point = {'x': 1, 'y': 2, 'z': 3) match point: # notices extra value and does NOT match, but requires ugly guard :/ case {'x': x, 'y': y, **rest} if rest == {}: pass case {'x': x, 'y': y, 'z': z, **rest} if rest == {}: pass To avoid the above problems, **I'd advocate for disallowing extra keys in mapping patterns by default**. For cases where extra keys want to be specifically allowed and ignored, I propose allowing a **_ wildcard. Some examples that illustrate behavior when *disallowing* extra keys in mapping patterns: 1. Strict parsing from typing import TypedDict, Union Point2D = TypedDict('Point2D', {'x': int, 'y': int}) Point3D = TypedDict('Point3D', {'x': int, 'y': int, 'z': int}) def parse_point(point_json: dict) -> Union[Point2D, Point3D]: match point_json: case {'x': int(x), 'y': int(y)}: return Point2D({'x': x, 'y': y}) case {'x': int(x), 'y': int(y), 'z': int(z)}: return Point3D({'x': x, 'y': y, 'z': z}) case _: raise ValueError(f'not a valid point: {point_json!r}') 2. Loose parsing, discarding unknown data. Common when reading JSON-like data when it's not necessary to output it again later. from typing import TypedDict TodoItem_ReadOnly = TypedDict('TodoItem_ReadOnly', {'title': str, 'completed': bool}) def parse_todo_item(todo_item_json: Dict) -> TodoItem_ReadOnly: match todo_item_json: case {'title': str(title), 'completed': bool(completed), **_}: return TodoItem_ReadOnly({'title': title, 'completed': completed}) case _: raise ValueError() input = {'title': 'Buy groceries', 'completed': True, 'assigned_to': ['me']} print(parse_todo_item(input)) # prints: {'title': 'Buy groceries', 'completed': True} 3. Loose parsing, preserving unknown data. Common when parsing JSON-like data when it needs to be round-tripped and output again later. from typing import Any, Dict, TypedDict TodoItem_ReadWrite = TypedDict('TodoItem_ReadWrite', {'title': str, 'completed': bool, 'extra': Dict[str, Any]}) def parse_todo_item(todo_item_json: Dict) -> TodoItem_ReadWrite: match todo_item_json: case {'title': str(title), 'completed': bool(completed), **extra}: return TodoItem_ReadWrite({'title': title, 'completed': completed, 'extra': extra}) case _: raise ValueError() def format_todo_item(item: TodoItem_ReadWrite) -> Dict: return {'title': item['title'], 'completed': item['completed'], **item['extra']} input = {'title': 'Buy groceries', 'completed': True, 'assigned_to': ['me']} output = format_todo_item(parse_todo_item(input)) print(output) # prints: {'title': 'Buy groceries', 'completed': True, 'assigned_to': ['me']} Comments? -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [View Less]

5 11

One Type for pattern matching a sequence and dictionary
by Abdulla Al Kathiri Nov. 20, 2020

Nov. 20, 2020

Greetings all, We can do one type annotations like list[int] and dict[str, int], my question: why don’t we have a similar thing in matching patterns? Especially if you want to enforce a type on a sequence or a dictionary with unknown length. For example, ls = [3.14, ‘hello’, 1, 2, 3, 4, ...] d = {‘name’: 1, ‘age’: 25, ...} match ls: case [float(), str(), *rest] if all(isinstance(elem, int) for elem in rest): ... (do something with rest) We do the same thing with the following: … [View More]

1 0

Introduce "__python__" built-in attribute
by William Pickard Nov. 20, 2020

Nov. 20, 2020

TL/DR: A new built-in attribute who's purpose is to provide a simple way for developers to detect the Python implementation like CPython, JPython, IronPython and PyPy among other information. Ok, so the reason I'm suggesting this is for another suggestion I'll submit a later date (once I feel this one has ran it's course, or the contributors decide about it). The goal of this attribute (as mentioned above) is to provide developers quick and simple information about the Python runtime like … [View More]

5 4

Extending collections.Counter with top_n() to return elements by rank
by Bora Alper Nov. 19, 2020

Nov. 19, 2020

collections.Counter has most_common([n]) method which returns the most common n elements of the counter, but in case of a tie the result is unspecified --- whereas in practice the order of insertion breaks the tie. For example: >>> Counter(["a","a","b","a","b","c","c","d"]).most_common(2) [('a', 3), ('b', 2)] >>> Counter(["a","a","c","a","b","b","c","d"]).most_common(2) [('a', 3), ('c', 2)] In some cases (which I believe are not rare) you would like to break the tie … [View More]

1 0

Enable subscription operator for generator expressions
by Nuri Jung Nov. 18, 2020

Nov. 18, 2020

How about enabling subscription operator (`[]`) for generator expressions? Also for all `zip()`, `key()`, etc. They could be evaluated in the background only for the requested amount, to avoid evaluating the whole expression to something like a list or tuple, then indexed.

7 10

Inline dict manipulations
by Tomasz Hławiczka Nov. 17, 2020

Nov. 17, 2020

Hello, This is my first mail here, so greetings for everyone :) I would like to introduce an idea of the new operators for dict objects. Any thoughts are welcome. Since Python3.9 there is a great feature for merging dicts using | (OR) operator: {1: "a"} | {2: "b"} == {1: "a", 2: "b"} Thus, this basic operation can be done smoothly (also inline). PEP: https://www.python.org/dev/peps/pep-0584/ Another common operation on dicts is a substitution - building a new dict with only a part of … [View More]

1 0

Re: PEP 634-636: Mapping patterns and extra keys
by Valentin Berlier Nov. 17, 2020

Nov. 17, 2020

I'm in favor of keeping the PEP as it currently is. Mappings are naturally structural subtypes of one another, therefore mapping patterns should be consistent with class patterns. car = Car(...) match car: case Vehicle(): pass case Car(): # will never match pass This example is analogous to the first one in the discussion. If Car is a subclass of Vehicle, then Vehicle() is a more general pattern than Car() and will always match despite the … [View More]

1 0

Use __bytes__ to access buffer-protocol from "user-land"
by phk＠freebsd.dk Nov. 16, 2020

Nov. 16, 2020

I am working on a toolbox for computer-archaeology where old data media are "excavated" and presented on a web-page. (https://github.com/Datamuseum-DK/AutoArchaeologist for anybody who cares). Since these data-media can easily sum tens of gigabytes, mmap and virtual memory is my weapons of choice and that has brought me into an obscure corner of python where few people seem to venture: I want to access the buffer-protocol from "userland". The fundamental problem is that if I have a image of … [View More]

3 2

constants in Python: Starting simple and gradually adding more features, was: Re: Pattern Matching controversy
by Paul Sokolovsky Nov. 16, 2020

Nov. 16, 2020

Hello, On Mon, 16 Nov 2020 08:39:30 +1100 Steven D'Aprano <steve(a)pearwood.info> wrote: [] > > The baseline of my version is much simpler: > > > > # This makes "const" a kind of hard keyword for this module > > from __future__ import const > > > > FOO: const = 1 # obviously, this is constant > Oh, well, To start with, in the original thread I wanted to concentrate on issues of the PEP634/PEP635, and whether these issues are prominent enough … [View More]to hope for them being addressed. So, I changed the subject (and we'd soon be excused to python-ideas, indeed, I cc: there). > if all it takes is to add a new keyword, then constants are > easy! A new annotation. And the new subject is "constants in Python: Starting simple and gradually adding more", which hopefully should set the theme. Please remember how we arrived here: it's from the fact that PEP634 doesn't allow for the following trivial code: MY_CONST = 1 match foo: case MY_CONST: We're considering (another alternative) how to address that. Please don't make a leap towards supporting (at once) const-ness equivalent to statically typed languages. > No need to worry about how constantness affects name resolution, > or how the syntax interacts with type-hints: "const" is *the* type hint. > spam: const: Widget = build_widget(*args) That syntax is clearly invalid. And composability of type annotations (aka type hints) is a known, looming issue. We now have https://www.python.org/dev/peps/pep-0593/ , but in all fairness, it seems like a stopgap measure rather than an elegant solution. (In full fairness, entire "typing" module's annotations aren't very elegant, but as we know, it's not a part of language core, but part of CPython's stdlib. That means it's only *one* of possible annotation schemes for *Python*). So, under PEP593, the above would be written spam: Annotated[Widget, const] = build_widget(*args) If you want a glimpse of what alternatives might look, then: given that "|" is going to be used for union types, why not try "&" for composition? spam: Widget & const = build_widget(*args) But again, for "initial support of constants in Python, prompted by the introduction of pattern matching facilities", we don't need to worry about all that right away. > # Maybe this is better? > # let spam: Widget = build_widget(*args) As you remember, at the beginning of my proposal, I wrote "The baseline of my version ...". Baseline == level 0. For "level 2", I considered const spam: Widget = ... I don't think that "let" should be used. We're putting emphasis on *constantness* (not lexical scoping of variables [immutable by functional tradition, though that's "minor"]). JavaScript (now) has both "const" and "let", and I think that's pretty reasonable approach. So, let's save "let" for possible later uses. So, what's "level 1"? As I mentioned, under "level 0", "from __future__ import const" has a magic meaning (akin to other "from __future__"'s). Under "level 1", "const" is just a normal annotation symbol imported from a module. So, following would be possible: ========= from __future__ import const FOO: const = 1 def fun(): const = 1 # Just a variable in function scope def sub(): # Gimme powerz back from __future__ import const BAR: const = 2 # Back to annotation in global scope BAZ: const = 3 ========= "level 0" should be implementable in CPython in ~1hr. "level 1" is realistically what we should shoot for. "level 2" (the dedicated keyword), I'm still not sure about. "const" is very baseline annotation, and deserves a dedicated keyword. But the underlying problem is composability of (arbitrary) annotations. I'd rather keep searching for graal in that regard, before giving up and introduce a dedicated thing just for "const". > or how "constant" interacts with mutability: > > spam: const = [] > spam.append(99) # Allowed? > spam.extend(items) > spam.sort() "const" is an annotation just like any other. And it affects *runtime* in the same as any other annotation in CPython affects it: in no way. "const" is however introduced as a hint for *compile-time*, so compiler could make some simple inferences and maybe even optimizations based on it. > or for that matter, whether or not constants are actually constant. > > spam: const = 1 > spam = 2 Compiler can, and thus would, catch that as an error (warning for a beta version?). > If constants aren't actually constant, but just a tag on symbols, > then you would be right, it probably would be trivially easy to add > "constants" to Python. Right. > But I think most people agree making them behave as constants is a > pretty important feature. As mentioned, "const" is just an annotation like any other, except compiler has some insight into it. Dealing with runtime is distant goal for CPython. (Which is for now offloaded to static type checkers and libraries/alternative Python implementations.) > *The* most critical feature of all, really. > > Given a declared constant: > > # Module a.py > x: const = None > > how do you prevent not just code in module `a` from rebinding the > value: Outside of the current scope of the discussion. However, if you're interested in that topic, then I've implemented it in my Python dialect, Pycopy (https://github.com/pfalcon/pycopy). It's on my TODO to post an RFC to python-ideas for beating. [] -- Best regards, Paul mailto:pmiscml@gmail.com [View Less]

2 2

Fwd: Re: Global flag for whether a module is __main__
by André Roberge Nov. 15, 2020

Nov. 15, 2020

Sorry, forgot to use "reply to all" ---------- Forwarded message --------- From: André Roberge <andre.roberge(a)gmail.com> Date: Sat, Nov 14, 2020 at 11:06 AM Subject: Re: [Python-ideas] Re: Global flag for whether a module is __main__ To: Steven D'Aprano <steve(a)pearwood.info> On Sat, Nov 14, 2020 at 10:45 AM Steven D'Aprano <steve(a)pearwood.info> wrote: > On Sat, Nov 14, 2020 at 08:10:44AM -0400, André Roberge wrote: > > > > What if you import the `… [View More]

3 2

Global flag for whether a module is __main__
by Matt Wozniski Nov. 15, 2020

Nov. 15, 2020

Currently, the simplest and most idiomatic way to check whether a module was run as a script rather than imported is: if __name__ == "__main__": People generally learn this by rote memorization, because users often want the ability to add testing code or command line interfaces to their modules before they understand enough about Python's data model to have any idea why this works. Understanding what's actually happening requires you to know that: 1. the script you ask Python to run is … [View More]

12 17

unix filter, one liners, awk
by Hans Ginzel Nov. 14, 2020

Nov. 14, 2020

Is there a reason, why not to make python useful for practical one liners to replace perl and awk? There is page Powerful Python One-Liners, https://wiki.python.org/moin/Powerful%20Python%20One-Liners. But almost none of them handles files, behaves like unix filter or is ugly – must import modules, etc. It seems that the command line options like -a, -n, -p, -F are still free. :-) https://docs.python.org/3/using/cmdline.html Why not use them almost the same way as in Perl? https://perldoc.… [View More]

8 7

filterwarnings on a module in the current stack
by Bruno Beltran Nov. 7, 2020

Nov. 7, 2020

AFAICT, there is no way to filter warnings based on anything but their most immediate caller (please correct me if I'm wrong). This can lead to situations where it's impossible to write a warnings filter that's not brittle. For example: Say I want to filter a certain warnings `sphinx.ext.autodoc` during my build process (e.g. to ignore deprecation warnings when autodoc imports deprecated modules that I need to keep documented). I can filter: ```python warnings.filterwarnings('ignore', … [View More]category=MatplotlibDeprecationWarning, message=r'(\n|.)*module was deprecated.*') ``` but then when sphinx is then building my gallery of examples, this hides the `DeprecationWarnings` that I do want to see! What I'd like to do is write ```python warnings.filterwarnings('ignore', category=MatplotlibDeprecationWarning, module='sphinx.ext.autodoc', message=r'(\n|.)*module was deprecated.*') ``` however the warning is not actually triggered by the `autodoc` module itself, but instead let's say hypothetically it's triggered by using `imp` internally to load the module. Then I would have to instead do something like the following: ```python warnings.filterwarnings('ignore', category=MatplotlibDeprecationWarning, module='imp', message=r'(\n|.)*module was deprecated.*') ``` but now let's say in the past couple of years, Sphinx, being the well-maintained library that it is, moves to using `importlib.import_module`. Now suddenly warnings are appearing and I have to go edit this guy to fix CI: ```python warnings.filterwarnings('ignore', category=MatplotlibDeprecationWarning, module='importlib', message=r'(\n|.)*module was deprecated.*') ``` This is (at worst) a minor inconvenience in this silly example case, but in general one could imagine that if you want to use an upstream library in your code internally, in a way that raises a warning (that you want to ignore), it's currently impossible to do that without relying on what *should be* implementation details of the library you're using. One obvious solution in my mind is to allow `filterwarnings` to inspect the current stack at the time that a warning is raised, since this contains all of the information needed in principle for the user to decide whether or not to ignore a warning. Has this kind of thing been discussed before? I tried my best to search but couldn't find anything, and while I'm a decades-long *user* of Python I don't tend to keep up with the development side of things. [View Less]

1 0

propose index heap
by master_lee Nov. 6, 2020

Nov. 6, 2020

Index heap is essential in efficient implementation of Dijkstra's algorithm, Prim's algorithm and other variants, I have implemented a concise version at https://github.com/yutao-li/libheap , I think it would be useful to have it in stdlib

2 1

Enum and the Standard Library (and __str__ and __repr__)
by Ethan Furman Nov. 4, 2020

Nov. 4, 2020

TL;DR Changes may be coming to Enum str() and repr() -- your (informed) opinion requested. :-) Python-Dev thread [0], summary below: > As you may have noticed, Enums are starting to pop up all over the stdlib [1]. > > To facilitate transforming existing module constants to IntEnums there is > `IntEnum._convert_`. In Issue36548 [2] Serhiy modified the __repr__ of RegexFlag: > > >>> import re > >>> re.I > re.IGNORECASE > > I think for … [View More]

3 4

Maxheap version of a heappush into heapq module
by Rudresh Veerkhare Nov. 1, 2020

Nov. 1, 2020

heapq module contains all the function to implement a Heap structure, Main functions required to implement Heap data structure are: function heappush - to push an element in Heap function heappop - to pop an element from Heap for implementing Minheap this functions are present in the module as : heappush - for adding element into Minheap heappop - to pop an element from Minheap for implementing Maxheap only one of this two required functions is present: _heappop_max - to pop an element … [View More]

1 0

Dict unpacking assignment
by Steven D'Aprano Nov. 1, 2020

Nov. 1, 2020

I would like to propose dict (or mapping) unpacking assignment. This is inspired in part by the Python-Ideas thread "f-strings as assignment targets", dict unpacking in function calls, and iterable unpacking assignment. Comments welcome. Background ---------- Iterable unpacking assignment: values = (1, 2, 3) a, b, c = *values is a very successful and powerful technique in Python. Likewise dict unpacking in function calls and dict displays: kwargs = {'a': 1, 'b': 2} … [View More]func(**kwargs) d = {**kwargs, 'c': 3} There have been various requests for allowing dict unpacking on the right-hand side of an assignment [citation required] but in my opinion none of them have had a good justification. Motivated by the idea of scanning text strings with a scanf-style function, I propose the following behaviour for dict unpacking assignment: items = {'eggs': 2, 'cheese': 3, 'spam': 1} spam, eggs, cheese = **items assert spam == 1 and eggs == 2 and cheese == 3 Syntax ------ target_list [, **target] = **expression `target_list` is a comma-separated list of targets. Targets may be: - simple names, e.g. `spam` and `eggs` - dotted names, e.g. `spam.eggs` - numbered subscripts, e.g. `spam[1]` but is not required to support arbitrary complex targets such as: spam(*args).eggs(2*x + y)[1].cheese # not supported Likewise only int literals are supported for subscripts. (These restrictions may be lifted.) This is similar to the limited range of fields acceptabled by the string format mini-language. The same restrictions apply to `**target`. Each target must be unique. `expression` must evaluate to a dict or other mapping. Assignment proceeds by matching up targets from the left to keys on the right: 1. Every target must be matched exactly by a key. If there is a target without a corresponding key, that is an error. 2. Any key which does not match up to a target is an error, unless a `**target` is given. 3. If `**target` is given, it will collect any excess key:value pairs remaining into a dict. 4. If the targets and keys match up, then the bindings are applied from left to right, binding the target to the value associated with that key. Examples: # Targets are not unique a, b, a = **items => SyntaxError # Too many targets a, b, c = **{'a': 1, 'b': 2} => raises a runtime exception # Too few targets a = **{'a': 1, 'b': 2} => raises a runtime exception a, **extras = **{'a': 1, 'b': 2} assert a == 1 assert extras == {'b': 2} # Equal targets and keys a, b, **extras = **{'a': 1, 'b': 2} assert a == 1 assert b == 2 assert extras == {} # Dotted names from types import SimpleNamespace obj = SimpleNamespace() obj.spam = **{'obj.spam': 1} assert obj.spam == 1 # Subscripts arr = [None]*5 arr[1], arr[3] = **{'arr[3]': 33, 'arr[1]': 11} assert arr == [None, 11, None, 33, None] Assignments to dotted names or subscripts may fail, in which case the assignment may only partially succeed: spam = 'something' eggs = None spam, eggs.attr = {'spam': 1, 'eggs.attr': 2} # raises AttributeError: 'NoneType' object has no attribute 'attr' # but `spam` may have already been bound to 1 (I think that this is undesirable but unavoidable.) Motivating use-cases -------------------- The motivation comes from the discussion for scanf-like functionality. The addition of dict unpacking assignment would allow something like this: pattern = "I'll have {main} and {extra} with {colour} coffee." string = "I'll have spam and eggs with black coffee." main, extra, colour = **scanf(pattern, string) assert main == 'spam' assert extra == 'eggs' assert colour == 'black' But the possibilities are not restricted to string scanning. This will allow functions that return multiple values to choose between returning them by position or by name: height, width = get_dimensions(window) # returns a tuple height, width = **get_dimensions(window) # returns a mapping Developers can choose whichever model best suits their API. Another use-case is dealing with kwargs inside functions and methods: def method(self, **kwargs): spam, eggs, **kw = **kwargs process(spam, eggs) super().method(**kw) -- Steve [View Less]

17 32