Python-ideas November 2020

python-ideas@python.org

85 participants
45 discussions

by Abdulla Al Kathiri

Instead of importing “Any" from the typing module, we can annotate our functions with “Any" right away without the extra step. What do you think? We have the builtin function “any” which some Python users could mistakingly use, but static type checkers should catch that.

2 months, 3 weeks

Re: Making the for statement work better with nested functions

by Paul Sokolovsky

Hello, On Mon, 30 Nov 2020 12:26:53 +0000 Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > On 30/11/2020 08:01, Paul Sokolovsky wrote: > > Hello, > > > > On Sun, 29 Nov 2020 18:53:57 -0800 > > Christopher Barker <pythonchb(a)gmail.com> wrote: > > > >> This is a massively less ambitious idea, and doesn't solve the > >> original problem, but: > >> > >> I'd like to see a scope for names that are "obviously ;-)" meant > >> to be short lived. At the moment that's the for loop "counters" > >> and those created by context managers: > >> > >> for i in something: > >> # use i as usual > >> more_code > >> # now i is not defined > > What other language(s) implement such a scoping discipline? I know > > of none. > > > > On the other hand, block-scoped variables are implemented in: > > > > * C > > * C++ > > * Java > > * Rust > > * Lua > > * JavaScript (not by default, as opt-in) > > * Scheme > > * Common Lisp (as opt-in) > > * ML > > * Ocaml > > * Haskell > > * very long list of other languages... > > > > > > The aim of the block scoping proposal is to make Python *not worse* > > than these other languages, instead of adding funny workarounds > > again. > > > > [] > > > > CHB's proposal may or may not be a good idea. But please treat it on > its merits, Absolutely. I appreciate all the discussion and alternative ideas people put into this. I, on my side, may have a "strong opinion" that ideas I utter are "right", but that doesn't mean I'm not mistaken. These matters really require good consideration from as wider as possible Python community. > instead of saying, as you seem to be, "No other language > does this, so it's bound to be a bad idea which will make Python > worse". That's why I try to choose words carefully ;-). I said that if Python does it like many other languages, it definitely won't be worse than them. I said nothing what happens if it doesn't do it like that ;-). All in all, it's just an argument, as anyone else's. But I do think about those matters, I have to admit. And I'd hate yet another advanced Python programmer to leave for Haskell, Go, Rust, Julia, because, from their PoV, Python has a culture of applying workarounds instead of best-practice solutions. > With that attitude, there would never be any innovation in > language design. I'm first to subscribe under that. Recent example: Pattern Matching, a common mantra is "we did it like that because other languages do it like that" (example: https://mail.python.org/archives/list/python-dev@python.org/message/EWX3GWN…) My response: But did you consider Python's situation? Both "minor" issues, like lack of constants, which requires ugly workarounds, and overall audience of Python? Was the option to use sigils to mark capture (i.e. assignment) targets *really well* considered? They said yes. Ok. Works for me, and I tried to do as much noise as possible to draw other people's attention to that. > Best wishes Rob Cliffe [] -- Best regards, Paul mailto:pmiscml@gmail.com

2 months, 4 weeks

adding a timeit.Timer context manager and decorator to time code/functions

by sdementen＠gmail.com

When optimizing code, I often need to timeit a part of code or a function. I am using then the following class ``` import logging import time from functools import wraps from typing import Optional _logger = logging.getLogger("Timer") class Timer: def __init__(self, description: Optional[str] = None, logger: Optional[logging.Logger] = None): self.description = "<anonymous code block>" if description is None else description self.logger = _logger if logger is None else logger self.msg = "'%(description)s' ran in %(time).3f seconds" def __enter__(self): self.start = time.perf_counter() return self def __exit__(self, *args): self.end = time.perf_counter() self.interval = self.end - self.start self.logger.debug( self.msg, {"description": self.description, "time": self.interval} # f"Thread: '{get_ident()}' '{self.description}' ran in {round(self.interval, 3)} seconds" ) def __call__(self, f): # decorator call @wraps(f) def wrapper(*args, **kw): with Timer(description=f.__qualname__): return f(*args, **kw) return wrapper ``` that I can use either as a context manager in ``` with Timer("how long does this take?") as t: time.sleep(1) # output: DEBUG:Timer:'how long does this take?' ran in 1.000 seconds ``` or as a decorator ``` @Timer() def my_function(): """this is my function""" time.sleep(2) my_function() # output: DEBUG:Timer:'my_function' ran in 2.000 seconds ``` This class could be added to the timeit module as it fits well its functional scope.

2 months, 4 weeks

[PoC] Implementing block-scope for the "for" statement variables

by Paul Sokolovsky

Hello, This is continuation of the "Making the for statement work better with nested functions" thread (https://mail.python.org/archives/list/python-ideas@python.org/thread/3ZKYV6…). I post this as a separate thread to draw more attention to and promote this way of prototyping Python changes. The changes are implemented using Python3 port of Python2's "compiler" module: https://github.com/pfalcon/python-compiler . For more information on background, see the parallel thread: https://mail.python.org/archives/list/python-ideas@python.org/message/W2FRE… python-compiler module requires CPython3.5. So, please be prepared to enjoy again the golden age of Python3 (comparing to iron (perhaps, oil?) age of now). If you don't have that handy, the repository has a script which will build it for you in a couple of minutes (on Linux). The changes are implemented in the following branch: https://github.com/pfalcon/python-compiler/commits/for-block-scope . The top 3 commits are that, shouldn't be too hard to review (and have some additional info in the commit messages). As what's available there is only AST-to-bytecode compiler, not source parser, I couldn't easily implement "for let"/"for const" syntax, so went to switch *all* for statements to the block scope for this prototype. I also prototyped only "block scope" part, not the "constness" part. To describe high-level idea of the changes: The conceptual idea is that we can rewrite each definition of block-scoped variable to have a unique name, and then rewrite usages accordingly in all inner scopes (including inner functions). Direct implementation of that was deemed "too naive", based on the grounds that (again naive) implementation of it would require whole extra pass on AST, and patching it. Note that it's not necessarily that (what? bad), and if it ever comes to implementing in the C compiler, it may be a good idea to actually go that way first and see how actually "bad" it is. It may be possible to combine that pass with another pass (variable scoping AST pass, literally). The benefit of this approach is that it's "obviously correct" and doesn't introduce other overheads. Anyway, my point was to show that even doing it "better" is "not rocket science". I did it half-way still, because there's obvious rising conceptual complexity and rising storage requirements for intermediate info (what made me think that maybe "naive" approach described above isn't that bad). Here's example of it in action: $ cat example_for1.py def fun(): x = 123 for x in range(5): print(x) print("old x:", x) fun() $ ./python3.5-nopeephole -m compiler --dis example_for1.py 1 0 LOAD_CONST 0 (<code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>) 3 LOAD_CONST 1 ('fun') 6 MAKE_FUNCTION 0 9 STORE_NAME 0 (fun) 7 12 LOAD_NAME 0 (fun) 15 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 18 POP_TOP 19 LOAD_CONST 2 (None) 22 RETURN_VALUE Disassembly of <code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>: 2 0 LOAD_CONST 1 (123) 3 STORE_FAST 0 (x) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_GLOBAL 0 (range) 12 LOAD_CONST 2 (5) 15 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_FAST 1 (x.1) 4 25 LOAD_GLOBAL 1 (print) 28 LOAD_FAST 1 (x.1) 31 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 34 POP_TOP 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_GLOBAL 1 (print) 42 LOAD_CONST 3 ('old x:') 45 LOAD_FAST 0 (x) 48 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 51 POP_TOP 52 LOAD_CONST 0 (None) 55 RETURN_VALUE 0 1 2 3 4 old x: 123 -- Best regards, Paul mailto:pmiscml@gmail.com

2 months, 4 weeks

Variadic generics PEP draft

by Matthew Rahtz

Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew

2 months, 4 weeks

async types?

by Ben Avrahami

Hi all, this is a general feeler for if this idea has any traction: All too often I see the following pattern in asyncio 3rd-party libs, either in their own source code or in the inusage: ``` inst = SomeClass() await inst.initialize() ``` What happens here is that, since coroutines cannot be used inside __init__ methods, the __init__ method only stores the parameters, and another, asynchronous method actually initializes the object. This led many 3rd-party libs to use factory methods to create their class instances, which is unideal for both users and developers. We see this pattern in nearly any async 3rd-party lib I came across. To solve this, I propose a new metaclass: `AsyncType`. The core difference between AsyncType and type is that when called, AsyncType will produce a coroutine that asynchronously calls __new__ and __init__, and returns the instance, allowing simply for `instance = await SomeClass()`. In classes of `AsyncType`, the __new__ and __init__ methods must be async methods (returning coroutines). As an additional syntactic sugar, we could also have an `async class(base)` declaration that implicitly sets the class's metaclass to be AsyncType (or raise a TypeError if that is impossible). This proposal is obviously incomplete, and there are many open questions: what about __del__? How would one easily make an async ABC (without a metaclass conflict)? How would an async class easily inherit from a sync class (suppose sync class A implements __new__, and async class B(A) implements __init__)? Perhaps `AsyncType` should only make the __init__ method async and leave __new__ synchronous? I'd just like to get ahead of the (not unjustified) argument that constructors should be lightweight: 1. Python's __init__ is not strictly a constructor. It is, as the name implies, an initializer, that makes the type usable. Constructing the type without initializing it makes no sense and has no usages, so why seperate the operations? 2. We can see numerous examples of (syncronous) connect-on-initialize classes in the standard library, including ftplib, smtplib, and pretty much most *lib modules in the standard library "Internet Protocols and Support" doc page. 3. The "initialize in a separate method" pattern produces some strange-looking code where the class holds on to its initialization params even though it doesn't really need them after the separate methods. The classes should now also consider and safe-guard against cases where the separate method is called twice, or not at all. What does the community think about this idea?

2 months, 4 weeks

Move semantics

by 3mir33＠gmail.com

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

3 months

Getting rid of FOR loops and simplifying cicular conviolutions with Circular Indexing

by Mathew M. Noel

If circular indexing is used then instead of using a double FOR loop to go through a list M times we can iterate from 0 to M*N (where N is the length of the list) !!! Almost all Machine Learning (ML) algorithms iterate for some predefined epochs over a large data-set. So a double FOR loop is extremely common in ML. Using circular indexing gets rid of this extra FOR loop. If we have to iterate 2 times you can iterate using range(-n,n) but in most cases you need to iterate over 10 or more epochs in ML. Most scientific applications of Python involve an outer FOR loop which progressively refines an approximation with an inner FOR loop by going through a list of items. So circular indexing is useful. In the following I discuss increasingly compelling reasons for adopting a circular indexing scheme in Python. Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc). If circular indexing is used then we don't need the arbitrary rule that -1 is the index of the last element. Since -1 is the same as n-1 automatically in modular arithmetic. A trivial objection: "why not use list_name[i%n] whenever we need this feature?" By the same token we could do away with negative indices and use -1%n for example when we need to index with -1! Its unclear why that people have an irrational preference for indices that lie to the left of 0 while strongly rejecting the idea of indices that lie to the right of n-1! Python does not raise a "index out of bound" exception for negative indices like other programming languages. If this negative indexing is a "feature" (although it allows some fatal errors to slip) then indices above n-1 can also be considered a feature! Are there any deep mathematical reasons for adopting circular convention? Circular convolution is a most important operation in a wide variety of scientific disciplines since the Discrete Fourier Transform (DFT) of the circular convolution of two signals is the product of the transforms. Because of the universal applicability of Fourier ideas in science and the close mathematical relationship between finite length and periodic sequences circular indexing is extensively used in signal processing and mathematics. We can extend the idea of circular indexing to multidimensional arrays. A 2D array can be folded into a cylinder for indexing. Further this cylinder can be folded into a toroid to reduce a triple FOR loop to a single FOR loop. A deep mathematical justification for cylindrical indexing of 2D and in general nD arrays is offered by the fact that n-dimensional DFT reduces n-dimensional circular convolution to element-wise multiplication.

3 months

A python_include() for cgi programming

by warp_drive enterprise

I have recently have the opportunity to work on a Python web application. I've found it beneficial to reduce duplicate code with the following code lines import textwrap from pathlib import Path exec(textwrap.dedent(Path('myfile.py').read_text())) Perhaps such code could be the basis for a python_include() statement similar to how includes, requires work in PHP. Thank you for any consideration.

3 months

Circular Indexing

by Mathew M. Noel

Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc).

3 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas November 2020