Python-ideas November 2020

python-ideas@python.org

85 participants
52 discussions

[PoC] Implementing block-scope for the "for" statement variables
by Paul Sokolovsky 29 Nov '20

29 Nov '20

Hello, This is continuation of the "Making the for statement work better with nested functions" thread (https://mail.python.org/archives/list/python-ideas@python.org/thread/3ZKYV6…). I post this as a separate thread to draw more attention to and promote this way of prototyping Python changes. The changes are implemented using Python3 port of Python2's "compiler" module: https://github.com/pfalcon/python-compiler . For more information on background, see the parallel thread: https://mail.python.org/archives/list/python-ideas@python.org/message/W2FRE… python-compiler module requires CPython3.5. So, please be prepared to enjoy again the golden age of Python3 (comparing to iron (perhaps, oil?) age of now). If you don't have that handy, the repository has a script which will build it for you in a couple of minutes (on Linux). The changes are implemented in the following branch: https://github.com/pfalcon/python-compiler/commits/for-block-scope . The top 3 commits are that, shouldn't be too hard to review (and have some additional info in the commit messages). As what's available there is only AST-to-bytecode compiler, not source parser, I couldn't easily implement "for let"/"for const" syntax, so went to switch *all* for statements to the block scope for this prototype. I also prototyped only "block scope" part, not the "constness" part. To describe high-level idea of the changes: The conceptual idea is that we can rewrite each definition of block-scoped variable to have a unique name, and then rewrite usages accordingly in all inner scopes (including inner functions). Direct implementation of that was deemed "too naive", based on the grounds that (again naive) implementation of it would require whole extra pass on AST, and patching it. Note that it's not necessarily that (what? bad), and if it ever comes to implementing in the C compiler, it may be a good idea to actually go that way first and see how actually "bad" it is. It may be possible to combine that pass with another pass (variable scoping AST pass, literally). The benefit of this approach is that it's "obviously correct" and doesn't introduce other overheads. Anyway, my point was to show that even doing it "better" is "not rocket science". I did it half-way still, because there's obvious rising conceptual complexity and rising storage requirements for intermediate info (what made me think that maybe "naive" approach described above isn't that bad). Here's example of it in action: $ cat example_for1.py def fun(): x = 123 for x in range(5): print(x) print("old x:", x) fun() $ ./python3.5-nopeephole -m compiler --dis example_for1.py 1 0 LOAD_CONST 0 (<code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>) 3 LOAD_CONST 1 ('fun') 6 MAKE_FUNCTION 0 9 STORE_NAME 0 (fun) 7 12 LOAD_NAME 0 (fun) 15 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 18 POP_TOP 19 LOAD_CONST 2 (None) 22 RETURN_VALUE Disassembly of <code object fun at 0x7f7533bb1ed0, file "example_for1.py", line 1>: 2 0 LOAD_CONST 1 (123) 3 STORE_FAST 0 (x) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_GLOBAL 0 (range) 12 LOAD_CONST 2 (5) 15 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_FAST 1 (x.1) 4 25 LOAD_GLOBAL 1 (print) 28 LOAD_FAST 1 (x.1) 31 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 34 POP_TOP 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_GLOBAL 1 (print) 42 LOAD_CONST 3 ('old x:') 45 LOAD_FAST 0 (x) 48 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 51 POP_TOP 52 LOAD_CONST 0 (None) 55 RETURN_VALUE 0 1 2 3 4 old x: 123 -- Best regards, Paul mailto:pmiscml@gmail.com

2 2

Variadic generics PEP draft
by Matthew Rahtz 29 Nov '20

29 Nov '20

Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew

5 4

async types?
by Ben Avrahami 29 Nov '20

29 Nov '20

Hi all, this is a general feeler for if this idea has any traction: All too often I see the following pattern in asyncio 3rd-party libs, either in their own source code or in the inusage: ``` inst = SomeClass() await inst.initialize() ``` What happens here is that, since coroutines cannot be used inside __init__ methods, the __init__ method only stores the parameters, and another, asynchronous method actually initializes the object. This led many 3rd-party libs to use factory methods to create their class instances, which is unideal for both users and developers. We see this pattern in nearly any async 3rd-party lib I came across. To solve this, I propose a new metaclass: `AsyncType`. The core difference between AsyncType and type is that when called, AsyncType will produce a coroutine that asynchronously calls __new__ and __init__, and returns the instance, allowing simply for `instance = await SomeClass()`. In classes of `AsyncType`, the __new__ and __init__ methods must be async methods (returning coroutines). As an additional syntactic sugar, we could also have an `async class(base)` declaration that implicitly sets the class's metaclass to be AsyncType (or raise a TypeError if that is impossible). This proposal is obviously incomplete, and there are many open questions: what about __del__? How would one easily make an async ABC (without a metaclass conflict)? How would an async class easily inherit from a sync class (suppose sync class A implements __new__, and async class B(A) implements __init__)? Perhaps `AsyncType` should only make the __init__ method async and leave __new__ synchronous? I'd just like to get ahead of the (not unjustified) argument that constructors should be lightweight: 1. Python's __init__ is not strictly a constructor. It is, as the name implies, an initializer, that makes the type usable. Constructing the type without initializing it makes no sense and has no usages, so why seperate the operations? 2. We can see numerous examples of (syncronous) connect-on-initialize classes in the standard library, including ftplib, smtplib, and pretty much most *lib modules in the standard library "Internet Protocols and Support" doc page. 3. The "initialize in a separate method" pattern produces some strange-looking code where the class holds on to its initialization params even though it doesn't really need them after the separate methods. The classes should now also consider and safe-guard against cases where the separate method is called twice, or not at all. What does the community think about this idea?

3 2

Move semantics
by 3mir33＠gmail.com 27 Nov '20

27 Nov '20

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

15 38

Getting rid of FOR loops and simplifying cicular conviolutions with Circular Indexing
by Mathew M. Noel 26 Nov '20

26 Nov '20

If circular indexing is used then instead of using a double FOR loop to go through a list M times we can iterate from 0 to M*N (where N is the length of the list) !!! Almost all Machine Learning (ML) algorithms iterate for some predefined epochs over a large data-set. So a double FOR loop is extremely common in ML. Using circular indexing gets rid of this extra FOR loop. If we have to iterate 2 times you can iterate using range(-n,n) but in most cases you need to iterate over 10 or more epochs in ML. Most scientific applications of Python involve an outer FOR loop which progressively refines an approximation with an inner FOR loop by going through a list of items. So circular indexing is useful. In the following I discuss increasingly compelling reasons for adopting a circular indexing scheme in Python. Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc). If circular indexing is used then we don't need the arbitrary rule that -1 is the index of the last element. Since -1 is the same as n-1 automatically in modular arithmetic. A trivial objection: "why not use list_name[i%n] whenever we need this feature?" By the same token we could do away with negative indices and use -1%n for example when we need to index with -1! Its unclear why that people have an irrational preference for indices that lie to the left of 0 while strongly rejecting the idea of indices that lie to the right of n-1! Python does not raise a "index out of bound" exception for negative indices like other programming languages. If this negative indexing is a "feature" (although it allows some fatal errors to slip) then indices above n-1 can also be considered a feature! Are there any deep mathematical reasons for adopting circular convention? Circular convolution is a most important operation in a wide variety of scientific disciplines since the Discrete Fourier Transform (DFT) of the circular convolution of two signals is the product of the transforms. Because of the universal applicability of Fourier ideas in science and the close mathematical relationship between finite length and periodic sequences circular indexing is extensively used in signal processing and mathematics. We can extend the idea of circular indexing to multidimensional arrays. A 2D array can be folded into a cylinder for indexing. Further this cylinder can be folded into a toroid to reduce a triple FOR loop to a single FOR loop. A deep mathematical justification for cylindrical indexing of 2D and in general nD arrays is offered by the fact that n-dimensional DFT reduces n-dimensional circular convolution to element-wise multiplication.

7 11

A python_include() for cgi programming
by warp_drive enterprise 26 Nov '20

26 Nov '20

I have recently have the opportunity to work on a Python web application. I've found it beneficial to reduce duplicate code with the following code lines import textwrap from pathlib import Path exec(textwrap.dedent(Path('myfile.py').read_text())) Perhaps such code could be the basis for a python_include() statement similar to how includes, requires work in PHP. Thank you for any consideration.

5 4

Circular Indexing
by Mathew M. Noel 26 Nov '20

26 Nov '20

Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc).

10 12

Circular Indexing and FOR loop minimization
by Mathew M. Noel 25 Nov '20

25 Nov '20

As discussed earlier on the post on 'Circular Indexing', considering interpreting indices that lie outside the range 0 to n-1 modulo n for list indexing. Thus element n corresponds to element 0, n+1 to element 1 and so on. This has two consequences: 1.) Makes the Python convention of interpreting index -1 as the last element more logical since -1 is the same as n-1 when taken modulo n. If -1 is the first element then to be logically consistent we have to interpret index n as the element 0 as well! Obviously this does allow certain errors to slip but if you are going to allow an index of -1 (indices that occur before 0) then you might just as well allow indices above n-1 to be logically consistent. 2.) If circular indexing is used then instead of using a double FOR loop to go through a loop twice we can iterate from 0 to 2n ! Although trivial for the case of one dimensional lists/arrays, cylindrical indexing (for 2D arrays) and toroidal indexing (2D, 3D and nD arrays) schemes might be worth exploring for n-dimensional NumPy arrays. Circular and in general cylindrical or toroidal indexing schemes might simplify coding by reducing the number of FOR loops when iterating over nD arrays in NumPy.

3 2

Packaging Apps (Was Adding PyInstaller to the stdlib)
by Abdur-Rahmaan Janhangeer 25 Nov '20

25 Nov '20

Greetings list, > Then I would suggest starting a new thread, Long overdue i guess. My view on the topic is like Mr Moore with the exception that i'm for native capabilities to be in the stdlib. Since Zipapp does not seem to go the C-extension route, i propose a new tool that turn projects into independent executables This solves the issue of Gui etc etc etc etc etc etc Mr Moore's mail of Nov 24, 2020 has ton of ideas and a lot of potential Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius

1 0

A pypi-based app store solution for platform specific installation
by Ricky Teachey 25 Nov '20

25 Nov '20

I was reading the pyinstaller thread and had this idea but didn't want to hijack. Maybe a wild idea, and very possible totally impractical or hopelessly complex, but: could the existing pypi infrastructure be leveraged into a set of platform-specific app stores? Maybe maybe we could make it as simple as a single line in a pyproject.toml file. Imagine if all the end user does is install the store, and clicks the link to the project app. The store takes care of the rest. The developer marks their package as an installable program to be published to the store, optionally specifying things like specific platforms if needed. All the project website does for users to install is provide a url to their app in the store. Very very rough idea of how it might work: a store app API would provide an installation GUI if desired, and the store would build the environment needed for installing the package and dependencies. Once the specified environment is built, pip would take care of installing the package as usual. If this is an awful idea feel free to just say so. I have given it no deep thought and do not have the expertise to even think through what would be needed to create such a thing.

5 6

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas November 2020