Python-ideas

python-ideas@python.org

81 participants
4909 discussions

Making the for statement work better with nested functions

by Greg Ewing

Starting a new thread for this as suggested by Guido. On 18/11/20 7:20 pm, Guido van Rossum wrote: > On Tue, Nov 17, 2020 at 22:12 Greg Ewing <greg.ewing(a)canterbury.ac.nz > <mailto:greg.ewing@canterbury.ac.nz>> wrote: > > If there's anything I would change, it would be to have the for > statement create a new binding on each iteration, so that capturing > it with a def or lambda inside the loop works as expected. I even > came up with a way to do that while still allowing the last-bound > value to be seen afterwards, but the idea didn't gain any traction. > > I wonder if we should try that idea again? The problem may be backwards > compatibility — there’s always someone who depends on such a thing (even > if by accident, it would be a pain to debug). Maybe we could invent new > syntax to signal this behavior, though that might defeat the purpose. I think it can be made close enough to backwards compatible, although I'm not greatly opposed to new syntax if it's thought desirable. Essentially the idea is this: If the loop body contains a def or lambda that captures the loop variable, then on each iteration, a new binding is created, instead of changing an existing binding. Note that this is *not* the same as introducing a new scope. All the bindings share the same name, and from the perspective of code outside the nested function, nothing changes -- the name is still bound to the most recent value, and can still be accessed after the loop has finished. In CPython, it would be implemented by creating a new cell for each iteration, instead of changing the value of an existing cell. For other implementations, I don't know -- it would depend on how those implementations currently deal with nested functions. Do we need new syntax? My feeling is, probably not. I believe that the vast majority of existing code, perhaps all of it, would be unaffected by this change. All the existing workarounds for the current behaviour would continue to work. The only things that would break would be code that somehow relies on capturing the same instance of the loop variable every time, and I find it hard to imagine code that needs to do that. If we do want new syntax, my suggestion would be for new x in some_iterator: ... The downside of requiring special syntax is that we would still regularly get people asking why their lambdas in for statements don't do what they expect. We would have a better answer for them, but the questions wouldn't go away. -- Greg

56 minutes

adding a timeit.Timer context manager and decorator to time code/functions

by sdementen＠gmail.com

When optimizing code, I often need to timeit a part of code or a function. I am using then the following class ``` import logging import time from functools import wraps from typing import Optional _logger = logging.getLogger("Timer") class Timer: def __init__(self, description: Optional[str] = None, logger: Optional[logging.Logger] = None): self.description = "<anonymous code block>" if description is None else description self.logger = _logger if logger is None else logger self.msg = "'%(description)s' ran in %(time).3f seconds" def __enter__(self): self.start = time.perf_counter() return self def __exit__(self, *args): self.end = time.perf_counter() self.interval = self.end - self.start self.logger.debug( self.msg, {"description": self.description, "time": self.interval} # f"Thread: '{get_ident()}' '{self.description}' ran in {round(self.interval, 3)} seconds" ) def __call__(self, f): # decorator call @wraps(f) def wrapper(*args, **kw): with Timer(description=f.__qualname__): return f(*args, **kw) return wrapper ``` that I can use either as a context manager in ``` with Timer("how long does this take?") as t: time.sleep(1) # output: DEBUG:Timer:'how long does this take?' ran in 1.000 seconds ``` or as a decorator ``` @Timer() def my_function(): """this is my function""" time.sleep(2) my_function() # output: DEBUG:Timer:'my_function' ran in 2.000 seconds ``` This class could be added to the timeit module as it fits well its functional scope.

7 hours, 11 minutes

Move semantics

by 3mir33＠gmail.com

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

1 day, 3 hours

Getting rid of FOR loops and simplifying cicular conviolutions with Circular Indexing

by Mathew M. Noel

If circular indexing is used then instead of using a double FOR loop to go through a list M times we can iterate from 0 to M*N (where N is the length of the list) !!! Almost all Machine Learning (ML) algorithms iterate for some predefined epochs over a large data-set. So a double FOR loop is extremely common in ML. Using circular indexing gets rid of this extra FOR loop. If we have to iterate 2 times you can iterate using range(-n,n) but in most cases you need to iterate over 10 or more epochs in ML. Most scientific applications of Python involve an outer FOR loop which progressively refines an approximation with an inner FOR loop by going through a list of items. So circular indexing is useful. In the following I discuss increasingly compelling reasons for adopting a circular indexing scheme in Python. Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc). If circular indexing is used then we don't need the arbitrary rule that -1 is the index of the last element. Since -1 is the same as n-1 automatically in modular arithmetic. A trivial objection: "why not use list_name[i%n] whenever we need this feature?" By the same token we could do away with negative indices and use -1%n for example when we need to index with -1! Its unclear why that people have an irrational preference for indices that lie to the left of 0 while strongly rejecting the idea of indices that lie to the right of n-1! Python does not raise a "index out of bound" exception for negative indices like other programming languages. If this negative indexing is a "feature" (although it allows some fatal errors to slip) then indices above n-1 can also be considered a feature! Are there any deep mathematical reasons for adopting circular convention? Circular convolution is a most important operation in a wide variety of scientific disciplines since the Discrete Fourier Transform (DFT) of the circular convolution of two signals is the product of the transforms. Because of the universal applicability of Fourier ideas in science and the close mathematical relationship between finite length and periodic sequences circular indexing is extensively used in signal processing and mathematics. We can extend the idea of circular indexing to multidimensional arrays. A 2D array can be folded into a cylinder for indexing. Further this cylinder can be folded into a toroid to reduce a triple FOR loop to a single FOR loop. A deep mathematical justification for cylindrical indexing of 2D and in general nD arrays is offered by the fact that n-dimensional DFT reduces n-dimensional circular convolution to element-wise multiplication.

2 days, 7 hours

A python_include() for cgi programming

by warp_drive enterprise

I have recently have the opportunity to work on a Python web application. I've found it beneficial to reduce duplicate code with the following code lines import textwrap from pathlib import Path exec(textwrap.dedent(Path('myfile.py').read_text())) Perhaps such code could be the basis for a python_include() statement similar to how includes, requires work in PHP. Thank you for any consideration.

2 days, 10 hours

Circular Indexing

by Mathew M. Noel

Python uses an index of -1 to index the last element in a list. Since -1 occurs before 0 we might think of the elements of the linear list are being bent into a circle making the last element occur before the 0th element. Consider a list with n elements: it would be perfectly reasonable to address the element 0 of the list using an index of n since n occurs after n-1 (if we assume that the list is bent into a circle). This feature can prove to be extremely useful. Consider the following example: days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"] It would be nice if days_of_the_week[0] is the same as days_of_the_week[7] is the same as days_of_the_week[14] etc In other words use modular indexing. In other words if the index is outside the range 0 to n-1, we simply take the remainder when the index is divided by n as the index. Because of the close relationship between finite length sequences and periodic sequences this feature might simplify scientific computing(circular convolution etc).

2 days, 16 hours

async types?

by Ben Avrahami

Hi all, this is a general feeler for if this idea has any traction: All too often I see the following pattern in asyncio 3rd-party libs, either in their own source code or in the inusage: ``` inst = SomeClass() await inst.initialize() ``` What happens here is that, since coroutines cannot be used inside __init__ methods, the __init__ method only stores the parameters, and another, asynchronous method actually initializes the object. This led many 3rd-party libs to use factory methods to create their class instances, which is unideal for both users and developers. We see this pattern in nearly any async 3rd-party lib I came across. To solve this, I propose a new metaclass: `AsyncType`. The core difference between AsyncType and type is that when called, AsyncType will produce a coroutine that asynchronously calls __new__ and __init__, and returns the instance, allowing simply for `instance = await SomeClass()`. In classes of `AsyncType`, the __new__ and __init__ methods must be async methods (returning coroutines). As an additional syntactic sugar, we could also have an `async class(base)` declaration that implicitly sets the class's metaclass to be AsyncType (or raise a TypeError if that is impossible). This proposal is obviously incomplete, and there are many open questions: what about __del__? How would one easily make an async ABC (without a metaclass conflict)? How would an async class easily inherit from a sync class (suppose sync class A implements __new__, and async class B(A) implements __init__)? Perhaps `AsyncType` should only make the __init__ method async and leave __new__ synchronous? I'd just like to get ahead of the (not unjustified) argument that constructors should be lightweight: 1. Python's __init__ is not strictly a constructor. It is, as the name implies, an initializer, that makes the type usable. Constructing the type without initializing it makes no sense and has no usages, so why seperate the operations? 2. We can see numerous examples of (syncronous) connect-on-initialize classes in the standard library, including ftplib, smtplib, and pretty much most *lib modules in the standard library "Internet Protocols and Support" doc page. 3. The "initialize in a separate method" pattern produces some strange-looking code where the class holds on to its initialization params even though it doesn't really need them after the separate methods. The classes should now also consider and safe-guard against cases where the separate method is called twice, or not at all. What does the community think about this idea?

3 days, 7 hours

Circular Indexing and FOR loop minimization

by Mathew M. Noel

As discussed earlier on the post on 'Circular Indexing', considering interpreting indices that lie outside the range 0 to n-1 modulo n for list indexing. Thus element n corresponds to element 0, n+1 to element 1 and so on. This has two consequences: 1.) Makes the Python convention of interpreting index -1 as the last element more logical since -1 is the same as n-1 when taken modulo n. If -1 is the first element then to be logically consistent we have to interpret index n as the element 0 as well! Obviously this does allow certain errors to slip but if you are going to allow an index of -1 (indices that occur before 0) then you might just as well allow indices above n-1 to be logically consistent. 2.) If circular indexing is used then instead of using a double FOR loop to go through a loop twice we can iterate from 0 to 2n ! Although trivial for the case of one dimensional lists/arrays, cylindrical indexing (for 2D arrays) and toroidal indexing (2D, 3D and nD arrays) schemes might be worth exploring for n-dimensional NumPy arrays. Circular and in general cylindrical or toroidal indexing schemes might simplify coding by reducing the number of FOR loops when iterating over nD arrays in NumPy.

3 days, 15 hours

Packaging Apps (Was Adding PyInstaller to the stdlib)

by Abdur-Rahmaan Janhangeer

Greetings list, > Then I would suggest starting a new thread, Long overdue i guess. My view on the topic is like Mr Moore with the exception that i'm for native capabilities to be in the stdlib. Since Zipapp does not seem to go the C-extension route, i propose a new tool that turn projects into independent executables This solves the issue of Gui etc etc etc etc etc etc Mr Moore's mail of Nov 24, 2020 has ton of ideas and a lot of potential Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius

3 days, 18 hours

A pypi-based app store solution for platform specific installation

by Ricky Teachey

I was reading the pyinstaller thread and had this idea but didn't want to hijack. Maybe a wild idea, and very possible totally impractical or hopelessly complex, but: could the existing pypi infrastructure be leveraged into a set of platform-specific app stores? Maybe maybe we could make it as simple as a single line in a pyproject.toml file. Imagine if all the end user does is install the store, and clicks the link to the project app. The store takes care of the rest. The developer marks their package as an installable program to be published to the store, optionally specifying things like specific platforms if needed. All the project website does for users to install is provide a url to their app in the store. Very very rough idea of how it might work: a store app API would provide an installation GUI if desired, and the store would build the environment needed for installing the package and dependencies. Once the specified environment is built, pip would take care of installing the package as usual. If this is an awful idea feel free to just say so. I have given it no deep thought and do not have the expertise to even think through what would be needed to create such a thing.

3 days, 19 hours

Jump to page:

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas