Python-ideas November 2020

python-ideas@python.org

85 participants
52 discussions

Add command-line option to unittest for enabling post-mortem debugging
by Dominik Vilsmeier 09 Jan '21

09 Jan '21

Consider the following example: import unittest def foo(): for x in [1, 2, 'oops', 4]: print(x + 100) class TestFoo(unittest.TestCase): def test_foo(self): self.assertIs(foo(), None) if __name__ == '__main__': unittest.main() If we were calling `foo` directly we could enter post-mortem debugging via `python -m pdb test.py`. However since `foo` is wrapped in a test case, `unittest` eats the exception and thus prevents post-mortem debugging. `--failfast` doesn't help, the exception is still swallowed. Since I am not aware of a solution that enables post-mortem debugging in such a case (without modifying the test scripts, please correct me if one exists), I propose adding a command-line option to `unittest` for [running test cases in debug mode](https://docs.python.org/3/library/unittest.html#unittest.TestCase.deb… so that post-mortem debugging can be used. P.S.: There is also [this SO question](https://stackoverflow.com/q/4398967/3767239) on a similar topic.

1 1

Python JIT Compilation Thoughts
by redradist＠gmail.com 05 Jan '21

05 Jan '21

Hi all, I do not know maybe it was already discussed ... but the toolchain like LLVM is very mature and it can provide the simpler JIT compilation to machine code functionality and it will improve performance of the Python a lot !!

9 15

Add decorator_with_params function to functools module
by Yurii Karabas 24 Dec '20

24 Dec '20

In the current python codebases, there are a lot of decorators with parameters, and their code in most cases contains a lot of code boilerplates. The purpose of this feature is to add `decorator_with_params` (I think the name can be changed to smth better) function to `functools` module. This function will allow using a decorated function as a simple decorator and decorator with parameter. I believe the real example will bring more context, for instance, `typing._tp_cache` function can be rewritten from: ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` To ``` @decorator_with_params def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ``` And `_tp_cache` can be used as: ``` @_tp_cache(typed=True) def __getitem__(self, parameters): return self._getitem(self, parameters) ... @_tp_cache def __getitem__(self, parameters): return self._getitem(self, parameters) ``` Proposed implementation is quite simple (I used it in a lot of projects): ``` def decorator_with_params(deco): @wraps(deco) def decorator(func=None, /, *args, **kwargs): if func is not None: return deco(func, *args, **kwargs) def wrapper(f): return deco(f, *args, **kwargs) return wrapper return decorator ``` I believe it will be a great enhancement for the Python standard library and will save such important indentations. I have already opened an issue https://bugs.python.org/issue42455 (Sorry about that I didn't know that smth like python-ideas exists).

3 5

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Add the brotli & zstandard compression algorithms as modules
by Omer Katz 14 Dec '20

14 Dec '20

Hello everyone, We have many compression algorithms as standard modules, but we lack two important new algorithms: brotli and zstandard. Brotli is an important compression algorithm both for HTTP2 and for compressing/decompressing HTTP responses. ZStandard is another algorithm for compression that can easily beat gzip with a better compression ratio and less compression time. Both are now stable enough and are past the 1.0 mark. Let's include them in the standard library and, of course, make them optional at build time like the rest of our compression modules. What do you think?

12 21

Experimental syntax proposal
by André Roberge 05 Dec '20

05 Dec '20

# Experimental Syntax Proposal I would like to propose that Python adopts a modified process before introducing significant changes of its syntax. ## Preamble Given the following file: ```python # coding: experimental-syntax from experimental-syntax import fraction_literal from experimental-syntax import decimal_literal assert 1 /3F == Fraction(1, 3) assert 0.33D == Decimal('0.33') print("simple_test.py ran successfully.") ``` This is what happens when I run it, with the standard Python interpreter: ``` $ python simple_test.py simple_test.py ran successfully. ``` In what follows, I use this as a concrete example for one of the three possible options mentioned. ## The problem Python evolves in many ways including with the addition of new modules in the standard library and with the introduction of new syntax. Before considering the addition of a module in the standard library, it is often suggested to have a version on Pypi so that users can experiment with it, which can lead to significant improvements. However, when it comes to proposed syntax changes, this is currently not possible to do, at least not in any standard way that would immediately be easily recognized as such by the wider community. ## Proposed solutions For those that agree that this is something that could be improved upon, I see at least three possible solutions. 1. Adoption of a simple convention to identify the use of non-standard syntax. This could take the form of a comment with a specific form introduced near the top of the file. This comment could be used as a search term, to identify modules or projects that make use of some custom syntax. 2. Addition of a special encoding module in the Python standard library that can be used to implement nonstandard syntax through source transformation. The short program written at the top provides an example of what this might look like. Note the comment at the top defining a special codec, which could also 3. Addition of a standard import hook in the standard library which could be used, like the special encoding example above, to implement syntactic changes. By doing searches in standard locations (Github, Gitlab, etc.) on, one could identify how many modules are making use of a given experimental syntax, giving an idea of the desirability of incorporating into Python. Imagine how different the discussion about the walrus operator would have been if people had had the opportunity to try it out on their own for a few months prior to a decision being made about adding it to Python. ## New Syntax: two approaches There are currently at least two ways in which one can write a Python program that can use some non standard syntax: * By using an import hook. This is the most powerful approach as it allows one to do changes either at the source level (prior to parsing), or at the AST level, or both. * By using a custom encoding. This only allows transformations at the source level. Both approaches currently require a two-step process. To use an import hook, one has first to load a function that sets up this import hook before importing the module that contains the experimental syntax. This makes it impossible to simply write ``` python module_with_new_syntax.py ``` and have it being executed. Import hooks, like the name implies, only apply to modules being imported - not to the first module executed by Python. With a custom encoding, it is however possible to first register a custom codec either via a site.py or usercustomize.py file. Once this is done, the above way of running a module with new syntax is possible, provided that the appropriate encoding declaration is included in the script. This is what I did with the example shown above. ### Current limitation of these two approaches Using an import hook for enabling new syntax does not work for code entered in the standard Python REPL, which limits the possibility of experimentation. While I have read that specifying the environment variable `PYTHONIOENCODING` enables the Python REPL to use a custom encoding, I have not been able to make it work on Windows and confirm that it is indeed possible to do so. However, I have been able to use a simple customized REPL that can make use of the above. Perhaps the standard REPL could be modified in a similar way. ## New suggested process Assuming one of the options mentioned above is adopted, before changes to the syntax are introduced in a Python version, they would be made available to be used either as an encoding variant or an import hook giving enough time for interested pythonistas to experiment with the new syntax, writing actual code and possibly trying out various alternatives. ## Proof of concept As a proof of concept shown at the beginning of this post, I have created two tiny modules that introduce new syntax that had been discussed many times on this list: 1. a new syntax for decimal literals 2. a new syntax for rationals Both modules have been uploaded separately to Pypi; I wanted to simulate what could happen if a proposal such as this one were adopted. To make use of these, you need to use the `experimental-syntax` codec found in my `ideas` package. To run the example as shown above, you first need to register the codec, which can be done by using either the `site.py` or `usercustomize.py` approach. I chose the latter, by setting the environment variable `PYTHONPATH` to be a path where the following `usercustomize.py` file is found. ```python from ideas import experimental_syntax_encoding print(f" --> {__file__} was executed") ``` Doing this on Windows, I found that it did not seem to work when using a virtual environment (I added the print statement to confirm that it was loaded). Here's a sample session using the code available currently on pypi. ``` C:\Users\andre\github\ideas>python simple_test.py --> C:\Users\andre\github\ideas\usercustomize.py was executed simple_test.py ran successfully. C:\Users\andre\github\ideas>python --> C:\Users\andre\github\ideas\usercustomize.py was executed Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:30:28) [MSC v.1926 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from ideas import experimental_syntax_encoding >>> from ideas import console >>> console.start() Configuration values for the console: transform_source: <function transform_source at 0x00A8AE38> -------------------------------------------------- Ideas Console version 0.0.19. [Python version: 3.8.4] ~>> import simple_test simple_test.py ran successfully. ~>> from experimental-syntax import decimal_literal ~>> 3.46D Decimal('3.46') ~>> from experimental-syntax import fraction_literal ~>> 2/3F Fraction(2, 3) ~>> ``` Installation: python -m pip install ideas # will also install token-utils python -m pip install decimal-literal python -m pip install fraction-literal More information about "ideas" can be found at https://aroberge.github.io/ideas/docs/html/ It has not (yet) been updated to include anything about the `experimental_syntax_encoding` module, which I just worked on today. ## Final thoughts This proposal is really about the idea of adopting a standard process of some sort that enables users to experiment with any proposed new syntax, rather than the specific "silly" examples I have chosen to illustrate it. André Roberge P.S. If anyone knows how to make a usercustomize file run in a virtual environment on Windows, please let me know.

4 7

Making the for statement work better with nested functions
by Greg Ewing 05 Dec '20

05 Dec '20

Starting a new thread for this as suggested by Guido. On 18/11/20 7:20 pm, Guido van Rossum wrote: > On Tue, Nov 17, 2020 at 22:12 Greg Ewing <greg.ewing(a)canterbury.ac.nz > <mailto:greg.ewing@canterbury.ac.nz>> wrote: > > If there's anything I would change, it would be to have the for > statement create a new binding on each iteration, so that capturing > it with a def or lambda inside the loop works as expected. I even > came up with a way to do that while still allowing the last-bound > value to be seen afterwards, but the idea didn't gain any traction. > > I wonder if we should try that idea again? The problem may be backwards > compatibility — there’s always someone who depends on such a thing (even > if by accident, it would be a pain to debug). Maybe we could invent new > syntax to signal this behavior, though that might defeat the purpose. I think it can be made close enough to backwards compatible, although I'm not greatly opposed to new syntax if it's thought desirable. Essentially the idea is this: If the loop body contains a def or lambda that captures the loop variable, then on each iteration, a new binding is created, instead of changing an existing binding. Note that this is *not* the same as introducing a new scope. All the bindings share the same name, and from the perspective of code outside the nested function, nothing changes -- the name is still bound to the most recent value, and can still be accessed after the loop has finished. In CPython, it would be implemented by creating a new cell for each iteration, instead of changing the value of an existing cell. For other implementations, I don't know -- it would depend on how those implementations currently deal with nested functions. Do we need new syntax? My feeling is, probably not. I believe that the vast majority of existing code, perhaps all of it, would be unaffected by this change. All the existing workarounds for the current behaviour would continue to work. The only things that would break would be code that somehow relies on capturing the same instance of the loop variable every time, and I find it hard to imagine code that needs to do that. If we do want new syntax, my suggestion would be for new x in some_iterator: ... The downside of requiring special syntax is that we would still regularly get people asking why their lambdas in for statements don't do what they expect. We would have a better answer for them, but the questions wouldn't go away. -- Greg

17 76

Making "Any" a builtin
by Abdulla Al Kathiri 02 Dec '20

02 Dec '20

Instead of importing “Any" from the typing module, we can annotate our functions with “Any" right away without the extra step. What do you think? We have the builtin function “any” which some Python users could mistakingly use, but static type checkers should catch that.

8 14

Re: Making the for statement work better with nested functions
by Paul Sokolovsky 30 Nov '20

30 Nov '20

Hello, On Mon, 30 Nov 2020 12:26:53 +0000 Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > On 30/11/2020 08:01, Paul Sokolovsky wrote: > > Hello, > > > > On Sun, 29 Nov 2020 18:53:57 -0800 > > Christopher Barker <pythonchb(a)gmail.com> wrote: > > > >> This is a massively less ambitious idea, and doesn't solve the > >> original problem, but: > >> > >> I'd like to see a scope for names that are "obviously ;-)" meant > >> to be short lived. At the moment that's the for loop "counters" > >> and those created by context managers: > >> > >> for i in something: > >> # use i as usual > >> more_code > >> # now i is not defined > > What other language(s) implement such a scoping discipline? I know > > of none. > > > > On the other hand, block-scoped variables are implemented in: > > > > * C > > * C++ > > * Java > > * Rust > > * Lua > > * JavaScript (not by default, as opt-in) > > * Scheme > > * Common Lisp (as opt-in) > > * ML > > * Ocaml > > * Haskell > > * very long list of other languages... > > > > > > The aim of the block scoping proposal is to make Python *not worse* > > than these other languages, instead of adding funny workarounds > > again. > > > > [] > > > > CHB's proposal may or may not be a good idea. But please treat it on > its merits, Absolutely. I appreciate all the discussion and alternative ideas people put into this. I, on my side, may have a "strong opinion" that ideas I utter are "right", but that doesn't mean I'm not mistaken. These matters really require good consideration from as wider as possible Python community. > instead of saying, as you seem to be, "No other language > does this, so it's bound to be a bad idea which will make Python > worse". That's why I try to choose words carefully ;-). I said that if Python does it like many other languages, it definitely won't be worse than them. I said nothing what happens if it doesn't do it like that ;-). All in all, it's just an argument, as anyone else's. But I do think about those matters, I have to admit. And I'd hate yet another advanced Python programmer to leave for Haskell, Go, Rust, Julia, because, from their PoV, Python has a culture of applying workarounds instead of best-practice solutions. > With that attitude, there would never be any innovation in > language design. I'm first to subscribe under that. Recent example: Pattern Matching, a common mantra is "we did it like that because other languages do it like that" (example: https://mail.python.org/archives/list/python-dev@python.org/message/EWX3GWN…) My response: But did you consider Python's situation? Both "minor" issues, like lack of constants, which requires ugly workarounds, and overall audience of Python? Was the option to use sigils to mark capture (i.e. assignment) targets *really well* considered? They said yes. Ok. Works for me, and I tried to do as much noise as possible to draw other people's attention to that. > Best wishes Rob Cliffe [] -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

adding a timeit.Timer context manager and decorator to time code/functions
by sdementen＠gmail.com 30 Nov '20

30 Nov '20

When optimizing code, I often need to timeit a part of code or a function. I am using then the following class ``` import logging import time from functools import wraps from typing import Optional _logger = logging.getLogger("Timer") class Timer: def __init__(self, description: Optional[str] = None, logger: Optional[logging.Logger] = None): self.description = "<anonymous code block>" if description is None else description self.logger = _logger if logger is None else logger self.msg = "'%(description)s' ran in %(time).3f seconds" def __enter__(self): self.start = time.perf_counter() return self def __exit__(self, *args): self.end = time.perf_counter() self.interval = self.end - self.start self.logger.debug( self.msg, {"description": self.description, "time": self.interval} # f"Thread: '{get_ident()}' '{self.description}' ran in {round(self.interval, 3)} seconds" ) def __call__(self, f): # decorator call @wraps(f) def wrapper(*args, **kw): with Timer(description=f.__qualname__): return f(*args, **kw) return wrapper ``` that I can use either as a context manager in ``` with Timer("how long does this take?") as t: time.sleep(1) # output: DEBUG:Timer:'how long does this take?' ran in 1.000 seconds ``` or as a decorator ``` @Timer() def my_function(): """this is my function""" time.sleep(2) my_function() # output: DEBUG:Timer:'my_function' ran in 2.000 seconds ``` This class could be added to the timeit module as it fits well its functional scope.

6 9

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas November 2020