Mailman 3 July 2020 - Python-ideas

Allowing -b (BytesWarning) to be activated in other ways
by Shai Berger 16 Jul '20

16 Jul '20

Hi Pythonistas, The -b flag, which turns on checks which emit BytesWarnings on operations mixing bytes and str objects, is very useful. However, the only way to set this flag is via the Python invocation. This limits its usability in contexts where the user's control of the Python invocation is limited, e.g when using Python embedded in another executable (such as uwsgi). There appears to be no function which can set the flag, and no environment variable which controls it. Up to Python 3.7, the extension module provided by the bytes-warning package[1] works around this (it exposes a function which allows setting the flag from within Python). But with 3.8 (and I suspect, because of PEP-587[2] related changes), this fails silently and the sys flag bytes_warning remains unaffected. Can we have a non-invocation way to control this flag? Thanks, Shai. [1] https://pypi.org/project/bytes-warning/ [2] https://www.python.org/dev/peps/pep-0587/

4 3

Missing link to git repo on the “Source code” page
by Hans Ginzel 16 Jul '20

16 Jul '20

Hello, there are links to each releases on the “Source code” page, https://www.python.org/downloads/source/. But I am missing a ling to the (official) git repository, e.g. https://github.com/python or which one is it. Thank you in advance, Hans

2 2

python interpreter docker builder
by Antonio Cavallo 15 Jul '20

15 Jul '20

Hi is there any interest (or anyone has done it before), building the python interpreter using docker? The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows. Thanks

5 15

[Feature] Body closure
by redradist＠gmail.com 13 Jul '20

13 Jul '20

Today I think about lambda in Python and what if we introduce the new syntax: ```python def lock(*args, closure): # Do some stuff closure() # Call closure # Finish stuff if __name__ == '__main__': lock(): # Do some things here is thread safe ``` This feature could be very similar as in `Kotlin` inline functions https://kotlinlang.org/docs/reference/inline-functions.html

3 2

Add random.choice_iter for randomly selecting from an arbitrary iterable
by Oscar Benjamin 13 Jul '20

13 Jul '20

I posted this in the thread about indexing dict items but it seems to have been buried there so I'm starting a new thread. Maybe there could be a function in the random module for making a random choice from an arbitrary (finite) iterable. This implementation can choose a random element from an iterable by fully iterating over it so is O(N) in terms of CPU operations but O(1) for memory usage: # rnd.py from random import randrange def choice_iter(iterable): iterator = iter(iterable) try: value = next(iterator) except StopIteration: raise ValueError("Empty iterable") for n, candidate in enumerate(iterator, 2): if not randrange(n): value = candidate return value You could use that to get a choice from a dict, set etc: >>> choice_iter({'q', 'w', 'e'}) 'e' >>> choice_iter({'q', 'w', 'e'}) 'q' >>> choice_iter({'q', 'w', 'e'}) 'e' >>> choice_iter({'q', 'w', 'e'}) 'w' You can make a random choice from the keys, values or items of a dict. For a large dict/set it will be slow compared to converting to a list because it's calling randrange once per item. It can probably be made a lot faster by doing something like calling randbits and extracting the bits for multiple iterations of the loop. Although choice_iter is slower than choice it works for arbitrary iterables and has constant memory requirements when used with a lazy iterator. There are situations in which you would not want to build a list and use random.choice and where the computational overhead is insignificant. For example choice_iter can be used to randomly choose a line from an arbitrarily large text file without loading it entirely into memory: >>> from rnd import choice_iter >>> with open('rnd.py') as fin: ... line = choice_iter(fin) ... >>> line ' except StopIteration:\n' When reading from a large text file this is roughly optimal in performance terms because it does precisely the minimum amount of IO (one pass) while having constant memory overhead. Of course if you wanted to select multiple lines from the file then calling choice_iter repeatedly is immediately suboptimal because it would read the file more than once. It's not hard though to generalise the function above to something that e.g. makes a selection of k values from an iterable in a single pass. Here is a version that works without replacement: def sample_iter(population, k): iterator = iter(population) values = [] for _ in range(k): try: value = next(iterator) except StopIteration: raise ValueError("Too few items") values.append(value) for n, candidate in enumerate(iterator, k+1): random_index = randrange(n) if random_index < k: values[random_index] = candidate return values # maybe shuffle first? The memory cost of this is O(k) regardless of the size of the input iterable. -- Oscar

4 6

Add builtin function for min(max())
by Federico Salerno 11 Jul '20

11 Jul '20

Even after years of coding it sometimes takes me a moment to correctly parse expressions like `min(max(value, minimum), maximum)`, especially when the parentheses enclose some local computation instead of only references, and the fact that `min` actually needs the *maximum* value as an argument (and vice-versa) certainly doesn't help. It's a little surprising to me how shorthand functions akin to CSS's `clamp()` aren't more popular among modern programming languages. Such a tool is, in my opinion, even more missed in Python, what with it being focussed on readability and all. There would likely also be some (probably minor) performance advantages with implementing it at a builtins level. Example usage: >>> val = 100 >>> clamp(10, val, 50) 50 >>> val = 3 >>> clamp(10, val, 50) 10 >>> val = 25 >>> clamp(10, val, 50) 25 I'm undecided whether I would like `clamp`, `minmax` or something else as a name. I'm curious of other ideas. As far as the signature is concerned, I like both `clamp(min, val, max)` for the logical position of the three arguments (which mirrors expressions such as `min < val < max`) and `clamp(val, min=x, max=y)`. I prefer the former, but declaring them as normal positional-and-keyword arguments would allow the programmer to use an alternative order if they so choose.

27 71

Python: Different native runtime state tiers.
by William Pickard 09 Jul '20

09 Jul '20

CPython at the very least has 2 different type of native states: Interpreter & Module state. Unfortunately, the multi-phase initialization has a weakness when it comes to Module states. You can't access the module state without a pointer to the module. PyState_GetModule from a standpoint looks to be the obvious answer to use, but it's documentation states it's unfit for multi-phase initialization. I'm proposing an idea here for discussion on a new state system for at least CPython. Tier 1: Core state - This state lives within CPython's core binary and exists the entire lifetime of the binary. - The data held within this state is available to the main interpreter and subsequent sub-interpreters. (Example: sys.executable) Tier 2-1: Interpreter state (Branches from Core state) - This state lives within CPython's core binary and is tied to a specific interpreter. - The data held within this state is only available to the interpreter it's tied to. (Example: Modules loaded into memory) Tier 2-2: Extension state (Branches from Core state) - This state lives within any CPython's core binary EXCEPT it's size and structure is defined by the extension CPython has loaded. - The purpose of this state is to allow an extension to hold data that can't be tied to a specific module. (Examples can be: Windows WSA, MySQL) Tier 3-1: Thread state (Branches from Interpreter State) - This state lives within CPython's core binary and is tied to a specific Python thread (IE: Threading library threads), - The data held within this state is only available to the thread it's tied to. (No known examples available) Tier 3-2: Module state (Branches from Interpreter State) - This type of state is already available in CPython, explaining it is not required.

2 1

An alternative to using a clamp / clip / trim function
by Jonathan Fine 09 Jul '20

09 Jul '20

Hi All This is related to discussion https://mail.python.org/archives/list/python-ideas@python.org/thread/KWAOQF… In Python, lists don't have a join method. Instead, it's strings that have the join method. Hence we have: >>> ', '.join('abcde') 'a, b, c, d, e' The intermediate method we can save and store and use again. >>> joiner = ', '.join >>> joiner('fghijk') 'f, g, h, i, j, k' We can do something similar when clamping, clipping or trimming a value. For example, suppose we want limits on the room temperature, that the thermostat cannot override. >>> aircon_clipper = Clamper(50, 80) >>> thermostat_temp = 40 >>> target_temp = aircon_temp_clipper(thermostat_temp) >>> target_temp 50 What I like best about this is that the name of the function -- aircon_temp_clipper -- gives the reason for the clipping (or clamping). And this name can be chosen to suit the domain. I also like that it wraps the high and low values (50 and 80) as data used by a method. Here's an implementation of Clamper. class Clamper: def __init__(self, lo, hi): if not lo <= hi: raise ValueError self._lo = lo self._hi = hi def __call__(self, num): lo, hi = self._lo, self._hi at_least_lo = (lo <= num) at_most_hi = (hi >= num) if at_least_lo and at_most_hi: return num elif at_least_lo: return hi elif at_most_hi: return lo else: raise ValueError If performance is important to you, you could instead use a closure to create a function. This would also allow you to provide a docstring. I hope this helps at least some of the people some of the time. -- Jonathan

5 5

Add __eq__ to colletions.abc.Sequence ?
by Joao S. O. Bueno 08 Jul '20

08 Jul '20

Is there any reason why collections.abc.[Sequence|MutableSequence] do not have "__eq__" for free? I was writing some tests now and was caught by surprise there. (chcking the docs, if my custom MutbaleSequence also inherits from "collections.abc.Set" I will have working "__eq__" - and "__ne__", methods but also weirdly behaved "__le__" and cousins as a side efffect - weird enough I think i will just hand-write the "__eq__") While theoretically adding "__eq__" and "__ne__" _could_ hab backward compatibility issues, I think it is more probable it would actually fix some undetected bugs i a couple projets, as the default __eq__ is an identity comparison (is). Anyway, maybe there is a reason it is not a given. Any thoughts?

6 14

extending ast.parse with some lib2to3.pytree features
by Peter Ludemann 08 Jul '20

08 Jul '20

With lib2to3 going away (https://bugs.python.org/issue40360), it seems to me that some of its functionality for handling "whitespace" can be fairly easily added to the ast module. (By "whitespace", I mean things like indent, dedent, comments, backslash; and also the ability to manipulate the encoded bytes in the original source.) Off the top of my head, I know of the following projects that use lib2to3 or similar to access the "whitespace" in the parse tree and will need a new solution: yapf, black, mypy, pytype, kythe. (If they don't use lib2to3, they need to maintain a custom parser that changes with each release of Python, so my proposal would potentially help them.) Are there other projects that need access to the parse tree and "whitespace"? I propose implementing an optional pass over the parse tree that records lib2to3's "prefix" with each leaf node. The interface would be something like: def detect_encoding(source: bytes) -> str def parse_with_whitespace(source: bytes, encoding: str, filename: str) -> ast.Module def unparse_bytes(tree: ast.Module) -> bytes def unparse_str(tree: ast.Module) -> str # Various convenience functions/properties, similar to pytree.next_sibling etc. parse_with_whitespace() calls ast.parse(), then does a pass over the parse tree, adding to the leaf nodes: prefix: str # whitespace and comments preceding token in the input pieces: List # see below col_byte_offset: int # start byte offset within line src_byte_offset: int # start byte offset within source The "pieces" field is intended to handle things like: x = 'abc' \ "def" y = (f'abc{x}' # comment "def") Each "piece" would include: - byte offset from the beginning of the token (negative for a prefix piece) - detailed type (single-quote string, double-quote string, format-string, int, float, etc.) - source bytes - decoded value (Unicode str) The ast.Module class would also be extended with additional attributes that apply to the entire source, such as the encoding. All of this is quite fiddly, but I already have some code for dealing with the conversions between byte and string offsets, so I don't anticipate a huge amount of work. (The design is also inherently a bit inefficient; but I don't want to get involved with the internals of compile().) A related item: "Parser module in the stdlib": https://mail.python.org/archives/list/python-dev@python.org/thread/RHZ6JOEX…

1 0