Mailman 3 December 2020 - Python-ideas

Using "&" to combine different annotations (replacement for typing.Annotated)
by Paul Sokolovsky 23 Dec '20

23 Dec '20

Hello, Some time ago we discussed with Steven D'Aprano how "const" annotation would be combined with other annotations: https://mail.python.org/archives/list/python-dev@python.org/message/SQTOWJ6… I mentioned that while current way is to use `typing.Annotated`: spam: Annotated[Widget, const] = build_widget(*args) , it looks ugly, and we can shoot for better. I proposed to use "&" to combine annotations: spam: Widget & const = build_widget(*args) Today I came by another new proglingo, https://flix.dev/ , which uses Python-ish syntax and just very this idea, e.g.: /// We can also declare an impure function. def printAndInc(x: Int): Int & Impure = Console.printLine("Hello"); x + 1 It's always helpful to look at some idea "from outside". And well, I'm sold on it! (As if I wasn't before.) -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

Post: Vague possibility of tracing context manager function calls [with lock objects]
by Mohamad Kanj 23 Dec '20

23 Dec '20

Hi, Any new scope entered or exited could be traced by defining a custom tracing function and setting it using the sys <https://docs.python.org/3/library/sys.html> module through sys.settrace() <https://docs.python.org/3/library/sys.html#sys.settrace> or sys.setprofile() <https://docs.python.org/3/library/sys.html#sys.setprofile>. This is also true for multithreaded applications using the threading <https://docs.python.org/3/library/threading.html> module. Thus, it is rather vague whether the __enter__() and __exit__() functions executed by a with statement could be traced, and if so, in what cases. For instance, _thread.LockType or simply primitive lock objects from threading being used with the context management (i.e. with statement) might not allow us to trace a lock being acquired or released. In other words whether the __enter__() function that calls acquire() is called. After some experimentation, unfortunately this is not the case if either Python or C functions (for CPython) are traced. So the question whether with-statements could be traced by the python interpreter ought to be clearly stated since in some cases it works and in some doesn't. _____________________________________________________________________________ *EXAMPLE:* import threading import sys def tracer(frame, event, arg): if event == 'call': print(f"Entering: {frame.f_code.co_name}") return tracer elif event == 'c_call': print(f"C --- Entering: {arg.__name__}") elif event == 'return': print(f"Returning: {arg!r}") elif event == 'c_return': print(f"C --- Returning from: {arg.__name__}") lock = threading.Lock() sys.setprofile(tracer) threading.setprofile(tracer) print("Begin...") # In the following case, we would see the C functions # being called: acquire and release lock.acquire() print("Do some routine with explicit locking") lock.release() print("In the middle...") # In the following case, we cannot see that any locking is happening, # nor if the __enter__() and __exit__() methods called with lock: print("Do some routine with locking using context management") print("End...") _______________________________________________________________________________________ Best, Mohamad

1 0

Possibility to decorate single code line or code block?
by Marco Sulla 23 Dec '20

23 Dec '20

Maybe it's a crazy idea, but what if we could decorate a single line of code? For example: @Timer a = time_consuming_function() This will be equivalent to using Steven's context manager, but the decorator is more simple to comment and uncomment. Maybe it could be possible to comment also code blocks: @parallel for x in y: do_something(x) now_something_completely_different(x) This could be a shortcut for multiprocessing.

5 9

REPL preview / throwOnSideEffect / Sandboxing
by Jun Wu 23 Dec '20

23 Dec '20

Context: I was thinking about having embedding Python in a REPL with live preview of evaluation result of pure functions. Similar to nodejs REPL or Chrome DevTools (which uses {throwOnSideEffect: true} under the hood). This is somewhat similar to sandboxing. I'm aware of tricks like `object.__subclasses__` to get "hidden" types, and `type(func)(...)` to construct functions from bytecode. Currently I'm approaching it slightly differently and I didn't find previous discussions. If we define "side effects" as effects outside the current process (ex. filesystem), and allow mutation of in-process global state (ex. `sys.modules`). Then it seems all pure Python code is side-effect free, and only those C functions (ex. `open`) need to be checked. In this world, we allow arbitrary pure Python code including constructing a Python function from bytecode, as long as native functions are checked. Looking at existing APIs, I think `PyEval_SetProfile` can be abused to achieve that kind of "throw on side effect" easily - when the profile function gets a `PyTrace_C_CALL` event, it can examines the C function and decide to raise if necessary. The missing piece is to define the whitelist of side-effect free native functions, such as `sorted`, `join`, etc. I wonder what people think about this approach, and whether this is worth some sort of core support (ex. native functions whitelist per `PyThreadState`, builtin function to run another function in restricted mode). I can envision this to be useful in IPython, potentially interesting to other applications wanting a "declarative", "pure" scripting language, especially when the embedding applications can define their types and tweak the list of side-effect free functions.

1 0

Standalone bool?
by Paul Bryan 22 Dec '20

22 Dec '20

I know this has come up in the past. Could the consensus have changed regarding bool's inheritance from int? This is not intuitive (to me), and recently bit me as a bug: Python 3.9.1 (default, Dec 13 2020, 11:55:53) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> {1,True} {1} I've worked around it by storing tuple(type, value) in the set, which is fugly. Yes, I need mixed types in the set, and I'm using the set for efficient lookup. A contrived example I dreamed-up, which I also find non-intuitive: >>> x = {} >>> x[True] = 1 >>> x {True: 1} >>> x[1] = 1 >>> x {True: 1} Maybe a wish list item for Python 4.0?

11 28

Add something to `yield from` to allow delegating to an already started generator
by Paolo Lammens 22 Dec '20

22 Dec '20

Currently, there is no way to correctly delegate to a partially consumed generator with `yield from` without re-implementing the entire semantics of `yield from` in user code. (In particular, I'm referring to full-fledged bidirectional generators that make use of .send(), .throw(), and .close().) This is because `yield from` always `send`s `None` as the first value—and this is consistent with how freshly created generator iterators work, but if the generator iterator that is being yielded from has already been partially consumed, one might need to start the `yield from` with a specific value. Here I propose to add something to the language/stdlib to achieve this (without the user having to re-implement `yield from`). Let me illustrate what I mean. Suppose we have an arbitrary "bidirectional" generator function `subgen`, and we want to wrap this in another generator `gen`. The purpose of `gen` is to intercept the first yielded value of `subgen` and delegate the rest of the iteration/generation to the sub-generator. The first thing that might come to mind could be this: ```python def gen(): g = subgen() first = next(g) # do something with first... yield "intercepted" # delegate the rest yield from g ``` But this is wrong, because when the caller `.send`s something back to the generator after getting the first value, it will end up as the value of the `yield "intercepted"` expression, which is ignored, and instead `g` will receive `None` as the first `.send` value, as part of the semantics of `yield from`. So we might think to do this: ```python def gen(): g = subgen() first = next(g) # do something with first... received = yield "intercepted" g.send(received) # delegate the rest yield from g ``` But what we've done here is just moving the problem back by one step: as soon as we call `g.send(received)`, the generator resumes its execution and doesn't stop until it reaches the next yield statement, whose value becomes the return value of the `.send` call. So we'd also have to intercept that and re-send it. And then send *that*, and *that* again, and so on... So this won't do. At this point it is clear that no number of manual `yield` and `send`s will solve this, because `yield from` will always send `None` as the first value, while instead we would like to send in whatever the caller sent last (at the last `yield` expression). So there is no way to use the `yield from` syntax directly; one has to implement its semantics anew with something like this: ```python def adaptor(generator, init_send_value=None): send = init_send_value try: while True: send = yield generator.send(send) except StopIteration as e: return e.value ``` (And then `yield from adaptor(g, init_send_value=received)`.) This is not even a good implementation because it's missing handling of .throw(), .close(), etc. So my proposal is to either add a piece of syntax to `yield from` so that one can specify a starting value other than `None`, e.g. ```python yield from <generator iterator> with <initial value> ``` or add a standard library function like `adaptor` above (but correctly implemented).

1 0

What if exec() took a module object directly?
by Paul Sokolovsky 22 Dec '20

22 Dec '20

Hello, I posted on python-dev a question regarding builtin vars() vs .__dict__ attribute dichotomy: https://mail.python.org/archives/list/python-dev@python.org/thread/JAFIBBUU… It could as well be one among the earliest cases of the violation of "There should be one-- and preferably only one --obvious way to do it." Such earlier, that it could as well turn out that this principle was never thoroughly followed by Python at all. Anyway, how the question pops up, is that there's very rare when there's a need to access *writable* object namespace as dictionary. One of such cases is executing code in a module context, where you need to pass module's globals to exec(), and so I wonder what's "the most canonical way" of getting that - var(mod) or mod.__dict__. But thinking about it, the problem lies on the exec()'s side. If exec could take a module object directly as its "globals" param, there wouldn't be a need to expose internal namespace implementation of the module object. So, I wonder what old-timers would think of the following signature: exec(object[, globals_or_module[, locals]]) https://docs.python.org/3/library/functions.html#exec -- Best regards, Paul mailto:pmiscml@gmail.com

2 2

Capture manylinux environment set-up in pyproject.toml
by christopher.antonellis＠gmail.com 21 Dec '20

21 Dec '20

The manylinux project does not specify a way to record "environment set up" information required to build wheels from certain packages. Some packages require that the stock manylinux environment be modified by installing system dependencies, setting environment variables, modifying the filesystem etc prior to building, or the building and bundling will fail. Currently, package authors capture this information in project specific documentation, or not at all. In the event that a wheel is not provided by the author, consumers that want a compiled distribution (users focused on reproducible builds, build efficiency) are required to locate this information in project specific documentation or do research on the internals of the package to gather this information themselves. Currently, package consumers can not easily modify or contribute to this data as it is not standardized. What are everyone's thoughts about adding a section to `pyproject.toml` to capture this information? This allows package consumers to easily find and contribute back to this data, and programs like installers, builders etc can read this data as well. Is this PEP-able or where could this best be contributed? I am interested in (and intend on) contributing to solving this problem if it is accepted as a good idea! Thank you for your input! Something like this: ```toml [manylinux1] system_dependencies: - foo-1.0.0 environment_variables: - FOO=BAR python_packages: - foo==1.0.0 ``` - Chris Antonellis

1 0

Expose the count of acquires in asyncio.Semaphore
by Paolo Lammens 21 Dec '20

21 Dec '20

Sometimes it is useful to get the number of tasks currently holding the semaphore. Currently, assuming that every call to acquire is paired with a call to release in the same task (e.g. the semaphore is only used in [async] with statements), one can do INITIAL_VALUE - sem._value where INITIAL_VALUE is the value that was used to construct the semaphore and sem is the semaphore object. But, as the name indicates, `sem._value` is intended for internal use only. I would proposing exposing this value "officially" by adding a property: @property def count(self): return self._initial_value - self._value The implementation would probably have to be a bit more sophisticated than this, because there can be spurious calls to .release() that aren't actually releasing the semaphore because it was never acquired, and it would need to be decided whether reentrant acquires of the semaphore in the same task are counted separately or not. Perhaps this could be added only to asyncio.BoundedSemaphore.

1 0

Typed Python execution mode
by redradist＠gmail.com 20 Dec '20

20 Dec '20

It would be nice to have "Typed Python" mode that will look like this: ```bash #!/usr/bin/env bash set -e python -m mypy $1 python $1 ``` https://gist.github.com/redradist/dd7253a55081a4dc13fdf3f1549f43b5 It could be achieved by adding special flag like `-t` (typed)

6 15