Python-ideas May 2016

python-ideas@python.org

89 participants
43 discussions

Lib vs ( App | Project | ... )
by Thomas Güttler 23 May '16

23 May '16

This blog post from Donald Stufft explains the difference between a Library and a Application. https://caremad.io/2013/07/setup-vs-requirement/ The term "Library" is well known - ok But how to call the container for the libraries, the one thing providing the environment? Donald Stufft uses the term "Application" I would like an official term defined by the python docs. What do you think? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

7 9

Some food for thought here. (Just dashed this off from my phone)
by Gerald Britton 21 May '16

21 May '16

Needs pythonizing of course but I like the flexibility. ----- https://msdn.microsoft.com/visualfsharpdocs/conceptual/pattern-matching-%5b…

3 2

Re: [Python-ideas] Add ability to get current event loop within asyncio coroutine
by Vincent Michel 20 May '16

20 May '16

On Thu, May 19, 2016 at 11:49 AM, Ilya Kulakov <kulakov.ilya(a)gmail.com> wrote: > I think Python would benefit from providing either a new method (e.g. > `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` > to return current event loop when called from a coroutine. It should, in my > opinion, reduce necessity of passing event loop > between coroutines in application's code as it is done in asyncio itself and > 3rd party libraries (like aiohttp). I was about to make the same proposal. I can add two practical arguments in favor. First, some coroutines could be simplified. For instance, asyncio.sleep takes an optional loop argument, defaulting to get_event_loop(). But get_event_loop is called inside the coroutine, and therefore inside a running event loop. This loop is actually the only valid choice, and using anything else should raise an exception. If get_event_loop is guaranteed to return the current running loop, then there is no reason to pass it as an argument to asyncio.sleep. Other coroutines could benefit from that, including wait and wait_for. It would also help for queues, locks and other task functions (gather, shield, as_completed, etc.). Here the loop argument is still useful, in case those objects are declared outside their event loop. However, it would be really convenient to assume loop=None always works as expected if the object is created inside a coroutine. In many cases, libraries tend to forward the loop argument from coroutine to coroutine, just in case someone writes: loop = asyncio.new_event_loop() loop.run_until_complete(my_coro(loop=loop)) instead of: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(my_coro()) TL;DR `async def coro(*, loop=None)` doesn't really make sense, and could be discouraged if get_event_loop was guaranteed to return the running event loop when called inside a coroutine.

3 3

Type hinting for path-related functions
by Koos Zevenhoven 20 May '16

20 May '16

I actually proposed this already in one of the pathlib threads on python-dev, but I decided to repost here, because this is easily seen as a separate issue. I'll start with some introduction, then moving on to the actual type hinting part. In our seemingly never-ending discussions about pathlib support in the stdlib in various threads, first here on python-ideas, then even more extensively on python-dev, have perhaps almost converged. The required changes involve a protocol method, probably named __fspath__, which any path-like type could implement to return a more, let's say, "classical" path object such as a str. However, the protocol is polymorphic and may also return bytes, which has a lot do do with the fact that the stdlib itself is polymophic and currently accepts str as well as bytes paths almost everywhere, including the newly-introduced os.scandir + DirEntry combination. The upcoming improvements will further allow passing pathlib path objects as well as DirEntry objects to any stdlib function that take paths. It came up, for instance here [1], that the function associated with the protocol, potentially named os.fspath, will end up needing type hints. This function takes pathlike objects and turns them into str or bytes. There are various different scenarios [2] that can be considered for code dealing with paths, but let's consider the case of os.path.* and other traditional python path-related functions. Some examples: os.path.join Currently, it takes str or bytes paths and returns a joined path of the same type (mixing different types raises an exception). In the future, it will also accept pathlib objects (underlying type always str) and DirEntry (underlying type str or bytes) or third-party path objects (underlying type str or bytes). The function will then return a pathname of the underlying type. os.path.dirname Currently, it takes a str or bytes and returns the dirname of the same type. In the future, it will also accept Path and DirEntry and return the underlying type. Let's consider the type hint of os.path.dirname at present and in the future: Currently, one could write def dirname(p: Union[str, bytes]) -> Union[str, bytes]: ... While this is valid, it could be more precise: pathstring = typing.TypeVar('pathstring', str, bytes) def dirname(p: pathstring) -> pathstring: ... This now contains the information that the return type is the same as the argument type. The name 'pathstring' may be considered slightly misleading because "byte strings" are not actually strings in Python 3, but at least it does not advertise the use of bytes as paths, which is very rarely desirable. But what about the future. There are two kinds of rich path objects, those with an underlying type of str and those with an underlying type of bytes. These should implement the __fspath__() protocol and return their underlying type. However, we do care about what (underlying) type is provided by the protocol, so we might want to introduce something like typing.FSPath[underlying_type]: FSPath[str] # str-based pathlike, including str FSPath[bytes] # bytes-based pathlike, including bytes And now, using the above defined TypeVar pathstring, the future version of dirname would be type annotated as follows: def dirname(p: FSPath[pathstring]) -> pathstring: ... It's getting late. I hope this made sense :). -Koos [1] https://mail.python.org/pipermail/python-dev/2016-April/144246.html [2] https://mail.python.org/pipermail/python-dev/2016-April/144239.html

10 55

Add ability to get current event loop within asyncio coroutine
by Ilya Kulakov 19 May '16

19 May '16

asyncio is a great library which with recent changes in 3.5 was made even better. However there is an inconvenience that bothers me and looks very unnatural: inability to access event loop that executes a coroutine from within that coroutine. This is unnatural, because we have `self` and `cls` to access object from a method it is bound to, we have `current_thread` to get an instance of Thread that currently executes code, but for a coroutine we have no such method, we cannot get its context of execution. Current implementation of `get_event_loop` method is not sufficient, it will not work if thread has more than one event loop. I think Python would benefit from providing either a new method (e.g. `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` to return current event loop when called from a coroutine. It should, in my opinion, reduce necessity of passing event loop between coroutines in application's code as it is done in asyncio itself and 3rd party libraries (like aiohttp). Best Regards, Ilya Kulakov

3 2

Adding a frame evaluation API to CPython
by Dino Viehland 18 May '16

18 May '16

Adding a frame evaluation API to CPython Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <mailto:brett@python.org>, Dino Viehland <mailto:dinov@microsoft.com> https://github.com/Microsoft/Pyjion/blob/master/pep.rst Abstract This PEP proposes to expand CPython's C API https://github.com/Microsoft/Pyjion/blob/master/pep.rst#c-api to allow for the specification of a per-interpreter function pointer to handle the evaluation of frames https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex. This proposal also suggests adding a new field to code objects https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id21 to store arbitrary data for use by the frame evaluation function. Rationale One place where flexibility has been lacking in Python is in the direct execution of Python code. While CPython's C API https://github.com/Microsoft/Pyjion/blob/master/pep.rst#c-api allows for constructing the data going into a frame object and then evaluating it via PyEval_EvalFrameEx() https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex, control over the execution of Python code comes down to individual objects instead of a hollistic control of execution at the frame level. While wanting to have influence over frame evaluation may seem a bit too low-level, it does open the possibility for things such as a JIT to be introduced into CPython without CPython itself having to provide one. By allowing external C code to control frame evaluation, a JIT can participate in the execution of Python code at the key point where evaluation occurs. This then allows for a JIT to conditionally recompile Python bytecode to machine code as desired while still allowing for executing regular CPython bytecode when running the JIT is not desired. This can be accomplished by allowing interpreters to specify what function to call to evaluate a frame. And by placing the API at the frame evaluation level it allows for a complete view of the execution environment of the code for the JIT. This ability to specify a frame evaluation function also allows for other use-cases beyond just opening CPython up to a JIT. For instance, it would not be difficult to implement a tracing or profiling function at the call level with this API. While CPython does provide the ability to set a tracing or profiling function at the Python level, this would be able to match the data collection of the profiler and quite possibly be faster for tracing by simply skipping per-line tracing support. It also opens up the possibility of debugging where the frame evaluation function only performs special debugging work when it detects it is about to execute a specific code object. In that instance the bytecode could be theoretically rewritten in-place to inject a breakpoint function call at the proper point for help in debugging while not having to do a heavy-handed approach as required by sys.settrace(). To help facilitate these use-cases, we are also proposing the adding of a "scratch space" on code objects via a new field. This will allow per-code object data to be stored with the code object itself for easy retrieval by the frame evaluation function as necessary. The field itself will simply be a PyObject * type so that any data stored in the field will participate in normal object memory management. Proposal All proposed C API changes below will not be part of the stable ABI. Expanding PyCodeObject One field is to be added to the PyCodeObject struct https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id23: typedef struct { ... PyObject *co_extra; /* "Scratch space" for the code object. */ } PyCodeObject; The co_extra will be NULL by default and will not be used by CPython itself. Third-party code is free to use the field as desired. The field will be freed like all other fields on PyCodeObject during deallocation using Py_XDECREF(). It is not recommended that multiple users attempt to use the co_extra simultaneously. While a dictionary could theoretically be set to the field and various users could use a key specific to the project, there is still the issue of key collisions as well as performance degradation from using a dictionary lookup on every frame evaluation. Users are expected to do a type check to make sure that the field has not been previously set by someone else. Expanding PyInterpreterState The entrypoint for the frame evalution function is per-interpreter: // Same type signature as PyEval_EvalFrameEx(). typedef PyObject* (__stdcall *PyFrameEvalFunction)(PyFrameObject*, int); typedef struct { ... PyFrameEvalFunction eval_frame; } PyInterpreterState; By default, the eval_frame field will be initialized to a function pointer that represents what PyEval_EvalFrameEx() currently is (called PyEval_EvalFrameDefault(), discussed later in this PEP). Third-party code may then set their own frame evaluation function instead to control the execution of Python code. A pointer comparison can be used to detect if the field is set to PyEval_EvalFrameDefault() and thus has not been mutated yet. Changes to Python/ceval.c PyEval_EvalFrameEx() https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex as it currently stands will be renamed to PyEval_EvalFrameDefault(). The new PyEval_EvalFrameEx() will then become: PyObject * PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag) { PyThreadState *tstate = PyThreadState_GET(); return tstate->interp->eval_frame(frame, throwflag); } This allows third-party code to place themselves directly in the path of Python code execution while being backwards-compatible with code already using the pre-existing C API. Performance impact As this PEP is proposing an API to add pluggability, performance impact is considered only in the case where no third-party code has made any changes. Several runs of pybench https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id25 consistently showed no performance cost from the API change alone. A run of the Python benchmark suite https://github.com/Microsoft/Pyjion/blob/master/pep.rst#py-benchmarks showed no measurable cost in performance. In terms of memory impact, since there are typically not many CPython interpreters executing in a single process that means the impact of co_extra being added to PyCodeObject is the only worry. According to https://github.com/Microsoft/Pyjion/blob/master/pep.rst#code-object-count, a run of the Python test suite results in about 72,395 code objects being created. On a 64-bit CPU that would result in 4,633,280 bytes of extra memory being used if all code objects were alive at once and had nothing set in their co_extra fields. Example Usage A JIT for CPython Pyjion The Pyjion project https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id18 has used this proposed API to implement a JIT for CPython using the CoreCLR's JIT https://github.com/Microsoft/Pyjion/blob/master/pep.rst#coreclr. Each code object has its co_extra field set to a PyjionJittedCode object which stores four pieces of information: 1. Execution count 2. A boolean representing whether a previous attempt to JIT failed 3. A function pointer to a trampoline (which can be type tracing or not) 4. A void pointer to any JIT-compiled machine code The frame evaluation function has (roughly) the following algorithm: def eval_frame(frame, throw_flag): pyjion_code = frame.code.co_extra if not pyjion_code: frame.code.co_extra = PyjionJittedCode() elif not pyjion_code.jit_failed: if not pyjion_code.jit_code: return pyjion_code.eval(pyjion_code.jit_code, frame) elif pyjion_code.exec_count > 20_000: if jit_compile(frame): return pyjion_code.eval(pyjion_code.jit_code, frame) else: pyjion_code.jit_failed = True pyjion_code.exec_count += 1 return PyEval_EvalFrameDefault(frame, throw_flag) The key point, though, is that all of this work and logic is separate from CPython and yet with the proposed API changes it is able to provide a JIT that is compliant with Python semantics (as of this writing, performance is almost equivalent to CPython without the new API). This means there's nothing technically preventing others from implementing their own JITs for CPython by utilizing the proposed API. Other JITs It should be mentioned that the Pyston team was consulted on an earlier version of this PEP that was more JIT-specific and they were not interested in utilizing the changes proposed because they want control over memory layout they had no interest in directly supporting CPython itself. An informal discusion with a developer on the PyPy team led to a similar comment. Numba https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, on the other hand, suggested that they would be interested in the proposed change in a post-1.0 future for themselves https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba-interest. Debugging In conversations with the Python Tools for Visual Studio team (PTVS) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#ptvs, they thought they would find these API changes useful for implementing more performant debugging. As mentioned in the https://github.com/Microsoft/Pyjion/blob/master/pep.rst#rationale section, this API would allow for switching on debugging functionality only in frames where it is needed. This could allow for either skipping information that sys.settrace() normally provides and even go as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode. Implementation A set of patches implementing the proposed API is available through the Pyjion project https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id18. In its current form it has more changes to CPython than just this proposed API, but that is for ease of development instead of strict requirements to accomplish its goals. Open Issues Allow eval_frame to be NULL Currently the frame evaluation function is expected to always be set. It could very easily simply default to NULL instead which would signal to use PyEval_EvalFrameDefault(). The current proposal of not special-casing the field seemed the most straight-forward, but it does require that the field not accidentally be cleared, else a crash may occur. Rejected Ideas A JIT-specific C API Originally this PEP was going to propose a much larger API change which was more JIT-specific. After soliciting feedback from the Numba team https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, though, it became clear that the API was unnecessarily large. The realization was made that all that was truly needed was the opportunity to provide a trampoline function to handle execution of Python code that had been JIT-compiled and a way to attach that compiled machine code along with other critical data to the corresponding Python code object. Once it was shown that there was no loss in functionality or in performance while minimizing the API changes required, the proposal was changed to its current form. References [1] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id11, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id16) Pyjion project (https://github.com/microsoft/pyjion) [2] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id1, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id4) CPython's C API (https://docs.python.org/3/c-api/index.html) [3] PyCodeObject (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id12 .NET Core Runtime (CoreCLR) (https://github.com/dotnet/coreclr) [5] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id2, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id5, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id7) PyEval_EvalFrameEx() (https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyE…) [6] PyCodeObject (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) [7] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id13, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id17) Numba (http://numba.pydata.org/) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id14 numba-users mailing list: "Would the C API for a JIT entrypoint being proposed by Pyjion help out Numba?" (https://groups.google.com/a/continuum.io/forum/#!topic/numba-users/yRl_0t8-…) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id10 [Python-Dev] Opcode cache in ceval loop (https://mail.python.org/pipermail/python-dev/2016-February/143025.html) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id9 Python benchmark suite (https://hg.python.org/benchmarks) [11] Pyston (http://pyston.org/) [12] PyPy (http://pypy.org/) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id15 Python Tools for Visual Studio (http://microsoft.github.io/PTVS/) Copyright This document has been placed in the public domain.

6 7

Add contextlib.DummyContext
by Giampaolo Rodola' 18 May '16

18 May '16

This is one of those things which are so easy to implement which makes you think it is probably not worth adding to the stdlib, but then again, this is something I've ended up doing and rewriting pretty often over the years. Real world example: class DummyLock(object): def __init__(self, *args, **kwargs): pass def __enter__(self, *args, **kwargs): return self def __exit__(self, *args, **kwargs): pass def get_lock(name, bypass_lock=False): lock_cls = DummyLock if bypass_lock else RedisLock return lock with get_lock('foo', bypass_lock=True): ... Similarly to contextlib.closing and contextlib.suppress, perhaps it would be nice to have contextlib.DummyContext just because it's something which is done (I think) fairly often. On the bikeshedding front, in order to be consistent with closing(), redirect_stderr(), redirect_stdout() and suppress(), a better name for this would probably be contextlib.dummy_context. Extra: the same thing can be achieved by using mock.MagicMock, which probably makes this proposal useless and kills it entirely. The additional value is that it would be more explicit/clear/immediate to have this in contextlib itself as opposed to unittest module, which is kinda weird. But then again, "there should be (possibly) only one way to do it" so I'm not sure. OK, I should stop talking with myself. =) -- Giampaolo - http://grodola.blogspot.com

6 6

Bytecode for calling function with keyword arguments
by Serhiy Storchaka 15 May '16

15 May '16

Currently the call of function with keyword arguments (say `f(a, b, x=c, y=d)`) is compiled to following bytecode: 0 LOAD_NAME 0 (f) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 LOAD_CONST 0 ('x') 12 LOAD_NAME 3 (c) 15 LOAD_CONST 1 ('y') 18 LOAD_NAME 4 (d) 21 CALL_FUNCTION 514 (2 positional, 2 keyword pair) For every positional argument its value is pushed on the stack, and for every keyword argument its name and its value are pushed on the stack. But keyword arguments are always constant strings! We can reorder the stack, and push keyword argument values and names separately. And we can save all keyword argument names for this call in a constant tuple and load it by one bytecode command. 0 LOAD_NAME 0 (f) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 LOAD_NAME 3 (c) 12 LOAD_NAME 4 (d) 15 LOAD_CONST 0 (('x', 'y')) 18 CALL_FUNCTION2 2 Benefits: 1. We save one command for every keyword parameter after the first. This saves bytecode size and execution time. 2. Since the number of keyword arguments is obtained from tuple's size, new CALL_FUNCTION opcode needs only the number of positional arguments. It's argument is simpler and needs less bits (important for wordcode). Disadvantages: 1. Increases the number of constants.

4 5

Exposing flat bytecode representation to optimizers
by Andrew Barnert 15 May '16

15 May '16

The biggest pain with dealing with the peephole optimizer is that it happens after all the complicated flattening and fixup[^1] the compiler does, which means you have to hack up the jump targets as you go along. The future bytecode optimizers that PEP 511 enables will have the same headache. But this isn't actually necessary. The optimizer could work on a flat array of instructions[^2] instead of an array of bytes, with relocatable jump targets instead of fixed byte offsets, and then the compiler could do the fixup _after_ the optimization.[^3] It would break the optimizer APIs, but `PyCode_Optimize` isn't public, and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. I don't think we need to expose the intermediate representation any farther along than the `PyCode_Optimize` step.[^4] Just moving the optimize one step earlier in the chain solves more than enough to be worth it. [^1]: If you think numbering the offsets and emitting the jump targets is easy: Every time you fix up a jump, that may require adding an `EXTENDED_ARG`, which means you have to redo any fixups that cross the the current instruction. The compiler does this by just looping until it hasn't added any more `EXTENDED_ARG`s. [^2]: In case anyone's thinking that wordcode would solve this problem, it doesn't. The `EXTENDED_ARG` jump targets are a much bigger hassle than the 1-or-3-byte-ops, and wordcode not only can't eliminate those, but makes `EXTENDED_ARG` more common. [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. [^4]: Later stages--import hooks, optimizing decorators, etc.--have the same pain as the peephole optimizer, but they also have code objects that contain other code objects, and they can be written in Python, and so on, so the same solution isn't feasible there. Of course we could add a function to convert bytecode back to a list of instructions, in some format that can be exposed to Python (in fact, `dis` already does 90% of that), and then another one to convert that back to bytecode and redo the fixup (which is basically the third-party `byteplay` module). But that's almost certainly overkill. (If we wanted that, I'd rather add `byteplay` to the stdlib, port it and `dis` to C, and expose a C API for them. And then we could use that for everything, including the peephole optimizer and PEP 511 optimizers. Which would all be really cool, but I think it's more work than we want to do, and I don't know if we'd actually want something like `byteplay` builtin even if it were easy...)

6 16

Boolean parameters guidelines
by Serhiy Storchaka 15 May '16

15 May '16

I propose to add following recommendations in PEP 8 or other documents: 1. It is preferable to pass boolean arguments as keyword arguments (if this is not the only argument). 2. It is preferable to declare boolean parameters as keyword-only parameters. What are you think about this?

22 46

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas May 2016