Python-Dev
Threads by month
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
July 2021
- 51 participants
- 38 discussions
Hi folks,
It's been a long time coming, but I've finally made enough progress on
the reference implementation that I think it's time to ask Nathaniel
to pronounce on the current iteration of PEP 558 (Defined semantics
for locals()).
The rendered version is up at
https://www.python.org/dev/peps/pep-0558/, and I've included the plain
text version below.
For those that are reading the PEP for the first time, the gist is:
* standardise on Python 3.10 behaviour *except* that locals() at
function scope returns a fresh snapshot every time instead of a
reference to the frame level state cache
* make the Python level frame.f_locals on optimised frames a
write-through proxy that keeps both the real fast locals storage and
the C level f_locals state cache up to date
* add new C APIs that allow C code to explicitly request the semantics
the client code actually wants ("behave like the Python locals()
builtin", "always make a copy", "always provide a read-only view")
* soft-deprecate the legacy PyEval_GetLocals() API (while ensuring it
still works)
* use the new features to significantly improve the performance of
code execution tracing hooks implemented in Python
For those that remember reading older versions of the PEP, the key
changes relative to the last discourse thread (back in late 2019/early
2020) are:
* incorporating the C API design improvements from the 2019/20 Discourse thread
* incorporating Mark Shannon's feedback from earlier this year (most
notably, changing the proxy design to only create a reference cycle
from the frames back to the fast locals proxies that reference them if
you store a reference to the proxy as a local variable on the frame)
* trying (and failing) to remove the fast locals proxy dependency on
the C level f_locals cache on optimised frame objects. Instead, the
fast locals proxy more explicitly uses that dictionary as a state
cache to speed up certain operations (e.g. dict equality comparisons,
iteration, and rendering the proxy contents as a string), and to store
keys that don't correspond to frame level variables, while also
exposing a new ``sync_frame_cache()`` method to sync the state cache
on the underlying frame with changes made via other mechanisms.
Cheers,
Nick.
P.S. The PEP text in the email already incorporates the text review
edits in https://github.com/python/peps/pull/2038/files that haven't
been merged to the web version yet.
Suggested clarifications to the text of the PEP that don't affect the
overall design can be added as comments on that PR rather than being
added to the mailing list thread.
=====================
PEP: 558
Title: Defined semantics for locals()
Author: Nick Coghlan <ncoghlan(a)gmail.com>
BDFL-Delegate: Nathaniel J. Smith
Discussions-To: <python-dev(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Sep-2017
Python-Version: 3.11
Post-History: 2017-09-08, 2019-05-22, 2019-05-30, 2019-12-30, 2021-07-18
Abstract
========
The semantics of the ``locals()`` builtin have historically been underspecified
and hence implementation dependent.
This PEP proposes formally standardising on the behaviour of the CPython 3.10
reference implementation for most execution scopes, with some adjustments to the
behaviour at function scope to make it more predictable and independent of the
presence or absence of tracing functions.
In addition, it proposes that the following functions be added to the stable
Python C API/ABI::
PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
PyObject * PyLocals_GetCopy();
PyObject * PyLocals_GetView();
It also proposes the addition of several supporting functions and type
definitions to the CPython C API.
Rationale
=========
While the precise semantics of the ``locals()`` builtin are nominally undefined,
in practice, many Python programs depend on it behaving exactly as it behaves in
CPython (at least when no tracing functions are installed).
Other implementations such as PyPy are currently replicating that behaviour,
up to and including replication of local variable mutation bugs that
can arise when a trace hook is installed [1]_.
While this PEP considers CPython's current behaviour when no trace hooks are
installed to be largely acceptable, it considers the current
behaviour when trace hooks are installed to be problematic, as it causes bugs
like [1]_ *without* even reliably enabling the desired functionality of allowing
debuggers like ``pdb`` to mutate local variables [3]_.
Review of the initial PEP and the draft implementation then identified an
opportunity for simplification of both the documentation and implementation
of the function level ``locals()`` behaviour by updating it to return an
independent snapshot of the function locals and closure variables on each
call, rather than continuing to return the semi-dynamic intermittently updated
shared copy that it has historically returned in CPython.
Proposal
========
The expected semantics of the ``locals()`` builtin change based on the current
execution scope. For this purpose, the defined scopes of execution are:
* module scope: top-level module code, as well as any other code executed using
``exec()`` or ``eval()`` with a single namespace
* class scope: code in the body of a ``class`` statement, as well as any other
code executed using ``exec()`` or ``eval()`` with separate local and global
namespaces
* function scope: code in the body of a ``def`` or ``async def`` statement,
or any other construct that creates an optimized code block in CPython (e.g.
comprehensions, lambda functions)
This PEP proposes elevating most of the current behaviour of the CPython
reference implementation to become part of the language specification, *except*
that each call to ``locals()`` at function scope will create a new dictionary
object, rather than caching a common dict instance in the frame object that
each invocation will update and return.
This PEP also proposes to largely eliminate the concept of a separate "tracing"
mode from the CPython reference implementation. In releases up to and including
Python 3.10, the CPython interpreter behaves differently when a trace hook has
been registered in one or more threads via an implementation dependent mechanism
like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or
``PyEval_SetTrace`` ([5]_) in CPython's C API.
This PEP proposes changes to CPython's behaviour at function scope that make
the ``locals()`` builtin semantics when a trace hook is registered identical to
those used when no trace hook is registered, while also making the related frame
API semantics clearer and easier for interactive debuggers to rely on.
The proposed elimination of tracing mode affects the semantics of frame object
references obtained through other means, such as via a traceback, or via the
``sys._getframe()`` API, as the write-through semantics needed for trace hook
support are always provided by the ``f_locals`` attribute on frame objects,
rather than being runtime state dependent.
New ``locals()`` documentation
------------------------------
The heart of this proposal is to revise the documentation for the ``locals()``
builtin to read as follows:
Return a mapping object representing the current local symbol table, with
variable names as the keys, and their currently bound references as the
values.
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, this function returns the same namespace as ``globals()``.
At class scope, it returns the namespace that will be passed to the
metaclass constructor.
When using ``exec()`` or ``eval()`` with separate local and global
namespaces, it returns the local namespace passed in to the function call.
In all of the above cases, each call to ``locals()`` in a given frame of
execution will return the *same* mapping object. Changes made through
the mapping object returned from ``locals()`` will be visible as bound,
rebound, or deleted local variables, and binding, rebinding, or deleting
local variables will immediately affect the contents of the returned mapping
object.
At function scope (including for generators and coroutines), each call to
``locals()`` instead returns a fresh dictionary containing the current
bindings of the function's local variables and any nonlocal cell references.
In this case, name binding changes made via the returned dict are *not*
written back to the corresponding local variables or nonlocal cell
references, and binding, rebinding, or deleting local variables and nonlocal
cell references does *not* affect the contents of previously returned
dictionaries.
There would also be a versionchanged note for the release making this change:
In prior versions, the semantics of mutating the mapping object returned
from ``locals()`` were formally undefined. In CPython specifically,
the mapping returned at function scope could be implicitly refreshed by
other operations, such as calling ``locals()`` again, or the interpreter
implicitly invoking a Python level trace function. Obtaining the legacy
CPython behaviour now requires explicit calls to update the initially
returned dictionary with the results of subsequent calls to ``locals()``.
For reference, the current documentation of this builtin reads as follows:
Update and return a dictionary representing the current local symbol table.
Free variables are returned by locals() when it is called in function
blocks, but not in class blocks.
Note: The contents of this dictionary should not be modified; changes may
not affect the values of local and free variables used by the interpreter.
(In other words: the status quo is that the semantics and behaviour of
``locals()`` are formally implementation defined, whereas the proposed
state after this PEP is that the only implementation defined behaviour will be
that associated with whether or not the implementation emulates the CPython
frame API, with the behaviour in all other cases being defined by the language
and library references)
Module scope
------------
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, ``locals()`` must return the same object as ``globals()``,
which must be the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).
Variable assignments during subsequent code execution in the same scope must
dynamically change the contents of the returned mapping, and changes to the
returned mapping must change the values bound to local variable names in the
execution environment.
To capture this expectation as part of the language specification, the following
paragraph will be added to the documentation for ``locals()``:
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, this function returns the same namespace as ``globals()``.
This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.
Class scope
-----------
At class scope, as well as when using ``exec()`` or ``eval()`` with separate
global and local namespaces, ``locals()`` must return the specified local
namespace (which may be supplied by the metaclass ``__prepare__`` method
in the case of classes). As for module scope, this must be a direct reference
to the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).
Variable assignments during subsequent code execution in the same scope must
change the contents of the returned mapping, and changes to the returned mapping
must change the values bound to local variable names in the
execution environment.
The mapping returned by ``locals()`` will *not* be used as the actual class
namespace underlying the defined class (the class creation process will copy
the contents to a fresh dictionary that is only accessible by going through the
class machinery).
For nested classes defined inside a function, any nonlocal cells referenced from
the class scope are *not* included in the ``locals()`` mapping.
To capture this expectation as part of the language specification, the following
two paragraphs will be added to the documentation for ``locals()``:
When using ``exec()`` or ``eval()`` with separate local and global
namespaces, [this function] returns the given local namespace.
At class scope, it returns the namespace that will be passed to the metaclass
constructor.
This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.
Function scope
--------------
At function scope, interpreter implementations are granted significant freedom
to optimise local variable access, and hence are NOT required to permit
arbitrary modification of local and nonlocal variable bindings through the
mapping returned from ``locals()``.
Historically, this leniency has been described in the language specification
with the words "The contents of this dictionary should not be modified; changes
may not affect the values of local and free variables used by the interpreter."
This PEP proposes to change that text to instead say:
At function scope (including for generators and coroutines), each call to
``locals()`` instead returns a fresh dictionary containing the current
bindings of the function's local variables and any nonlocal cell references.
In this case, name binding changes made via the returned dict are *not*
written back to the corresponding local variables or nonlocal cell
references, and binding, rebinding, or deleting local variables and nonlocal
cell references does *not* affect the contents of previously returned
dictionaries.
This part of the proposal *does* require changes to the CPython reference
implementation, as CPython currently returns a shared mapping object that may
be implicitly refreshed by additional calls to ``locals()``, and the
"write back" strategy currently used to support namespace changes
from trace functions also doesn't comply with it (and causes the quirky
behavioural problems mentioned in the Rationale).
CPython Implementation Changes
==============================
Summary of proposed implementation-specific changes
---------------------------------------------------
* Changes are made as necessary to provide the updated Python level semantics
* Two new functions are added to the stable ABI to replicate the updated
behaviour of the Python ``locals()`` builtin::
PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
* One new function is added to the stable ABI to efficiently get a snapshot of
the local namespace in the running frame::
PyObject * PyLocals_GetCopy();
* One new function is added to the stable ABI to get a read-only view of the
local namespace in the running frame::
PyObject * PyLocals_GetView();
* Corresponding frame accessor functions for these new public APIs are added to
the CPython frame C API
* On optimised frames, the Python level ``f_locals`` API will become a direct
read/write proxy for the frame's local and closure variable storage, but
will use the C level ``f_locals`` struct field to hold a value cache that
also allows for storage of arbitrary additional keys. Additional details on
the expected behaviour of that fast locals proxy are given below.
* No C API function is added to get access to a mutable mapping for the local
namespace. Instead, ``PyObject_GetAttrString(frame, "f_locals")`` is used, the
same API as is used in Python code.
* ``PyEval_GetLocals()`` remains supported and does not emit a programmatic
warning, but will be deprecated in the documentation in favour of the new
APIs
* ``PyFrame_FastToLocals()`` and ``PyFrame_FastToLocalsWithError()`` remain
supported and do not emit a programmatic warning, but will be deprecated in
the documentation in favour of the new APIs
* ``PyFrame_LocalsToFast()`` always raises ``RuntimeError()``, indicating that
``PyObject_GetAttrString(frame, "f_locals")`` should be used to obtain a
mutable read/write mapping for the local variables.
* The trace hook implementation will no longer call ``PyFrame_FastToLocals()``
implicitly. The version porting guide will recommend migrating to
``PyFrame_GetLocalsView()`` for read-only access and
``PyObject_GetAttrString(frame, "f_locals")`` for read/write access.
Providing the updated Python level semantics
--------------------------------------------
The implementation of the ``locals()`` builtin is modified to return a distinct
copy of the local namespace rather than a direct reference to the internal
dynamically updated snapshot returned by ``PyEval_GetLocals()``.
Resolving the issues with tracing mode behaviour
------------------------------------------------
The current cause of CPython's tracing mode quirks (both the side effects from
simply installing a tracing function and the fact that writing values back to
function locals only works for the specific function being traced) is the way
that locals mutation support for trace hooks is currently implemented: the
``PyFrame_LocalsToFast`` function.
When a trace function is installed, CPython currently does the following for
function frames (those where the code object uses "fast locals" semantics):
1. Calls ``PyFrame_FastToLocals`` to update the dynamic snapshot
2. Calls the trace hook (with tracing of the hook itself disabled)
3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the dynamic
snapshot
This approach is problematic for a few different reasons:
* Even if the trace function doesn't mutate the snapshot, the final step resets
any cell references back to the state they were in before the trace function
was called (this is the root cause of the bug report in [1]_)
* If the trace function *does* mutate the snapshot, but then does something
that causes the snapshot to be refreshed, those changes are lost (this is
one aspect of the bug report in [3]_)
* If the trace function attempts to mutate the local variables of a frame other
than the one being traced (e.g. ``frame.f_back.f_locals``), those changes
will almost certainly be lost (this is another aspect of the bug report in
[3]_)
* If a ``locals()`` reference is passed to another function, and *that*
function mutates the snapshot namespace, then those changes *may* be written
back to the execution frame *if* a trace hook is installed
The proposed resolution to this problem is to take advantage of the fact that
whereas functions typically access their *own* namespace using the language
defined ``locals()`` builtin, trace functions necessarily use the implementation
dependent ``frame.f_locals`` interface, as a frame reference is what gets
passed to hook implementations.
Instead of being a direct reference to the internal dynamic snapshot used to
populate the independent snapshots returned by ``locals()``, the Python level
``frame.f_locals`` will be updated to instead return a dedicated proxy type
that has two internal attributes not exposed as part of the Python runtime
API:
* *frame*: the underlying frame that the snapshot is for
* *fast_refs*: a mapping from variable names to either fast local storage
offsets (for local variables) or to closure cells (for closure variables).
This mapping is lazily initialized on the first read or write access through
the proxy, rather than being eagerly populated as soon as the proxy
is created.
The C level ``f_locals`` attribute on the frame object is treated as a cache
by the fast locals proxy, as some operations (such as equality comparisons)
require a regular dictionary mapping from names to their respective values.
Fast local variables and cell variables are stored in the cache if they are
currently bound to a value. Arbitrary additional attributes may also be stored
in the cache. It *is* possible for the cache to get out of sync with the actual
frame state (e.g. as code executes binding and unbinding operations, or if
changes are made directly to the cache dict). A dedicated ``sync_frame_cache()``
method is provided that runs ``PyFrame_FastToLocalsWithError()`` to ensure the
cache is consistent with the current frame state.
``__getitem__`` operations on the proxy will populate the ``fast_refs`` mapping
(if it is not already populated), and then either return the relevant value
(if the key is found in either the ``fast_refs`` mapping or the ``f_locals``
dynamic snapshot stored on the frame), or else raise ``KeyError``. Variables
that are defined but not currently bound raise ``KeyError`` (just as they're
omitted from the result of ``locals()``).
As the frame storage is always accessed directly, the proxy will automatically
pick up name binding operations that take place as the function executes. The
cache dictionary is implicitly updated when individual variables are read
from the frame state (including for containment checks, which need to check if
the name is currently bound or unbound).
Similarly, ``__setitem__`` and ``__delitem__`` operations on the proxy will
directly affect the corresponding fast local or cell reference on the underlying
frame, ensuring that changes are immediately visible to the running Python code,
rather than needing to be written back to the runtime storage at some
later time.
Such changes are also immediately written to the ``f_locals`` cache to
reduce the
opportunities for the cache to get out of sync with the frame state.
Keys that are not defined as local or closure variables on the underlying frame
are still written to the ``f_locals`` cache on optimised frames. This allows
utilities like ``pdb`` (which writes ``__return__`` and ``__exception__``
values into the frame ``f_locals`` mapping) to continue working as they always
have. These additional keys that do not correspond to a local or closure
variable on the frame will be left alone by future cache sync operations.
Other ``Mapping`` and ``MutableMapping`` methods will behave as expected for a
mapping with these essential method semantics, with the exception that only
intrinsically ``O(n)`` operations (e.g. copying, rendering as a string) and
operations that operate on a single key (e.g. getting, setting, deleting, or
popping) will implicitly refresh the value cache. Other operations
(e.g. length checks, equality checks, iteration) may use the value cache without
first ensuring that it is up to date (as ensuring the cache is up to date is
itself an ``O(n)`` operation).
An additional benefit of storing only the variable value cache on the frame
(rather than storing an instance of the proxy type), is that it avoids
creating a reference cycle from the frame back to itself, so the frame will
only be kept alive if another object retains a reference to a proxy instance.
Changes to the stable C API/ABI
-------------------------------
Unlike Python code, extension module functions that call in to the Python C API
can be called from any kind of Python scope. This means it isn't obvious from
the context whether ``locals()`` will return a snapshot or not, as it depends
on the scope of the calling Python code, not the C code itself.
This means it is desirable to offer C APIs that give predictable, scope
independent, behaviour. However, it is also desirable to allow C code to
exactly mimic the behaviour of Python code at the same scope.
To enable mimicking the behaviour of Python code, the stable C ABI would gain
the following new functions::
PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
``PyLocals_Get()`` is directly equivalent to the Python ``locals()`` builtin.
It returns a new reference to the local namespace mapping for the active
Python frame at module and class scope, and when using ``exec()`` or ``eval()``.
It returns a shallow copy of the active namespace at
function/coroutine/generator scope.
``PyLocals_GetReturnsCopy()`` returns zero if ``PyLocals_Get()`` returns a
direct reference to the local namespace mapping, and a non-zero value if it
returns a shallow copy. This allows extension module code to determine the
potential impact of mutating the mapping returned by ``PyLocals_Get()`` without
needing access to the details of the running frame object.
To allow extension module code to behave consistently regardless of the active
Python scope, the stable C ABI would gain the following new functions::
PyObject * PyLocals_GetCopy();
PyObject * PyLocals_GetView();
``PyLocals_GetCopy()`` returns a new dict instance populated from the current
locals namespace. Roughly equivalent to ``dict(locals())`` in Python code, but
avoids the double-copy in the case where ``locals()`` already returns a shallow
copy.
``PyLocals_GetView()`` returns a new read-only mapping proxy instance for the
current locals namespace. This view immediately reflects all local variable
changes, independently of whether the running frame is optimised or not.
The existing ``PyEval_GetLocals()`` API will retain its existing behaviour in
CPython (mutable locals at class and module scope, shared dynamic snapshot
otherwise). However, its documentation will be updated to note that the
conditions under which the shared dynamic snapshot get updated have changed.
The ``PyEval_GetLocals()`` documentation will also be updated to recommend
replacing usage of this API with whichever of the new APIs is most appropriate
for the use case:
* Use ``PyLocals_GetView()`` for read-only access to the current locals
namespace.
* Use ``PyLocals_GetCopy()`` for a regular mutable dict that contains a copy of
the current locals namespace, but has no ongoing connection to the active
frame.
* Use ``PyLocals_Get()`` to exactly match the semantics of the Python level
``locals()`` builtin.
* Query ``PyLocals_GetReturnsCopy()`` explicitly to implement custom handling
(e.g. raising a meaningful exception) for scopes where ``PyLocals_Get()``
would return a shallow copy rather than granting read/write access to the
locals namespace.
* Use implementation specific APIs (e.g.
``PyObject_GetAttrString(frame, "f_locals")``)
if read/write access to the frame is required and
``PyLocals_GetReturnsCopy()``
is true.
Changes to the public CPython C API
-----------------------------------
The existing ``PyEval_GetLocals()`` API returns a borrowed reference, which
means it cannot be updated to return the new shallow copies at function
scope. Instead, it will continue to return a borrowed reference to an internal
dynamic snapshot stored on the frame object. This shared mapping will behave
similarly to the existing shared mapping in Python 3.10 and earlier,
but the exact
conditions under which it gets refreshed will be different. Specifically, it
will be updated only in the following circumstance:
* any call to ``PyEval_GetLocals()``, ``PyLocals_Get()``,
``PyLocals_GetCopy()``,
or the Python ``locals()`` builtin while the frame is running
* any call to ``PyFrame_GetLocals()``, ``PyFrame_GetLocalsCopy()``,
``_PyFrame_BorrowLocals()``, ``PyFrame_FastToLocals()``, or
``PyFrame_FastToLocalsWithError()`` for the frame
* retrieving the ``f_locals`` attribute from a Python level frame object
* any call to the ``sync_frame_cache()`` method on a fast locals proxy
referencing that frame
* any operation on a fast locals proxy object that requires the shared
mapping to be up to date on the underlying frame. In the initial reference
implementation, those operations are those that are intrinsically ``O(n)``
operations (``flp.copy()`` and rendering as a string), as well as those that
refresh the cache entries for individual keys.
Accessing the frame "view" APIs will *not* implicitly update the shared dynamic
snapshot, and the CPython trace hook handling will no longer implicitly update
it either.
(Note: even though ``PyEval_GetLocals()`` is part of the stable C API/ABI, the
specifics of when the namespace it returns gets refreshed are still an
interpreter implementation detail)
The additions to the public CPython C API are the frame level enhancements
needed to support the stable C API/ABI updates::
PyObject * PyFrame_GetLocals(frame);
int PyFrame_GetLocalsReturnsCopy(frame);
PyObject * PyFrame_GetLocalsCopy(frame);
PyObject * PyFrame_GetLocalsView(frame);
PyObject * _PyFrame_BorrowLocals(frame);
``PyFrame_GetLocals(frame)`` is the underlying API for ``PyLocals_Get()``.
``PyFrame_GetLocalsReturnsCopy(frame)`` is the underlying API for
``PyLocals_GetReturnsCopy()``.
``PyFrame_GetLocalsCopy(frame)`` is the underlying API for
``PyLocals_GetCopy()``.
``PyFrame_GetLocalsView(frame)`` is the underlying API for
``PyLocals_GetView()``.
``_PyFrame_BorrowLocals(frame)`` is the underlying API for
``PyEval_GetLocals()``. The underscore prefix is intended to discourage use and
to indicate that code using it is unlikely to be portable across
implementations. However, it is documented and visible to the linker in order
to avoid having to access the internals of the frame struct from the
``PyEval_GetLocals()`` implementation.
The ``PyFrame_LocalsToFast()`` function will be changed to always emit
``RuntimeError``, explaining that it is no longer a supported operation, and
affected code should be updated to use
``PyObject_GetAttrString(frame, "f_locals")`` to obtain a read/write proxy
instead.
In addition to the above documented interfaces, the draft reference
implementation also exposes the following undocumented interfaces::
PyTypeObject _PyFastLocalsProxy_Type;
#define _PyFastLocalsProxy_CheckExact(self) \
(Py_TYPE(self) == &_PyFastLocalsProxy_Type)
This type is what the reference implementation actually returns from
``PyObject_GetAttrString(frame, "f_locals")`` for optimized frames (i.e.
when ``PyFrame_GetLocalsReturnsCopy()`` returns true).
Reducing the runtime overhead of trace hooks
--------------------------------------------
As noted in [9]_, the implicit call to ``PyFrame_FastToLocals()`` in the
Python trace hook support isn't free, and could be rendered unnecessary if
the frame proxy read values directly from the frame instead of getting them
from the mapping.
As the new frame locals proxy type doesn't require separate data refresh steps,
this PEP incorporates Victor Stinner's proposal to no longer implicitly call
``PyFrame_FastToLocalsWithError()`` before calling trace hooks implemented in
Python.
Code using the new frame view APIs will have the dynamic locals snapshot
implicitly refreshed when accessing methods that need it, while code using the
``PyEval_GetLocals()`` API will implicitly refresh it when making that call.
The PEP necessarily also drops the implicit call to ``PyFrame_LocalsToFast()``
when returning from a trace hook, as that API now always raises an exception.
Design Discussion
=================
Changing ``locals()`` to return independent snapshots at function scope
-----------------------------------------------------------------------
The ``locals()`` builtin is a required part of the language, and in the
reference implementation it has historically returned a mutable mapping with
the following characteristics:
* each call to ``locals()`` returns the *same* mapping object
* for namespaces where ``locals()`` returns a reference to something other than
the actual local execution namespace, each call to ``locals()`` updates the
mapping object with the current state of the local variables and any
referenced
nonlocal cells
* changes to the returned mapping *usually* aren't written back to the
local variable bindings or the nonlocal cell references, but write backs
can be triggered by doing one of the following:
* installing a Python level trace hook (write backs then happen whenever
the trace hook is called)
* running a function level wildcard import (requires bytecode
injection in Py3)
* running an ``exec`` statement in the function's scope (Py2 only, since
``exec`` became an ordinary builtin in Python 3)
Originally this PEP proposed to retain the first two of these properties,
while changing the third in order to address the outright behaviour bugs that
it can cause.
In [7]_ Nathaniel Smith made a persuasive case that we could make the behaviour
of ``locals()`` at function scope substantially less confusing by retaining only
the second property and having each call to ``locals()`` at function scope
return an *independent* snapshot of the local variables and closure references
rather than updating an implicitly shared snapshot.
As this revised design also made the implementation markedly easier to follow,
the PEP was updated to propose this change in behaviour, rather than retaining
the historical shared snapshot.
Keeping ``locals()`` as a snapshot at function scope
----------------------------------------------------
As discussed in [7]_, it would theoretically be possible to change the semantics
of the ``locals()`` builtin to return the write-through proxy at function scope,
rather than switching it to return independent snapshots.
This PEP doesn't (and won't) propose this as it's a backwards incompatible
change in practice, even though code that relies on the current behaviour is
technically operating in an undefined area of the language specification.
Consider the following code snippet::
def example():
x = 1
locals()["x"] = 2
print(x)
Even with a trace hook installed, that function will consistently print ``1``
on the current reference interpreter implementation::
>>> example()
1
>>> import sys
>>> def basic_hook(*args):
... return basic_hook
...
>>> sys.settrace(basic_hook)
>>> example()
1
Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins
at function scope (either explicitly or implicitly) without risking unexpected
rebinding of local variables or closure references.
Provoking the reference interpreter into incorrectly mutating the local variable
state requires a more complex setup where a nested function closes over a
variable being rebound in the outer function, and due to the use of either
threads, generators, or coroutines, it's possible for a trace function to start
running for the nested function before the rebinding operation in the outer
function, but finish running after the rebinding operation has taken place (in
which case the rebinding will be reverted, which is the bug reported in [1]_).
In addition to preserving the de facto semantics which have been in place since
PEP 227 introduced nested scopes in Python 2.1, the other benefit of restricting
the write-through proxy support to the implementation-defined frame object API
is that it means that only interpreter implementations which emulate the full
frame API need to offer the write-through capability at all, and that
JIT-compiled implementations only need to enable it when a frame introspection
API is invoked, or a trace hook is installed, not whenever ``locals()`` is
accessed at function scope.
Returning snapshots from ``locals()`` at function scope also means that static
analysis for function level code will be more reliable, as only access to the
frame machinery will allow rebinding of local and nonlocal variable
references in a way that is hidden from static analysis.
What happens with the default args for ``eval()`` and ``exec()``?
-----------------------------------------------------------------
These are formally defined as inheriting ``globals()`` and ``locals()`` from
the calling scope by default.
There isn't any need for the PEP to change these defaults, so it doesn't, and
``exec()`` and ``eval()`` will start running in a shallow copy of the local
namespace when that is what ``locals()`` returns.
This behaviour will have potential performance implications, especially
for functions with large numbers of local variables (e.g. if these functions
are called in a loop, calling ``globals()`` and ``locals()`` once before the
loop and then passing the namespace into the function explicitly will give the
same semantics and performance characteristics as the status quo, whereas
relying on the implicit default would create a new shallow copy of the local
namespace on each iteration).
(Note: the reference implementation draft PR has updated the ``locals()`` and
``vars()``, ``eval()``, and ``exec()`` builtins to use ``PyLocals_Get()``. The
``dir()`` builtin still uses ``PyEval_GetLocals()``, since it's only using it
to make a list from the keys).
Changing the frame API semantics in regular operation
-----------------------------------------------------
Earlier versions of this PEP proposed having the semantics of the frame
``f_locals`` attribute depend on whether or not a tracing hook was currently
installed - only providing the write-through proxy behaviour when a tracing hook
was active, and otherwise behaving the same as the historical ``locals()``
builtin.
That was adopted as the original design proposal for a couple of key reasons,
one pragmatic and one more philosophical:
* Object allocations and method wrappers aren't free, and tracing functions
aren't the only operations that access frame locals from outside the function.
Restricting the changes to tracing mode meant that the additional memory and
execution time overhead of these changes would be as close to zero in regular
operation as we can possibly make them.
* "Don't change what isn't broken": the current tracing mode problems are caused
by a requirement that's specific to tracing mode (support for external
rebinding of function local variable references), so it made sense to also
restrict any related fixes to tracing mode
However, actually attempting to implement and document that dynamic approach
highlighted the fact that it makes for a really subtle runtime state dependent
behaviour distinction in how ``frame.f_locals`` works, and creates several
new edge cases around how ``f_locals`` behaves as trace functions are added
and removed.
Accordingly, the design was switched to the current one, where
``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
a snapshot, which is both simpler to implement and easier to explain.
Regardless of how the CPython reference implementation chooses to handle this,
optimising compilers and interpreters also remain free to impose additional
restrictions on debuggers, such as making local variable mutation through frame
objects an opt-in behaviour that may disable some optimisations (just as the
emulation of CPython's frame API is already an opt-in flag in some Python
implementations).
Continuing to support storing additional data on optimised frames
-----------------------------------------------------------------
One of the draft iterations of this PEP proposed removing the ability to store
additional data on optimised frames by writing to ``frame.f_locals`` keys that
didn't correspond to local or closure variable names on the underlying frame.
While this idea offered some attractive simplification of the fast locals proxy
implementation, ``pdb`` stores ``__return__`` and ``__exception__`` values on
arbitrary frames, so the standard library test suite fails if that functionality
no longer works.
Accordingly, the ability to store arbitrary keys was retained, at the expense
of certain operations on proxy objects currently either being slower
than desired
(as they need to update the dynamic snapshot in order to provide correct
behaviour), or else assuming that the cache is currently up to date (and hence
potentially giving an incorrect answer if the frame state has changed in a
way that doesn't automatically update the cache contents).
It is expected that the exact details of the interaction between the fast locals
proxy and the ``f_locals`` value cache on the underlying frame will evolve over
time as opportunities for improvement are identified.
Historical semantics at function scope
--------------------------------------
The current semantics of mutating ``locals()`` and ``frame.f_locals`` in CPython
are rather quirky due to historical implementation details:
* actual execution uses the fast locals array for local variable bindings and
cell references for nonlocal variables
* there's a ``PyFrame_FastToLocals`` operation that populates the frame's
``f_locals`` attribute based on the current state of the fast locals array
and any referenced cells. This exists for three reasons:
* allowing trace functions to read the state of local variables
* allowing traceback processors to read the state of local variables
* allowing ``locals()`` to read the state of local variables
* a direct reference to ``frame.f_locals`` is returned from ``locals()``, so if
you hand out multiple concurrent references, then all those references will be
to the exact same dictionary
* the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, were
removed in the migration to Python 3: ``exec`` is no longer a statement (and
hence can no longer affect function local namespaces), and the compiler now
disallows the use of ``from module import *`` operations at function scope
* however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is called
as part of returning from a trace function (which allows debuggers to make
changes to the local variable state), and you can also still inject the
``IMPORT_STAR`` opcode when creating a function directly from a code object
rather than via the compiler
This proposal deliberately *doesn't* formalise these semantics as is, since they
only make sense in terms of the historical evolution of the language and the
reference implementation, rather than being deliberately designed.
Proposing several additions to the stable C API/ABI
---------------------------------------------------
Historically, the CPython C API (and subsequently, the stable ABI) has
exposed only a single API function related to the Python ``locals`` builtin:
``PyEval_GetLocals()``. However, as it returns a borrowed reference, it is
not possible to adapt that interface directly to supporting the new ``locals()``
semantics proposed in this PEP.
An earlier iteration of this PEP proposed a minimalist adaptation to the new
semantics: one C API function that behaved like the Python ``locals()`` builtin,
and another that behaved like the ``frame.f_locals`` descriptor (creating and
returning the write-through proxy if necessary).
The feedback [8]_ on that version of the C API was that it was too heavily based
on how the Python level semantics were implemented, and didn't account for the
behaviours that authors of C extensions were likely to *need*.
The broader API now being proposed came from grouping the potential reasons for
wanting to access the Python ``locals()`` namespace from an extension module
into the following cases:
* needing to exactly replicate the semantics of the Python level ``locals()``
operation. This is the ``PyLocals_Get()`` API.
* needing to behave differently depending on whether writes to the result of
``PyLocals_Get()`` will be visible to Python code or not. This is handled by
the ``PyLocals_GetReturnsCopy()`` query API.
* always wanting a mutable namespace that has been pre-populated from the
current Python ``locals()`` namespace, but *not* wanting any changes to
be visible to Python code. This is the ``PyLocals_GetCopy()`` API.
* always wanting a read-only view of the current locals namespace, without
incurring the runtime overhead of making a full copy each time. This is the
``PyLocals_GetView()`` API.
Historically, these kinds of checks and operations would only have been
possible if a Python implementation emulated the full CPython frame API. With
the proposed API, extension modules can instead ask more clearly for the
semantics that they actually need, giving Python implementations more
flexibility in how they provide those capabilities.
Implementation
==============
The reference implementation update is in development as a draft pull
request on GitHub ([6]_).
Acknowledgements
================
Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in
[1]_ and pointing out some critical design flaws in earlier iterations of the
PEP that attempted to avoid introducing such a proxy.
Thanks to Steve Dower and Petr Viktorin for asking that more attention be paid
to the developer experience of the proposed C API additions [8]_.
Thanks to Mark Shannon for pushing for further simplification of the C level
API and semantics (and restarting discussion on the PEP in early 2021 after a
few years of inactivity).
References
==========
.. [1] Broken local variable assignment given threads + trace hook + closure
(https://bugs.python.org/issue30744)
.. [2] Clarify the required behaviour of ``locals()``
(https://bugs.python.org/issue17960)
.. [3] Updating function local variables from pdb is unreliable
(https://bugs.python.org/issue9633)
.. [4] CPython's Python API for installing trace hooks
(https://docs.python.org/dev/library/sys.html#sys.settrace)
.. [5] CPython's C API for installing trace hooks
(https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace)
.. [6] PEP 558 reference implementation
(https://github.com/python/cpython/pull/3640/files)
.. [7] Nathaniel's review of possible function level semantics for locals()
(https://mail.python.org/pipermail/python-dev/2019-May/157738.html)
.. [8] Discussion of more intentionally designed C API enhancements
(https://discuss.python.org/t/pep-558-defined-semantics-for-locals/2936/3)
.. [9] Disable automatic update of frame locals during tracing
(https://bugs.python.org/issue42197)
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
7
17
In 3.10 the union type (the type of the result of the | operator for
types) was added (https://www.python.org/dev/peps/pep-0604/). It is
exposed as types.Union. There are differences between typing.Union and
types.Union:
* typing.Union is indexable, types.Union is not.
* types.Union is a class, typing.Union is not.
types.Union corresponds to private class typing._UnionGenericAlias, not
typing.Union. It is confusing that typing.Union and types.Union have the
same name but are so different. Note also that most classes in the types
module have the "Type" suffix: FunctionType, MethodType, ModuleType,
etc. I think that it would be better to rename types.Union to
types.UnionType.
The name of types.Union is the part of already accepted PEP 604, so we
need to hear opinions of authors, sponsor and BDFL-delegate of the PEP.
I did not participate in the discussion about PEP 604 so I do not know
if there are arguments in favor of types.Union over types.UnionType.
4
3
Hi,
When I find what's new in 3.10 beta in
https://docs.python.org/whatsnew/3.10.html
It redirected me to https://docs.python.org/3/whatsnew/3.10.html which
shows 404 error not found nginx. Can you fix this?
Sincerely
xXPartOfMeXx
5
5
ACTIVITY SUMMARY (2021-07-16 - 2021-07-23)
Python tracker at https://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 7420 (+17)
closed 49041 (+54)
total 56461 (+71)
Open issues with patches: 2951
Issues opened (52)
==================
#44656: Dangerous mismatch between MAXPATHLEN and MAX_PATH on Windows
https://bugs.python.org/issue44656 opened by izbyshev
#44657: instancemethod_call should use PyInstanceMethod_GET_FUNCTION m
https://bugs.python.org/issue44657 opened by colesbury
#44658: No ValueError for duplicate key value in mapping patern when l
https://bugs.python.org/issue44658 opened by jack__d
#44660: email.feedparser Module Lacks Support for Section 3.5 of RFC 6
https://bugs.python.org/issue44660 opened by f18a14c09s
#44662: Add ability to annotate types.Union
https://bugs.python.org/issue44662 opened by uriyyo
#44663: Possible bug in datetime utc
https://bugs.python.org/issue44663 opened by gabhcosta
#44664: builtins.chr and the 'c' format flag raise different errors
https://bugs.python.org/issue44664 opened by bup
#44665: asyncio.create_task() documentation should mention user needs
https://bugs.python.org/issue44665 opened by bernat
#44666: compileall.compile_file fails when sys.stdout is redirected to
https://bugs.python.org/issue44666 opened by stefanhoelzl
#44667: tokenize.py emits spurious NEWLINE if file ends on a comment w
https://bugs.python.org/issue44667 opened by mdartiailh
#44668: More differences in instance and subclass checks between typin
https://bugs.python.org/issue44668 opened by serhiy.storchaka
#44669: TypeError: 'type' object is not subscriptable
https://bugs.python.org/issue44669 opened by michal.dziczkowski
#44671: Create a built-in yaml module
https://bugs.python.org/issue44671 opened by jarpri08
#44673: Embedded Python - local directories in pythonXX._pth
https://bugs.python.org/issue44673 opened by emve
#44674: dataclasses should allow frozendict default value
https://bugs.python.org/issue44674 opened by gianni
#44675: Cross-platform issues with private methods and multiprocessing
https://bugs.python.org/issue44675 opened by ymerej
#44677: CSV sniffing falsely detects space as a delimiter
https://bugs.python.org/issue44677 opened by pt12lol
#44678: Seperate error message for discontinuous padding in binascii.a
https://bugs.python.org/issue44678 opened by idan22moral
#44679: unittest.mock.sentinel documentation typo
https://bugs.python.org/issue44679 opened by gaydayav
#44680: Reference cycles from a WeakKeyDictionary value to its key are
https://bugs.python.org/issue44680 opened by andersk
#44682: Pdb commands allows to add commands to invalid breakpoint
https://bugs.python.org/issue44682 opened by andrei.avk
#44684: Docs for mock.call
https://bugs.python.org/issue44684 opened by guettli
#44685: Email package issue with Outlook msg files
https://bugs.python.org/issue44685 opened by heghine
#44686: use pkgutil.resolve_name in unittest.mock
https://bugs.python.org/issue44686 opened by graingert
#44687: io.BufferedReader:peek() closes underlying file, breaking peek
https://bugs.python.org/issue44687 opened by liquidpele
#44688: [sqlite3] Remove ASCII limitation from sqlite3.Connection.crea
https://bugs.python.org/issue44688 opened by erlendaasland
#44689: MacOS: Python binaries not portable between Catalina and Big S
https://bugs.python.org/issue44689 opened by bergkvist
#44690: Adopt binacii.a2b_base64's strict mode in base64.b64decode
https://bugs.python.org/issue44690 opened by idan22moral
#44693: Unclear definition of the "__future__" module in Docs
https://bugs.python.org/issue44693 opened by StevenHsuYL
#44694: Message from BytesParser cannot be flattened immediately
https://bugs.python.org/issue44694 opened by vitas1
#44695: asdict use deep copy to dataclass instances
https://bugs.python.org/issue44695 opened by Itayazolay
#44697: Memory leak when asyncio.open_connection raise
https://bugs.python.org/issue44697 opened by seer
#44698: Undefined behaviour in Objects/complexobject.c's complex_pow
https://bugs.python.org/issue44698 opened by twouters
#44699: Simple regex appears to take exponential time in length of inp
https://bugs.python.org/issue44699 opened by brezniczky
#44701: Create a @deprecated decorator (annotation)
https://bugs.python.org/issue44701 opened by Leonardofreua
#44702: Fix weakref doc
https://bugs.python.org/issue44702 opened by Prometheus3375
#44705: Support Windows file open modes for `open` built-in function
https://bugs.python.org/issue44705 opened by lukedeller1
#44707: runtime error: applying zero offset to null pointer in Objects
https://bugs.python.org/issue44707 opened by thatiparthy
#44709: [3.7] Popen Control Characters in stdout affect shell session
https://bugs.python.org/issue44709 opened by San
#44711: Optimize type check in pipes.py
https://bugs.python.org/issue44711 opened by anton.gruebel
#44712: Replace `type(literal)` with corresponding builtin types
https://bugs.python.org/issue44712 opened by serhiy.storchaka
#44717: Improve AttributeError on circular imports of submodules
https://bugs.python.org/issue44717 opened by FFY00
#44718: Incorrect arguments in function select() cause segfault
https://bugs.python.org/issue44718 opened by xxm
#44719: Incorrect callable object crashes Python 3.11.0a0
https://bugs.python.org/issue44719 opened by xxm
#44720: Finding string in iteratively deleted object cause segfault
https://bugs.python.org/issue44720 opened by xxm
#44722: RFC: string Multiline Formatter
https://bugs.python.org/issue44722 opened by creative-resort
#44723: Codec name normalization breaks custom codecs
https://bugs.python.org/issue44723 opened by bodograumann
#44724: Resource Tracker is never reaped
https://bugs.python.org/issue44724 opened by viktor.ivanov
#44725: Expose specialization stats in python
https://bugs.python.org/issue44725 opened by iritkatriel
#44726: Build macOS version with thin lto option
https://bugs.python.org/issue44726 opened by corona10
#44727: Stable ABI should avoid `enum`
https://bugs.python.org/issue44727 opened by petr.viktorin
#44728: Testsuite fails on x86_64
https://bugs.python.org/issue44728 opened by deep42thought
Most recent 15 issues with no replies (15)
==========================================
#44728: Testsuite fails on x86_64
https://bugs.python.org/issue44728
#44725: Expose specialization stats in python
https://bugs.python.org/issue44725
#44724: Resource Tracker is never reaped
https://bugs.python.org/issue44724
#44723: Codec name normalization breaks custom codecs
https://bugs.python.org/issue44723
#44720: Finding string in iteratively deleted object cause segfault
https://bugs.python.org/issue44720
#44719: Incorrect callable object crashes Python 3.11.0a0
https://bugs.python.org/issue44719
#44717: Improve AttributeError on circular imports of submodules
https://bugs.python.org/issue44717
#44702: Fix weakref doc
https://bugs.python.org/issue44702
#44701: Create a @deprecated decorator (annotation)
https://bugs.python.org/issue44701
#44698: Undefined behaviour in Objects/complexobject.c's complex_pow
https://bugs.python.org/issue44698
#44695: asdict use deep copy to dataclass instances
https://bugs.python.org/issue44695
#44694: Message from BytesParser cannot be flattened immediately
https://bugs.python.org/issue44694
#44690: Adopt binacii.a2b_base64's strict mode in base64.b64decode
https://bugs.python.org/issue44690
#44688: [sqlite3] Remove ASCII limitation from sqlite3.Connection.crea
https://bugs.python.org/issue44688
#44687: io.BufferedReader:peek() closes underlying file, breaking peek
https://bugs.python.org/issue44687
Most recent 15 issues waiting for review (15)
=============================================
#44726: Build macOS version with thin lto option
https://bugs.python.org/issue44726
#44725: Expose specialization stats in python
https://bugs.python.org/issue44725
#44722: RFC: string Multiline Formatter
https://bugs.python.org/issue44722
#44717: Improve AttributeError on circular imports of submodules
https://bugs.python.org/issue44717
#44712: Replace `type(literal)` with corresponding builtin types
https://bugs.python.org/issue44712
#44711: Optimize type check in pipes.py
https://bugs.python.org/issue44711
#44707: runtime error: applying zero offset to null pointer in Objects
https://bugs.python.org/issue44707
#44698: Undefined behaviour in Objects/complexobject.c's complex_pow
https://bugs.python.org/issue44698
#44690: Adopt binacii.a2b_base64's strict mode in base64.b64decode
https://bugs.python.org/issue44690
#44682: Pdb commands allows to add commands to invalid breakpoint
https://bugs.python.org/issue44682
#44678: Seperate error message for discontinuous padding in binascii.a
https://bugs.python.org/issue44678
#44677: CSV sniffing falsely detects space as a delimiter
https://bugs.python.org/issue44677
#44666: compileall.compile_file fails when sys.stdout is redirected to
https://bugs.python.org/issue44666
#44662: Add ability to annotate types.Union
https://bugs.python.org/issue44662
#44660: email.feedparser Module Lacks Support for Section 3.5 of RFC 6
https://bugs.python.org/issue44660
Top 10 most discussed issues (10)
=================================
#43950: Include column offsets for bytecode instructions
https://bugs.python.org/issue43950 13 msgs
#42414: unable to document fields of dataclass
https://bugs.python.org/issue42414 9 msgs
#44689: MacOS: Python binaries not portable between Catalina and Big S
https://bugs.python.org/issue44689 6 msgs
#43838: There is a way to access an underlying mapping in MappingProxy
https://bugs.python.org/issue43838 5 msgs
#44549: BZip 1.0.6 Critical Vulnerability
https://bugs.python.org/issue44549 5 msgs
#44663: Possible bug in datetime utc
https://bugs.python.org/issue44663 5 msgs
#44711: Optimize type check in pipes.py
https://bugs.python.org/issue44711 5 msgs
#30511: shutil.make_archive should not need to chdir (alternatively: m
https://bugs.python.org/issue30511 4 msgs
#44603: REPL: exit when the user types exit instead of asking them to
https://bugs.python.org/issue44603 4 msgs
#44699: Simple regex appears to take exponential time in length of inp
https://bugs.python.org/issue44699 4 msgs
Issues closed (54)
==================
#14879: invalid docs for subprocess exceptions with shell=True
https://bugs.python.org/issue14879 closed by iritkatriel
#27513: email.utils.getaddresses does not handle Header objects
https://bugs.python.org/issue27513 closed by lukasz.langa
#29555: Update Python Software Foundation Copyright Year
https://bugs.python.org/issue29555 closed by orsenthil
#33063: failed to build _ctypes: undefined reference to `ffi_closure_F
https://bugs.python.org/issue33063 closed by zach.ware
#40897: Inheriting from class that defines __new__ causes inspect.sign
https://bugs.python.org/issue40897 closed by lukasz.langa
#41249: TypedDict inheritance doesn't work with get_type_hints and pos
https://bugs.python.org/issue41249 closed by lukasz.langa
#41546: pprint() gives exception when ran from pythonw
https://bugs.python.org/issue41546 closed by iritkatriel
#41972: bytes.find consistently hangs in a particular scenario
https://bugs.python.org/issue41972 closed by lukasz.langa
#42095: plistlib: Add tests that compare with plutil(1)
https://bugs.python.org/issue42095 closed by lukasz.langa
#42355: symtable: get_namespace doesn't check whether if there are mul
https://bugs.python.org/issue42355 closed by BTaskaya
#42581: Docs site redirection doesn't work for 3.9
https://bugs.python.org/issue42581 closed by ned.deily
#42747: Remove Py_TPFLAGS_HAVE_VERSION_TAG flag?
https://bugs.python.org/issue42747 closed by petr.viktorin
#43086: Excess data in not handled properly in binascii.a2b_base64()
https://bugs.python.org/issue43086 closed by gregory.p.smith
#43629: fix _PyRun_SimpleFileObject create __main__ module and cache.
https://bugs.python.org/issue43629 closed by petr.viktorin
#44340: Add support for building cpython with clang thin lto
https://bugs.python.org/issue44340 closed by corona10
#44353: PEP 604 NewType
https://bugs.python.org/issue44353 closed by kj
#44435: There is no description of PY_SSIZE_T_CLEAN in docs
https://bugs.python.org/issue44435 closed by jack__d
#44490: PEP 604 Union (int | str) doesn't have __parameters__
https://bugs.python.org/issue44490 closed by gvanrossum
#44524: __name__ attribute in typing module
https://bugs.python.org/issue44524 closed by lukasz.langa
#44539: Support recognizing JPEG files without JFIF or Exif markers
https://bugs.python.org/issue44539 closed by lukasz.langa
#44554: pdb.main is unnecessarily complicated
https://bugs.python.org/issue44554 closed by iritkatriel
#44561: Some expired hyperlinks in Python documentation
https://bugs.python.org/issue44561 closed by StevenHsuYL
#44566: StopIteration subclass suppressed by contextlib.contextmanager
https://bugs.python.org/issue44566 closed by lukasz.langa
#44589: Pattern Matching - duplicate keys in mapping patterns
https://bugs.python.org/issue44589 closed by pablogsal
#44610: Format issue with strftime and %Y
https://bugs.python.org/issue44610 closed by bkaznowski
#44611: CPython uses deprecated randomness API
https://bugs.python.org/issue44611 closed by corona10
#44621: Python 3.9 traces async for/else incorrectly
https://bugs.python.org/issue44621 closed by lukasz.langa
#44628: Remove the broken link for issue #445902 in unixcompiler.py (d
https://bugs.python.org/issue44628 closed by eric.araujo
#44629: Some files from distutils module are importing all exceptions
https://bugs.python.org/issue44629 closed by eric.araujo
#44631: Refactoring the repr() of the _Environ class (os module)
https://bugs.python.org/issue44631 closed by lukasz.langa
#44633: Indexing the union type can return NotImplemented
https://bugs.python.org/issue44633 closed by serhiy.storchaka
#44634: Version is duplicated in name of app in list of installed apps
https://bugs.python.org/issue44634 closed by zach.ware
#44651: An unclear definition in Doc/glossary.rst
https://bugs.python.org/issue44651 closed by mark.dickinson
#44653: Parameter substitution in the union type does not work with ty
https://bugs.python.org/issue44653 closed by lukasz.langa
#44654: Refactor and clean up the union type implementation
https://bugs.python.org/issue44654 closed by serhiy.storchaka
#44655: Confusing message with AttributeError when attribute name matc
https://bugs.python.org/issue44655 closed by pablogsal
#44659: Remove Ivan from list of typing experts
https://bugs.python.org/issue44659 closed by lukasz.langa
#44661: Update property_descr_set to use vectorcall if possible.
https://bugs.python.org/issue44661 closed by corona10
#44670: bug on showing tuple on console
https://bugs.python.org/issue44670 closed by zach.ware
#44672: Final "pass" is traced incorrectly in 3.9 (and before)
https://bugs.python.org/issue44672 closed by lukasz.langa
#44676: Add ability to serialise types.Union
https://bugs.python.org/issue44676 closed by lukasz.langa
#44681: time.sleep(0.001) not working properly
https://bugs.python.org/issue44681 closed by therenoisfood
#44683: Can't subscript objects with the string "1" using str.format()
https://bugs.python.org/issue44683 closed by eric.smith
#44691: bug in interactions between argparse and random
https://bugs.python.org/issue44691 closed by steven.daprano
#44692: Const folding in parser with negative numbers doesn't match fl
https://bugs.python.org/issue44692 closed by steven.daprano
#44696: Python 2.0.1 Installation:
https://bugs.python.org/issue44696 closed by steven.daprano
#44700: Python fails to build (aarch64-apple-darwin20.5.0)
https://bugs.python.org/issue44700 closed by jack__d
#44703: deepcopy(frozenset()) returns a new object
https://bugs.python.org/issue44703 closed by rhettinger
#44704: frozenset.__hash__ vs. Set._hash
https://bugs.python.org/issue44704 closed by rhettinger
#44706: UUID constructor should accept another UUID instance
https://bugs.python.org/issue44706 closed by serhiy.storchaka
#44708: Regression tests with -w should only re-run affected test meth
https://bugs.python.org/issue44708 closed by lukasz.langa
#44710: Unexpected behavior in empty class with pass (Python 3.7.3)
https://bugs.python.org/issue44710 closed by Cheukting
#44713: subprocess.rst typo ``"shell=True"`` => ``shell=True``
https://bugs.python.org/issue44713 closed by eric.araujo
#44721: Problem in tkinter button widget
https://bugs.python.org/issue44721 closed by a.h.misaghi
1
0
RFC for PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors
by Ethan Furman 21 Jul '21
by Ethan Furman 21 Jul '21
21 Jul '21
PEP: 663
Title: Improving and Standardizing Enum str(), repr(), and format() behaviors
Version: $Revision$
Last-Modified: $Date$
Author: Ethan Furman <ethan(a)stoneleaf.us>
Discussions-To: python-dev(a)python.org
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 23-Feb-2013
Python-Version: 3.11
Post-History: 20-Jul-2021
Resolution:
Abstract
========
Now that we have a few years experience with Enum usage it is time to update
the ``repr()``, ``str()``, and ``format()`` of the various enumerations by their
intended purpose.
Motivation
==========
The addition of ``StrEnum`` with its requirement to have its ``str()`` be its
``value`` is inconsistent with other provided Enum's ``str``.
Having the ``str()`` of ``IntEnum`` and ``IntFlag`` not be the value causes
bugs and extra work when replacing existing constants.
Having the ``str()`` and ``format()`` of an enum member be different can be
confusing.
The iteration of ``Flag`` members, which directly affects their ``repr()``, is
inelegant at best, and buggy at worst.
Rationale
=========
Enums are becoming more common in the standard library; being able to recognize
enum members by their ``repr()``, and having that ``repr()`` be easy to parse, is
useful and can save time and effort in understanding and debugging code.
However, the enums with mixed-in data types (``IntEnum``, ``IntFlag``, and the new
``StrEnum``) need to be more backwards compatible with the constants they are
replacing -- specifically, ``str(replacement_enum_member) == str(original_constant)``
should be true (and the same for ``format()``).
IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement of
existing integer and string constants as is possible. Towards that goal, the
str() output of each should be its inherent value; i.e.::
>>> Color.RED
<Color.RED: 1>
>>> str(Color.RED)
1
>>> format(Color.RED)
'1'
Note that format() already produces the correct output, only str() needs
updating.
As much as possible, the ``str()`, ``repr()``, and ``format()`` of enum members
should be standardized across the stardard library.
The repr() of Flag currently includes aliases, which it should not; fixing that
will, of course, already change its ``repr()`` in certain cases.
Specification
=============
There a three broad categories of enum usage:
- standard: Enum or Flag
a new enum class is created, and the members are used as ``class.member_name``
- drop-in replacement: IntEnum, IntFlag, StrEnum
a new enum class is created which also subclasses ``int`` or ``str`` and uses
``int.__str__`` or ``str.__str__``; the ``repr`` can be changed by using the
``global_enum`` decorator
- user-mixed enums and flags
the user creates their own integer-, float-, str-, whatever-enums instead of
using enum.IntEnum, etc.
Some sample enums::
# module: tools.py
class Hue(Enum): # or IntEnum
LIGHT = -1
NORMAL = 0
DARK = +1
class Color(Flag): # or IntFlag
RED = 1
GREEN = 2
BLUE = 4
class Grey(int, Enum): # or (int, Flag)
BLACK = 0
WHITE = 1
Using the above enumerations, the following table shows the old and new
behavior, while the last shows the final result:
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| type | enum repr() | enum str() | enum format() | flag repr() | flag str()
| flag format() |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| standard | 3.9 | | | | <Color.RED|GREEN: 3> |
Color.RED|GREEN | Color.RED|GREEN |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | | | | <Color(3): RED|GREEN> |
Color.RED|Color.GREEN | Color.RED|Color.GREEN |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| user mixed | 3.9 | | | 1 | <Grey.WHITE: 1> |
| 1 |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | | | Grey.WHITE | <Grey(1): WHITE> |
| Grey.WHITE |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| int drop-in | 3.9 | | Hue.LIGHT | | <Color.RED|GREEN: 3> |
Color.RED|GREEN | |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | | -1 | | <Color(3): RED|GREEN> | 3
| |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| global | 3.9 | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color.RED|GREEN: 3> |
Color.RED|GREEN | Color.RED|GREEN |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN
| RED|GREEN |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| user mixed | 3.9 | <Grey.WHITE: 1 | Grey.WHITE | Grey.WHITE | <Grey.WHITE: 1> | Grey.WHITE
| 1 |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE
| WHITE |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| int drop-in | 3.9 | <Hue.LIGHT: -1> | Hue.LIGHT | | <Color.RED|GREEN: 3> |
Color.RED|GREEN | |
|
+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| | new | tools.LIGHT | -1 | | tools.RED|tools.GREEN | 3
| |
+-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
Which will result in:
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| type | enum repr() | enum str() | enum format() | flag repr() | flag str() |
flag format() |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| standard | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color(3): RED|GREEN> | Color.RED|Color.GREEN |
Color.RED|Color.GREEN |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| user mixed | <Grey.WHITE: 1> | Grey.WHITE | Grey.WHITE | <Grey(1): WHITE> | Grey.WHITE |
Grey.WHITE |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| int drop-in | <Hue.LIGHT: -1> | -1 | -1 | <Color(3): RED|GREEN> | 3 |
3 |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| global | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN |
RED|GREEN |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| user mixed | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE |
WHITE |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
| int drop-in | tools.LIGHT | -1 | -1 | tools.RED|tools.GREEN | 3 |
3 |
+-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+
As can be seen, ``repr()`` is primarily affected by whether the members are
global, while ``str()`` is affected by being global or by being a drop-in
replacement, with the drop-in replacement status having a higher priority.
Also, the basic ``repr()`` and ``str()`` have changed for flags as the old
style was very clunky.
The ``repr()`` for Enum vs Flag are different, primarily because the Enum
``repr()`` does not work well for flags. I like being able to tell whether
an enum member is a Flag or an Enum based on the ``repr()`` alone, but am open
to arguments for changing Enum's ``repr()`` to match Flag's.
Backwards Compatibility
=======================
My understanding is that ``str()`` and ``repr()`` output has much lower
backwards compatibility requirements. Even so, I expect the majority of
breakage to be in doc and unit tests. I'm less clear on the policy for
``format()``.
Note that by changing the ``str()`` of the drop-in category, we will actually
prevent future breakage when ``IntEnum``, et al, are used to replace existing
constants.
Mitigation
==========
Normal usage of enum members will not change: ``re.ASCII`` can still be used
as ``re.ASCII`` and will still compare equal to ``256``. If one wants their
own enums in their own code to remain the same they will need to write their
own base Enum class and then write the appropriate ``repr``, ``str()``, and
``format()`` methods (or copy them from the 3.10 enum module).
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
2
2
ACTIVITY SUMMARY (2021-07-09 - 2021-07-16)
Python tracker at https://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 7403 ( +6)
closed 48987 (+56)
total 56390 (+62)
Open issues with patches: 2951
Issues opened (40)
==================
#29753: [Linux] ctypes packs bitfields Incorrectly
https://bugs.python.org/issue29753 reopened by FFY00
#44184: crash on windows invoking flake8
https://bugs.python.org/issue44184 reopened by twouters
#44592: tkinter focus_get() with non-tkinter Tk widget
https://bugs.python.org/issue44592 reopened by terry.reedy
#44594: AsyncExitStack.enter_async_context() is mishandling exception
https://bugs.python.org/issue44594 opened by John Belmonte
#44597: Poll returns POLLOUT on Pipe read endpoint on MacOS 10.14
https://bugs.python.org/issue44597 opened by samh42
#44598: test_constructor (test.test_ssl.ContextTests) ... Fatal Python
https://bugs.python.org/issue44598 opened by sxt1001
#44600: match/case statements trace incorrectly in 3.10.0b4
https://bugs.python.org/issue44600 opened by nedbat
#44601: argparse: potential bugs on add_subparser due to allow_abbrev
https://bugs.python.org/issue44601 opened by zhongxiang117
#44603: REPL: exit when the user types exit instead of asking them to
https://bugs.python.org/issue44603 opened by theacodes
#44604: [Enhancement] Asyncio task decorator to provide interface to d
https://bugs.python.org/issue44604 opened by aa1371
#44605: functools.total_ordering doesn't work with metaclasses
https://bugs.python.org/issue44605 opened by glyph
#44607: Teleport method for turtle class
https://bugs.python.org/issue44607 opened by Muffinlicious
#44609: Buffer support in the stable ABI
https://bugs.python.org/issue44609 opened by tarun.johar
#44610: Format issue with strftime and %Y
https://bugs.python.org/issue44610 opened by bkaznowski
#44611: CPython uses deprecated randomness API
https://bugs.python.org/issue44611 opened by strombrg
#44614: Broken Pipe in Server of Manager in multiprocessing when final
https://bugs.python.org/issue44614 opened by mutax
#44615: Multiprocessing Pool example and comments incomplete/wrong?
https://bugs.python.org/issue44615 opened by mutax
#44617: Undesired Behavior on `match` using Singleton object
https://bugs.python.org/issue44617 opened by thepabloaguilar
#44618: inspect.signature does not work for datetime classes
https://bugs.python.org/issue44618 opened by mauvilsa
#44620: UUIDv1 is not RFC 4122 compliant
https://bugs.python.org/issue44620 opened by contact2
#44621: Python 3.9 traces async for/else incorrectly
https://bugs.python.org/issue44621 opened by nedbat
#44624: Script name for venv PowerShell activate
https://bugs.python.org/issue44624 opened by pwatson
#44625: Python C API version of `fractions` module
https://bugs.python.org/issue44625 opened by lycantropos
#44627: Python terminal cmd line recall
https://bugs.python.org/issue44627 opened by jggammon
#44628: Remove the broken link for issue #445902 in unixcompiler.py (d
https://bugs.python.org/issue44628 opened by Leonardofreua
#44629: Some files from distutils module are importing all exceptions
https://bugs.python.org/issue44629 opened by Leonardofreua
#44631: Refactoring the repr() of the _Environ class (os module)
https://bugs.python.org/issue44631 opened by Leonardofreua
#44633: Indexing the union type can return NotImplemented
https://bugs.python.org/issue44633 opened by serhiy.storchaka
#44634: Version is duplicated in name of app in list of installed apps
https://bugs.python.org/issue44634 opened by rakleed
#44637: Quoting issue on header Reply-To
https://bugs.python.org/issue44637 opened by Abridbus
#44640: Fix punctuation in isinstance() error message
https://bugs.python.org/issue44640 opened by wyz23x2
#44642: Union of a type and the typing module function
https://bugs.python.org/issue44642 opened by serhiy.storchaka
#44643: importing does not dispatch to __builtins__.__getitem__ to loo
https://bugs.python.org/issue44643 opened by pxeger
#44644: Implement ���Happy Eyeballs��� algorithim (RFC 8503) in socket
https://bugs.python.org/issue44644 opened by origin400-p
#44645: Python 3.10: Under some trivial circunstances, GIL not release
https://bugs.python.org/issue44645 opened by jcea
#44648: Inspect.getsource raises wrong error on classes in interactive
https://bugs.python.org/issue44648 opened by andrei.avk
#44649: dataclasses slots with init=False field raises AttributeExcept
https://bugs.python.org/issue44649 opened by tefra
#44651: An unclear definition in Doc/glossary.rst
https://bugs.python.org/issue44651 opened by StevenHsuYL
#44653: Parameter substitution in the union type does not work with ty
https://bugs.python.org/issue44653 opened by serhiy.storchaka
#44654: Refactor and clean up the union type implementation
https://bugs.python.org/issue44654 opened by serhiy.storchaka
Most recent 15 issues with no replies (15)
==========================================
#44653: Parameter substitution in the union type does not work with ty
https://bugs.python.org/issue44653
#44651: An unclear definition in Doc/glossary.rst
https://bugs.python.org/issue44651
#44649: dataclasses slots with init=False field raises AttributeExcept
https://bugs.python.org/issue44649
#44644: Implement ���Happy Eyeballs��� algorithim (RFC 8503) in socket
https://bugs.python.org/issue44644
#44631: Refactoring the repr() of the _Environ class (os module)
https://bugs.python.org/issue44631
#44629: Some files from distutils module are importing all exceptions
https://bugs.python.org/issue44629
#44628: Remove the broken link for issue #445902 in unixcompiler.py (d
https://bugs.python.org/issue44628
#44624: Script name for venv PowerShell activate
https://bugs.python.org/issue44624
#44620: UUIDv1 is not RFC 4122 compliant
https://bugs.python.org/issue44620
#44615: Multiprocessing Pool example and comments incomplete/wrong?
https://bugs.python.org/issue44615
#44614: Broken Pipe in Server of Manager in multiprocessing when final
https://bugs.python.org/issue44614
#44610: Format issue with strftime and %Y
https://bugs.python.org/issue44610
#44609: Buffer support in the stable ABI
https://bugs.python.org/issue44609
#44601: argparse: potential bugs on add_subparser due to allow_abbrev
https://bugs.python.org/issue44601
#44590: Create frame objects lazily when needed
https://bugs.python.org/issue44590
Most recent 15 issues waiting for review (15)
=============================================
#44654: Refactor and clean up the union type implementation
https://bugs.python.org/issue44654
#44648: Inspect.getsource raises wrong error on classes in interactive
https://bugs.python.org/issue44648
#44645: Python 3.10: Under some trivial circunstances, GIL not release
https://bugs.python.org/issue44645
#44640: Fix punctuation in isinstance() error message
https://bugs.python.org/issue44640
#44631: Refactoring the repr() of the _Environ class (os module)
https://bugs.python.org/issue44631
#44628: Remove the broken link for issue #445902 in unixcompiler.py (d
https://bugs.python.org/issue44628
#44618: inspect.signature does not work for datetime classes
https://bugs.python.org/issue44618
#44614: Broken Pipe in Server of Manager in multiprocessing when final
https://bugs.python.org/issue44614
#44611: CPython uses deprecated randomness API
https://bugs.python.org/issue44611
#44603: REPL: exit when the user types exit instead of asking them to
https://bugs.python.org/issue44603
#44594: AsyncExitStack.enter_async_context() is mishandling exception
https://bugs.python.org/issue44594
#44590: Create frame objects lazily when needed
https://bugs.python.org/issue44590
#44589: Pattern Matching - duplicate keys in mapping patterns
https://bugs.python.org/issue44589
#44584: Deprecate thread debugging PYTHONTHREADDEBUG=1
https://bugs.python.org/issue44584
#44583: Failure to build on OSF1.
https://bugs.python.org/issue44583
Top 10 most discussed issues (10)
=================================
#44603: REPL: exit when the user types exit instead of asking them to
https://bugs.python.org/issue44603 40 msgs
#42958: filecmp.cmp(shallow=True) isn't actually shallow when only mti
https://bugs.python.org/issue42958 8 msgs
#40897: Inheriting from class that defines __new__ causes inspect.sign
https://bugs.python.org/issue40897 7 msgs
#44589: Pattern Matching - duplicate keys in mapping patterns
https://bugs.python.org/issue44589 7 msgs
#44592: tkinter focus_get() with non-tkinter Tk widget
https://bugs.python.org/issue44592 7 msgs
#44645: Python 3.10: Under some trivial circunstances, GIL not release
https://bugs.python.org/issue44645 7 msgs
#29753: [Linux] ctypes packs bitfields Incorrectly
https://bugs.python.org/issue29753 6 msgs
#44571: itertools: takedowhile()
https://bugs.python.org/issue44571 6 msgs
#44604: [Enhancement] Asyncio task decorator to provide interface to d
https://bugs.python.org/issue44604 6 msgs
#44637: Quoting issue on header Reply-To
https://bugs.python.org/issue44637 6 msgs
Issues closed (52)
==================
#15907: move doctest test-data files into a subdirectory of Lib/test
https://bugs.python.org/issue15907 closed by iritkatriel
#26280: ceval: Optimize list[int] (subscript) operation similarly to C
https://bugs.python.org/issue26280 closed by iritkatriel
#26329: os.path.normpath("//") returns //
https://bugs.python.org/issue26329 closed by lukasz.langa
#27875: Syslogs /usr/sbin/foo as /foo instead of as foo
https://bugs.python.org/issue27875 closed by ned.deily
#31791: Ensure that all PyTypeObject fields are set to non-NULL defaul
https://bugs.python.org/issue31791 closed by iritkatriel
#33346: Syntax error with async generator inside dictionary comprehens
https://bugs.python.org/issue33346 closed by pablogsal
#34292: test_compile hangs in AMD Ubuntu buildbots
https://bugs.python.org/issue34292 closed by pablogsal
#34388: collect_gdb fails for test.pythoninfo in several AMD64 FreeBSD
https://bugs.python.org/issue34388 closed by pablogsal
#34932: Add macOS TCP_KEEPALIVE to available socket options
https://bugs.python.org/issue34932 closed by pablogsal
#37271: Make multiple passes of the peephole optimizer until bytecode
https://bugs.python.org/issue37271 closed by pablogsal
#38741: Definition of multiple ']' in header configparser
https://bugs.python.org/issue38741 closed by lukasz.langa
#38842: test_multiprocessing_spawn altered the execution environment i
https://bugs.python.org/issue38842 closed by pablogsal
#42073: classmethod does not pass "type/owner" when invoking wrapped _
https://bugs.python.org/issue42073 closed by lukasz.langa
#42194: Docs for argparse.BooleanOptionalAction missing "New in versio
https://bugs.python.org/issue42194 closed by lukasz.langa
#42799: Please document fnmatch LRU cache size (256) and suggest alter
https://bugs.python.org/issue42799 closed by lukasz.langa
#43126: IOBase.readlines(0) behaviour is inconsistent with documentati
https://bugs.python.org/issue43126 closed by lukasz.langa
#43207: InspectLoader.is_package is not an abstract method
https://bugs.python.org/issue43207 closed by lukasz.langa
#43219: shutil.copy raises IsADirectoryError when the directory does n
https://bugs.python.org/issue43219 closed by gregory.p.smith
#43948: sysconfig's osx_framework_user puts headers in different locat
https://bugs.python.org/issue43948 closed by lukasz.langa
#44317: Suggestion for better syntax errors in tokenizer errors
https://bugs.python.org/issue44317 closed by pablogsal
#44514: configparser.rst & bz2.rst leave temp files after 'make doctes
https://bugs.python.org/issue44514 closed by lukasz.langa
#44516: Update bundled pip to 21.1.3
https://bugs.python.org/issue44516 closed by sbidoul
#44517: test__xxsubinterpreters: AMD64 Fedora Stable 3.x buildbot abor
https://bugs.python.org/issue44517 closed by erlendaasland
#44568: test_constructor (test.test_ssl.ContextTests) ... Fatal Python
https://bugs.python.org/issue44568 closed by sxt1001
#44569: traceback.py: Allow customization of per-frame line formatting
https://bugs.python.org/issue44569 closed by pablogsal
#44572: Calls to platform._syscmd_ver() dependent functions consume ST
https://bugs.python.org/issue44572 closed by steve.dower
#44581: Interpreter can execute quickened opcodes in tracing mode
https://bugs.python.org/issue44581 closed by Mark.Shannon
#44593: _randbelow_with_getrandbits may request an unnecessary random
https://bugs.python.org/issue44593 closed by lormuc
#44595: round() gives wrong result
https://bugs.python.org/issue44595 closed by steven.daprano
#44596: Operand coercion breaks MappingProxyType encapsulation
https://bugs.python.org/issue44596 closed by serhiy.storchaka
#44599: Changing logging format for one handler changes it for all
https://bugs.python.org/issue44599 closed by vinay.sajip
#44602: Issue with get_build_version in msvc9compiler.py in distutils
https://bugs.python.org/issue44602 closed by steve.dower
#44606: Discrepancy between isinstance() and issubclass() for union ty
https://bugs.python.org/issue44606 closed by serhiy.storchaka
#44608: Memory use increase in function _tkinter._flatten
https://bugs.python.org/issue44608 closed by serhiy.storchaka
#44612: inspect module points that could be using f-String
https://bugs.python.org/issue44612 closed by ammar2
#44613: Make importlib.metadata non-provisional in Python 3.10
https://bugs.python.org/issue44613 closed by barry
#44616: Incorrect tracing for "except" with variable
https://bugs.python.org/issue44616 closed by pablogsal
#44619: Bug in Python 3.7.10
https://bugs.python.org/issue44619 closed by steven.daprano
#44622: async-for loops are traced incorrectly in Python 3.10
https://bugs.python.org/issue44622 closed by lukasz.langa
#44623: help(open('/dev/zero').writelines) gives no help
https://bugs.python.org/issue44623 closed by zach.ware
#44626: Incorrect tracing of nested if/if/for/yield
https://bugs.python.org/issue44626 closed by pablogsal
#44630: Assertion failure in csv module
https://bugs.python.org/issue44630 closed by gregory.p.smith
#44632: Union with TypeVar does not work as intended
https://bugs.python.org/issue44632 closed by serhiy.storchaka
#44635: Convert None to NoneType in the union type constructor
https://bugs.python.org/issue44635 closed by serhiy.storchaka
#44636: It is possible to create a 1-type union type
https://bugs.python.org/issue44636 closed by serhiy.storchaka
#44638: zipfile.ZipFile is closed when zipfile.Path is closed
https://bugs.python.org/issue44638 closed by jaraco
#44639: [sqlite3] Confusing typo "transation"
https://bugs.python.org/issue44639 closed by erlendaasland
#44641: [sqlite3] Use vectorcall in pysqlite_collation_callback
https://bugs.python.org/issue44641 closed by erlendaasland
#44646: hash() of the unity type is not consistent with equality
https://bugs.python.org/issue44646 closed by lukasz.langa
#44647: Non-ASCII characters in os.environ cause silent failures in te
https://bugs.python.org/issue44647 closed by lukasz.langa
#44650: Move test_futures files*.py into a subdirectory of Lib/test
https://bugs.python.org/issue44650 closed by iritkatriel
#44652: Preserve natural order of args in the union type
https://bugs.python.org/issue44652 closed by kj
1
0
importing does not dispatch to __builtins__.__getitem__ to lookup __import__
by Patrick Reader 14 Jul '21
by Patrick Reader 14 Jul '21
14 Jul '21
Hi there,
I've noticed that, when a frame's __builtins__ is a subclass of dict with an overridden __getitem__ method, this overriden method is not used by the IMPORT_NAME instruction to lookup __import__ in the dictionary; it uses the lookup function of normal dictionaries (via _PyDict_GetItemIdWithError). This is contrary to the behaviour of the similar LOAD_BUILD_CLASS, as well as the typical name lookup semantics of LOAD_GLOBAL/LOAD_NAME, which all use PyDict_CheckExact for a "fast path" before defaulting to PyObject_GetItem, which is unexpected.
Perhaps more seriously, if __builtins__ is not a dict at all, then it gets erroneously passed to some internal dict functions resulting in a mysterious SystemError ("Objects/dictobject.c:1440: bad argument to internal function") which, to me, indicates fragile behaviour that isn't supposed to happen.
I'm not sure if this intended, so I didn't want to open an issue yet. It also seems a highly specific use case and changing it would probably cause a bit of a slow-down in module importing so is perhaps not worth fixing. I just wanted to ask here in case this issue had been documented anywhere before, and to check if it might actually be supposed to happen before opening a bug report.
I cannot find evidence that this behaviour has changed at all in recent history and it seems to be the same on the main branch as in 3.9.6.
A short demo of these things is attached.
Links to relevant CPython code in v3.9.6:
IMPORT_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L5179
BUILD_CLASS: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2316
LOAD_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2488
LOAD_GLOBAL: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2546
Thanks,
Patrick Reader
2
1
I have spoken with Pablo (3.10 RM), and he agrees that a change to Enum str() in 3.10 and another in 3.11 is less than
ideal, so this new thread is to collect comments about Enum and it's str() and repr() and whether the changes take
effect in 3.10, 3.11, or both.
TL;DR -- sorry, there isn't one.
History:
--------
As Enum and IntEnum have started proliferating in the stdlib, some of those enumerations have changed the str() and
repr() of their members -- for example, in 3.8 re.RegexFlag had its repr() and str() changed to be module.name
(re.ASCII) for both instead of <enum_class.name: value> (<RegexFlag.ASCII: 256>) for the repr(); that change made sense
because all the members are exported to re's global name space, and they are accessed as `re.ASCII` in existing code.
While a good change (in my opinion and with which I had nothing to do), we now have stdlib enumerations with differing
str() and repr()s, which is a minor ding against recognizability and usability.
Current 3.10 Changes
--------------------
In an effort to standardize the stdlib enums, a new decorator was added: global_enum(). It can be called manually (for
example in re.RegexFlag), or automatically by Enum._convert_ (for example in ssl.VerifyFlags).
That changed was visually appealing, and I had users wanting that type of output for Enum in general, so after asking
for feedback on python-dev and python-ideas (and receiving little) I changed the basic str() and repr() of Enum to:
- str(): NAME
- repr(): enum.NAME
While working on some other bugs/changes in Enum.__format__ and serialization in general, I have come to believe that
IntEnum (and IntFlag) should be as near-perfect a drop-in replacement as possible for when they are used to replace
existing integer constants (which I believe is their most important use-case).
Reverting 3.10 Changes
----------------------
I have been increasingly unhappy with the general `Enum.__repr__` change, so while the stdlib global repr() change is
staying, Enum, IntEnum, etc., is going back to <enum_class.NAME: value>.
Proposed 3.10 Change (for 3.10.0 beta 4)
----------------------------------------
In order to improve the drop-in replacement use-case for IntEnum and IntFlag, I would like to change the str() for those
two to be simply the value of the member, as if it was still a plain integer and not part of an enumeration. At that
point the only obvious difference would be the repr(), which I think is the most important and useful change as that is
what (should) show up in log files, the REPL, etc.
This would also make IntEnum and IntFlag consistent with StrEnum as, for other reasons, StrEnum member str()s are the
values of the members rather than the names of the members. Note also that this change would not affect user created
enumerations that mixed in int on their own.
The Question
------------
With all the changes currently happening to the str()s and repr()s of the various enums, what are the good reasons to
not make this final change now, and instead have one more change in 3.11?
--
~Ethan~
Summary of Enum str()s and repr()s by category
| | repr() | str() | format()
| stdlib global | re.ASCII | re.ASCII | 256
| Enum | <Color.RED: 1> | Color.RED | RED
| IntEnum | <Color.RED: 1> | 1 | 1
| int, Enum | <Color.RED: 1> | Color.RED | 1 (will be RED in 3.12 due to back-compatibility
| | | constraints which should affect very few people as
| | | as they are probably using IntEnum instead of mixing
| | | in int themselves -- comments about this likelihood also
| | | appreciated)
5
10
Greetings!
A question [1] has arisen about the viability of `random.SystemRandom` in Pythons before and after the secrets module
was introduced (3.5 I think) -- specifically
does it give independent and uniform discrete distribution for
cryptographic purposes across CPython 3.x versions?
Any help appreciated!
--
~Ethan~
[1] https://stackoverflow.com/q/68319071/208880
6
9
Wow! A release on a Saturday? Do the release management team even rest? You
better believe it, because this is the last of the planned beta releases.
This means that the next pre-release will be the first release candidate of
Python 3.10.0. Remember that our goal is to have no ABI changes after this
beta and a few code changes as possible after 3.10.0rc1.
https://www.python.org/downloads/release/python-3100b4/
#This is a beta preview of Python 3.10
Python 3.10 is still in development. 3.10.0b4 is the fourth and last of the
beta release previews. Beta release previews are intended to give the wider
community the opportunity to test new features and bug fixes and to prepare
their projects to support the new feature release.
We strongly encourage maintainers of third-party Python projects to test
with 3.10 during the beta phase and report issues found to the Python bug
tracker as soon as possible. While the release is planned to be feature
complete entering the beta phase, it is possible that features may be
modified or, in rare cases, deleted up until the start of the release
candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes
after beta 4 and as few code changes as possible after 3.10.0rc1, the first
release candidate. To achieve that, it will be extremely important to get
as much exposure for 3.10 as possible during the beta phase.
Please keep in mind that this is a preview release and its use is not
recommended for production environments.
The next pre-release, the first release candidate of Python 3.10.0, will be
3.10.0rc1. It is currently scheduled for Monday, 2021-08-02.
#And now for something completely different
In quantum physics, the spin is an intrinsic form of angular momentum
carried by elementary particles, composite particles, and atomic nuclei.
The spin is one of two types of angular momentum in quantum mechanics, the
other being orbital angular momentum. The orbital angular momentum operator
is the quantum-mechanical counterpart to the classical angular momentum of
orbital revolution and appears when there is periodic structure to its
wavefunction as the angle varies. For photons, spin is the
quantum-mechanical counterpart of the polarization of light; for electrons,
the spin has no classical counterpart.
# We hope you enjoy those new releases!
Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.
Regards from very cloudy London,
Your friendly release team,
Pablo Galindo @pablogsal
Ned Deily @nad
Steve Dower @steve.dower
1
0