Mailman 3 October 2020 - Python-checkins

bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953) (GH-23021)
by serhiy-storchaka Oct. 29, 2020

Oct. 29, 2020

https://github.com/python/cpython/commit/60324d26b58c89d68abb23fb42f1563d39… commit: 60324d26b58c89d68abb23fb42f1563d395c3910 branch: 3.9 author: Miss Skeleton (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2020-10-29T13:02:50+02:00 summary: bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953) (GH-23021) func_dealloc() does not handle partially-created objects. Best not to give it any. (cherry picked from commit 350526105fa9b131d8b941ae753378b741dabb2f) Co-authored-by: Yonatan Goldschmidt <yon.goldschmidt(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst M Objects/funcobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst new file mode 100644 index 0000000000000..2b16e69da73b5 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst @@ -0,0 +1,2 @@ +Fix handling of errors during creation of ``PyFunctionObject``, which resulted +in operations on uninitialized memory. Patch by Yonatan Goldschmidt. diff --git a/Objects/funcobject.c b/Objects/funcobject.c index bd24f67b9740a..2c60275d90619 100644 --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -20,9 +20,23 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname return NULL; } + /* __module__: If module name is in globals, use it. + Otherwise, use None. */ + module = PyDict_GetItemWithError(globals, __name__); + if (module) { + Py_INCREF(module); + } + else if (PyErr_Occurred()) { + return NULL; + } + op = PyObject_GC_New(PyFunctionObject, &PyFunction_Type); - if (op == NULL) + if (op == NULL) { + Py_XDECREF(module); return NULL; + } + /* Note: No failures from this point on, since func_dealloc() does not + expect a partially-created object. */ op->func_weakreflist = NULL; Py_INCREF(code); @@ -35,6 +49,7 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_kwdefaults = NULL; /* No keyword only defaults */ op->func_closure = NULL; op->vectorcall = _PyFunction_Vectorcall; + op->func_module = module; consts = ((PyCodeObject *)code)->co_consts; if (PyTuple_Size(consts) >= 1) { @@ -48,20 +63,8 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_doc = doc; op->func_dict = NULL; - op->func_module = NULL; op->func_annotations = NULL; - /* __module__: If module name is in globals, use it. - Otherwise, use None. */ - module = PyDict_GetItemWithError(globals, __name__); - if (module) { - Py_INCREF(module); - op->func_module = module; - } - else if (PyErr_Occurred()) { - Py_DECREF(op); - return NULL; - } if (qualname) op->func_qualname = qualname; else

1 0

bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953)
by miss-islington Oct. 29, 2020

Oct. 29, 2020

https://github.com/python/cpython/commit/9ede1b071bf5250ba5f2d04d7ba86a24c0… commit: 9ede1b071bf5250ba5f2d04d7ba86a24c06d41a1 branch: 3.8 author: Miss Skeleton (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-29T03:24:12-07:00 summary: bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953) func_dealloc() does not handle partially-created objects. Best not to give it any. (cherry picked from commit 350526105fa9b131d8b941ae753378b741dabb2f) Co-authored-by: Yonatan Goldschmidt <yon.goldschmidt(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst M Objects/funcobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst new file mode 100644 index 0000000000000..2b16e69da73b5 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst @@ -0,0 +1,2 @@ +Fix handling of errors during creation of ``PyFunctionObject``, which resulted +in operations on uninitialized memory. Patch by Yonatan Goldschmidt. diff --git a/Objects/funcobject.c b/Objects/funcobject.c index df5cc2d3f5702..0fc4e127081bf 100644 --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -22,9 +22,23 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname return NULL; } + /* __module__: If module name is in globals, use it. + Otherwise, use None. */ + module = PyDict_GetItemWithError(globals, __name__); + if (module) { + Py_INCREF(module); + } + else if (PyErr_Occurred()) { + return NULL; + } + op = PyObject_GC_New(PyFunctionObject, &PyFunction_Type); - if (op == NULL) + if (op == NULL) { + Py_XDECREF(module); return NULL; + } + /* Note: No failures from this point on, since func_dealloc() does not + expect a partially-created object. */ op->func_weakreflist = NULL; Py_INCREF(code); @@ -37,6 +51,7 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_kwdefaults = NULL; /* No keyword only defaults */ op->func_closure = NULL; op->vectorcall = _PyFunction_Vectorcall; + op->func_module = module; consts = ((PyCodeObject *)code)->co_consts; if (PyTuple_Size(consts) >= 1) { @@ -50,20 +65,8 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_doc = doc; op->func_dict = NULL; - op->func_module = NULL; op->func_annotations = NULL; - /* __module__: If module name is in globals, use it. - Otherwise, use None. */ - module = PyDict_GetItemWithError(globals, __name__); - if (module) { - Py_INCREF(module); - op->func_module = module; - } - else if (PyErr_Occurred()) { - Py_DECREF(op); - return NULL; - } if (qualname) op->func_qualname = qualname; else

1 0

bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953)
by serhiy-storchaka Oct. 29, 2020

Oct. 29, 2020

https://github.com/python/cpython/commit/350526105fa9b131d8b941ae753378b741… commit: 350526105fa9b131d8b941ae753378b741dabb2f branch: master author: Yonatan Goldschmidt <yon.goldschmidt(a)gmail.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2020-10-29T11:58:52+02:00 summary: bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953) func_dealloc() does not handle partially-created objects. Best not to give it any. files: A Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst M Objects/funcobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst new file mode 100644 index 0000000000000..2b16e69da73b5 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-21-34-05.bpo-42143.N6KXUO.rst @@ -0,0 +1,2 @@ +Fix handling of errors during creation of ``PyFunctionObject``, which resulted +in operations on uninitialized memory. Patch by Yonatan Goldschmidt. diff --git a/Objects/funcobject.c b/Objects/funcobject.c index 09a188664e861..9b4302a13c10f 100644 --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -19,9 +19,23 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname return NULL; } + /* __module__: If module name is in globals, use it. + Otherwise, use None. */ + module = PyDict_GetItemWithError(globals, __name__); + if (module) { + Py_INCREF(module); + } + else if (PyErr_Occurred()) { + return NULL; + } + op = PyObject_GC_New(PyFunctionObject, &PyFunction_Type); - if (op == NULL) + if (op == NULL) { + Py_XDECREF(module); return NULL; + } + /* Note: No failures from this point on, since func_dealloc() does not + expect a partially-created object. */ op->func_weakreflist = NULL; Py_INCREF(code); @@ -34,6 +48,7 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_kwdefaults = NULL; /* No keyword only defaults */ op->func_closure = NULL; op->vectorcall = _PyFunction_Vectorcall; + op->func_module = module; consts = ((PyCodeObject *)code)->co_consts; if (PyTuple_Size(consts) >= 1) { @@ -47,20 +62,8 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_doc = doc; op->func_dict = NULL; - op->func_module = NULL; op->func_annotations = NULL; - /* __module__: If module name is in globals, use it. - Otherwise, use None. */ - module = PyDict_GetItemWithError(globals, __name__); - if (module) { - Py_INCREF(module); - op->func_module = module; - } - else if (PyErr_Occurred()) { - Py_DECREF(op); - return NULL; - } if (qualname) op->func_qualname = qualname; else

1 0

bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
by miss-islington Oct. 29, 2020

Oct. 29, 2020

https://github.com/python/cpython/commit/df59273c7a384ea8c890fa8e9b80c92825… commit: df59273c7a384ea8c890fa8e9b80c92825df841c branch: master author: Zackery Spytz <zspytz(a)gmail.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-29T02:44:35-07:00 summary: bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639) Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a hardcoded 3. files: A Misc/NEWS.d/next/Library/2020-04-21-17-18-33.bpo-34204.9wXTtY.rst M Doc/library/shelve.rst M Doc/whatsnew/3.10.rst M Lib/shelve.py M Lib/test/test_shelve.py diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst index f08c58179a2f9..07caf91d5b7d9 100644 --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -25,8 +25,9 @@ lots of shared sub-objects. The keys are ordinary strings. database file is opened for reading and writing. The optional *flag* parameter has the same interpretation as the *flag* parameter of :func:`dbm.open`. - By default, version 3 pickles are used to serialize values. The version of the - pickle protocol can be specified with the *protocol* parameter. + By default, pickles created with :data:`pickle.DEFAULT_PROTOCOL` are used + to serialize values. The version of the pickle protocol can be specified + with the *protocol* parameter. Because of Python semantics, a shelf cannot know when a mutable persistent-dictionary entry is modified. By default modified objects are @@ -40,6 +41,10 @@ lots of shared sub-objects. The keys are ordinary strings. determine which accessed entries are mutable, nor which ones were actually mutated). + .. versionchanged:: 3.10 + :data:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle + protocol. + .. note:: Do not rely on the shelf being closed automatically; always call @@ -108,9 +113,10 @@ Restrictions A subclass of :class:`collections.abc.MutableMapping` which stores pickled values in the *dict* object. - By default, version 3 pickles are used to serialize values. The version of the - pickle protocol can be specified with the *protocol* parameter. See the - :mod:`pickle` documentation for a discussion of the pickle protocols. + By default, pickles created with :data:`pickle.DEFAULT_PROTOCOL` are used + to serialize values. The version of the pickle protocol can be specified + with the *protocol* parameter. See the :mod:`pickle` documentation for a + discussion of the pickle protocols. If the *writeback* parameter is ``True``, the object will hold a cache of all entries accessed and write them back to the *dict* at sync and close times. @@ -130,6 +136,10 @@ Restrictions .. versionchanged:: 3.4 Added context manager support. + .. versionchanged:: 3.10 + :data:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle + protocol. + .. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index b2c6d10ba8deb..45258db492571 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -210,6 +210,13 @@ py_compile Added ``--quiet`` option to command-line interface of :mod:`py_compile`. (Contributed by Gregory Schevchenko in :issue:`38731`.) +shelve +------ + +The :mod:`shelve` module now uses :data:`pickle.DEFAULT_PROTOCOL` by default +instead of :mod:`pickle` protocol ``3`` when creating shelves. +(Contributed by Zackery Spytz in :issue:`34204`.) + sys --- diff --git a/Lib/shelve.py b/Lib/shelve.py index 5d443a0fa8d4f..e053c397345a0 100644 --- a/Lib/shelve.py +++ b/Lib/shelve.py @@ -56,7 +56,7 @@ the persistent dictionary on disk, if feasible). """ -from pickle import Pickler, Unpickler +from pickle import DEFAULT_PROTOCOL, Pickler, Unpickler from io import BytesIO import collections.abc @@ -85,7 +85,7 @@ def __init__(self, dict, protocol=None, writeback=False, keyencoding="utf-8"): self.dict = dict if protocol is None: - protocol = 3 + protocol = DEFAULT_PROTOCOL self._protocol = protocol self.writeback = writeback self.cache = {} diff --git a/Lib/test/test_shelve.py b/Lib/test/test_shelve.py index ac25eee2e52fd..cfdd67c26c5f5 100644 --- a/Lib/test/test_shelve.py +++ b/Lib/test/test_shelve.py @@ -1,6 +1,8 @@ import unittest import shelve import glob +import pickle + from test import support from test.support import os_helper from collections.abc import MutableMapping @@ -160,7 +162,7 @@ def test_with(self): def test_default_protocol(self): with shelve.Shelf({}) as s: - self.assertEqual(s._protocol, 3) + self.assertEqual(s._protocol, pickle.DEFAULT_PROTOCOL) from test import mapping_tests diff --git a/Misc/NEWS.d/next/Library/2020-04-21-17-18-33.bpo-34204.9wXTtY.rst b/Misc/NEWS.d/next/Library/2020-04-21-17-18-33.bpo-34204.9wXTtY.rst new file mode 100644 index 0000000000000..bce6d39148a37 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-04-21-17-18-33.bpo-34204.9wXTtY.rst @@ -0,0 +1,2 @@ +The :mod:`shelve` module now uses :data:`pickle.DEFAULT_PROTOCOL` by default +instead of :mod:`pickle` protocol ``3``.

1 0

[3.9] bpo-41805: Documentation for PEP 585 (GH-22615) (GH-23016)
by miss-islington Oct. 28, 2020

Oct. 28, 2020

https://github.com/python/cpython/commit/577d7c4e628260eb7926d043ca9c355ece… commit: 577d7c4e628260eb7926d043ca9c355ece583eb7 branch: 3.9 author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-28T08:33:36-07:00 summary: [3.9] bpo-41805: Documentation for PEP 585 (GH-22615) (GH-23016) Backport of #22615 to 3.9 since that couldn't be auto-merged due to conflicts. files: A Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst M Doc/glossary.rst M Doc/library/stdtypes.rst M Doc/library/types.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 847500e556056..4fd01e0160c26 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -483,6 +483,13 @@ Glossary See also the :term:`single dispatch` glossary entry, the :func:`functools.singledispatch` decorator, and :pep:`443`. + generic type + A :term:`type` that can be parameterized; typically a container like + :class:`list`. Used for :term:`type hints <type hint>` and + :term:`annotations <annotation>`. + + See :pep:`483` for more details, and :mod:`typing` or + :ref:`generic alias type <types-genericalias>` for its uses. GIL See :term:`global interpreter lock`. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 1de48e13e271f..e19e76f8311c8 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4731,6 +4731,199 @@ Compared to the overhead of setting up the runtime context, the overhead of a single class dictionary lookup is negligible. +.. _types-genericalias: + +Generic Alias Type +================== + +.. index:: + object: GenericAlias + pair: Generic; Alias + +``GenericAlias`` objects are created by subscripting a class (usually a +container), such as ``list[int]``. They are intended primarily for +:term:`type annotations <annotation>`. + +Usually, the :ref:`subscription <subscriptions>` of container objects calls the +method :meth:`__getitem__` of the object. However, the subscription of some +containers' classes may call the classmethod :meth:`__class_getitem__` of the +class instead. The classmethod :meth:`__class_getitem__` should return a +``GenericAlias`` object. + +.. note:: + If the :meth:`__getitem__` of the class' metaclass is present, it will take + precedence over the :meth:`__class_getitem__` defined in the class (see + :pep:`560` for more details). + +The ``GenericAlias`` object acts as a proxy for :term:`generic types +<generic type>`, implementing *parameterized generics* - a specific instance +of a generic which provides the types for container elements. + +The user-exposed type for the ``GenericAlias`` object can be accessed from +:data:`types.GenericAlias` and used for :func:`isinstance` checks. + +.. describe:: T[X, Y, ...] + + Creates a ``GenericAlias`` representing a type ``T`` containing elements + of types *X*, *Y*, and more depending on the ``T`` used. + For example, a function expecting a :class:`list` containing + :class:`float` elements:: + + def average(values: list[float]) -> float: + return sum(values) / len(values) + + Another example for :term:`mapping` objects, using a :class:`dict`, which + is a generic type expecting two type parameters representing the key type + and the value type. In this example, the function expects a ``dict`` with + keys of type :class:`str` and values of type :class:`int`:: + + def send_post_request(url: str, body: dict[str, int]) -> None: + ... + +The builtin functions :func:`isinstance` and :func:`issubclass` do not accept +``GenericAlias`` types for their second argument:: + + >>> isinstance([1, 2], list[str]) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: isinstance() argument 2 cannot be a parameterized generic + +The Python runtime does not enforce :term:`type annotations <annotation>`. +This extends to generic types and their type parameters. When creating +an object from a ``GenericAlias``, container elements are not checked +against their type. For example, the following code is discouraged, but will +run without errors:: + + >>> t = list[str] + >>> t([1, 2, 3]) + [1, 2, 3] + +Furthermore, parameterized generics erase type parameters during object +creation:: + + >>> t = list[str] + >>> type(t) + <class 'types.GenericAlias'> + + >>> l = t() + >>> type(l) + <class 'list'> + +Calling :func:`repr` or :func:`str` on a generic shows the parameterized type:: + + >>> repr(list[int]) + 'list[int]' + + >>> str(list[int]) + 'list[int]' + +The :meth:`__getitem__` method of generics will raise an exception to disallow +mistakes like ``dict[str][str]``:: + + >>> dict[str][str] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: There are no type variables left in dict[str] + +However, such expressions are valid when :ref:`type variables <generics>` are +used. The index must have as many elements as there are type variable items +in the ``GenericAlias`` object's :attr:`__args__ <genericalias.__args__>`. :: + + >>> from typing import TypeVar + >>> Y = TypeVar('Y') + >>> dict[str, Y][int] + dict[str, int] + + +Standard Generic Collections +---------------------------- + +These standard library collections support parameterized generics. + +* :class:`tuple` +* :class:`list` +* :class:`dict` +* :class:`set` +* :class:`frozenset` +* :class:`type` +* :class:`collections.deque` +* :class:`collections.defaultdict` +* :class:`collections.OrderedDict` +* :class:`collections.Counter` +* :class:`collections.ChainMap` +* :class:`collections.abc.Awaitable` +* :class:`collections.abc.Coroutine` +* :class:`collections.abc.AsyncIterable` +* :class:`collections.abc.AsyncIterator` +* :class:`collections.abc.AsyncGenerator` +* :class:`collections.abc.Iterable` +* :class:`collections.abc.Iterator` +* :class:`collections.abc.Generator` +* :class:`collections.abc.Reversible` +* :class:`collections.abc.Container` +* :class:`collections.abc.Collection` +* :class:`collections.abc.Callable` +* :class:`collections.abc.Set` +* :class:`collections.abc.MutableSet` +* :class:`collections.abc.Mapping` +* :class:`collections.abc.MutableMapping` +* :class:`collections.abc.Sequence` +* :class:`collections.abc.MutableSequence` +* :class:`collections.abc.ByteString` +* :class:`collections.abc.MappingView` +* :class:`collections.abc.KeysView` +* :class:`collections.abc.ItemsView` +* :class:`collections.abc.ValuesView` +* :class:`contextlib.AbstractContextManager` +* :class:`contextlib.AbstractAsyncContextManager` +* :ref:`re.Pattern <re-objects>` +* :ref:`re.Match <match-objects>` + + +Special Attributes of Generic Alias +----------------------------------- + +All parameterized generics implement special read-only attributes. + +.. attribute:: genericalias.__origin__ + + This attribute points at the non-parameterized generic class:: + + >>> list[int].__origin__ + <class 'list'> + + +.. attribute:: genericalias.__args__ + + This attribute is a :class:`tuple` (possibly of length 1) of generic + types passed to the original :meth:`__class_getitem__` + of the generic container:: + + >>> dict[str, list[int]].__args__ + (<class 'str'>, list[int]) + + +.. attribute:: genericalias.__parameters__ + + This attribute is a lazily computed tuple (possibly empty) of unique type + variables found in ``__args__``:: + + >>> from typing import TypeVar + + >>> T = TypeVar('T') + >>> list[T].__parameters__ + (~T,) + + +.. seealso:: + + * :pep:`585` -- "Type Hinting Generics In Standard Collections" + * :meth:`__class_getitem__` -- Used to implement parameterized generics. + * :ref:`generics` -- Generics in the :mod:`typing` module. + +.. versionadded:: 3.9 + + .. _typesother: Other Built-in Types diff --git a/Doc/library/types.rst b/Doc/library/types.rst index a88e882305ff3..1573e460f2f2b 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -242,6 +242,14 @@ Standard names are defined for the following types: Defaults to ``None``. Previously the attribute was optional. +.. data:: GenericAlias + + The type of :ref:`parameterized generics <types-genericalias>` such as + ``list[int]``. + + .. versionadded:: 3.9 + + .. class:: TracebackType(tb_next, tb_frame, tb_lasti, tb_lineno) The type of traceback objects such as found in ``sys.exc_info()[2]``. diff --git a/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst b/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst new file mode 100644 index 0000000000000..9c9134350a317 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst @@ -0,0 +1,3 @@ +Documented :ref:`generic alias type <types-genericalias>` and +:data:`types.GenericAlias`. Also added an entry in glossary for +:term:`generic types <generic type>`.

1 0

[3.9] bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111) (GH-23011)
by lysnikolaou Oct. 28, 2020

Oct. 28, 2020

https://github.com/python/cpython/commit/24a7c298d47658295673dc04d1b6d59f2b… commit: 24a7c298d47658295673dc04d1b6d59f2b200a7c branch: 3.9 author: Lysandros Nikolaou <lisandrosnik(a)gmail.com> committer: lysnikolaou <lisandrosnik(a)gmail.com> date: 2020-10-28T02:14:15+02:00 summary: [3.9] bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111) (GH-23011) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced. (cherry picked from commit bca701403253379409dece03053dbd739c0bd059) files: A Misc/NEWS.d/next/Core and Builtins/2020-10-23-02-43-24.bpo-42123.64gJWC.rst M Grammar/python.gram M Parser/pegen/parse.c M Parser/pegen/pegen.c M Parser/pegen/pegen.h M Tools/peg_generator/pegen/c_generator.py diff --git a/Grammar/python.gram b/Grammar/python.gram index 6c36b6acb5352..b709d3d5c3214 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -535,7 +535,7 @@ yield_expr[expr_ty]: arguments[expr_ty] (memo): | a=args [','] &')' { a } - | incorrect_arguments + | invalid_arguments args[expr_ty]: | a=','.(starred_expression | named_expression !'=')+ b=[',' k=kwargs {k}] { _PyPegen_collect_call_seqs(p, a, b, EXTRA) } | a=kwargs { _Py_Call(_PyPegen_dummy_name(p), @@ -620,7 +620,7 @@ t_atom[expr_ty]: # From here on, there are rules for invalid syntax with specialised error messages -incorrect_arguments: +invalid_arguments: | args ',' '*' { RAISE_SYNTAX_ERROR("iterable argument unpacking follows keyword argument unpacking") } | a=expression for_if_clauses ',' [args | expression for_if_clauses] { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "Generator expression must be parenthesized") } diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-23-02-43-24.bpo-42123.64gJWC.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-23-02-43-24.bpo-42123.64gJWC.rst new file mode 100644 index 0000000000000..6461efd76f0f9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-23-02-43-24.bpo-42123.64gJWC.rst @@ -0,0 +1,3 @@ +Run the parser two times. On the first run, disable all the rules that only +generate better error messages to gain performance. If there's a parse +failure, run the parser a second time with those enabled. diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c index 2eb9e03d52819..48a2443474684 100644 --- a/Parser/pegen/parse.c +++ b/Parser/pegen/parse.c @@ -218,7 +218,7 @@ static KeywordToken *reserved_keywords[] = { #define t_primary_type 1140 // Left-recursive #define t_lookahead_type 1141 #define t_atom_type 1142 -#define incorrect_arguments_type 1143 +#define invalid_arguments_type 1143 #define invalid_kwarg_type 1144 #define invalid_named_expression_type 1145 #define invalid_assignment_type 1146 @@ -537,7 +537,7 @@ static expr_ty target_rule(Parser *p); static expr_ty t_primary_rule(Parser *p); static void *t_lookahead_rule(Parser *p); static expr_ty t_atom_rule(Parser *p); -static void *incorrect_arguments_rule(Parser *p); +static void *invalid_arguments_rule(Parser *p); static void *invalid_kwarg_rule(Parser *p); static void *invalid_named_expression_rule(Parser *p); static void *invalid_assignment_rule(Parser *p); @@ -2219,7 +2219,7 @@ assignment_rule(Parser *p) return NULL; } } - { // invalid_assignment + if (p->call_invalid_rules) { // invalid_assignment if (p->error_indicator) { D(p->level--); return NULL; @@ -2892,7 +2892,7 @@ del_stmt_rule(Parser *p) D(fprintf(stderr, "%*c%s del_stmt[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'del' del_targets &(';' | NEWLINE)")); } - { // invalid_del_stmt + if (p->call_invalid_rules) { // invalid_del_stmt if (p->error_indicator) { D(p->level--); return NULL; @@ -3243,7 +3243,7 @@ import_from_targets_rule(Parser *p) D(fprintf(stderr, "%*c%s import_from_targets[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'*'")); } - { // invalid_import_from_targets + if (p->call_invalid_rules) { // invalid_import_from_targets if (p->error_indicator) { D(p->level--); return NULL; @@ -4036,7 +4036,7 @@ for_stmt_rule(Parser *p) return NULL; } } - { // invalid_for_target + if (p->call_invalid_rules) { // invalid_for_target if (p->error_indicator) { D(p->level--); return NULL; @@ -4337,7 +4337,7 @@ with_item_rule(Parser *p) D(fprintf(stderr, "%*c%s with_item[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "expression 'as' star_target &(',' | ')' | ':')")); } - { // invalid_with_item + if (p->call_invalid_rules) { // invalid_with_item if (p->error_indicator) { D(p->level--); return NULL; @@ -5072,7 +5072,7 @@ func_type_comment_rule(Parser *p) D(fprintf(stderr, "%*c%s func_type_comment[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "NEWLINE TYPE_COMMENT &(NEWLINE INDENT)")); } - { // invalid_double_type_comments + if (p->call_invalid_rules) { // invalid_double_type_comments if (p->error_indicator) { D(p->level--); return NULL; @@ -5127,7 +5127,7 @@ params_rule(Parser *p) } arguments_ty _res = NULL; int _mark = p->mark; - { // invalid_parameters + if (p->call_invalid_rules) { // invalid_parameters if (p->error_indicator) { D(p->level--); return NULL; @@ -5602,7 +5602,7 @@ star_etc_rule(Parser *p) D(fprintf(stderr, "%*c%s star_etc[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "kwds")); } - { // invalid_star_etc + if (p->call_invalid_rules) { // invalid_star_etc if (p->error_indicator) { D(p->level--); return NULL; @@ -6305,7 +6305,7 @@ block_rule(Parser *p) D(fprintf(stderr, "%*c%s block[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "simple_stmt")); } - { // invalid_block + if (p->call_invalid_rules) { // invalid_block if (p->error_indicator) { D(p->level--); return NULL; @@ -6799,7 +6799,7 @@ named_expression_rule(Parser *p) D(fprintf(stderr, "%*c%s named_expression[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "expression !':='")); } - { // invalid_named_expression + if (p->call_invalid_rules) { // invalid_named_expression if (p->error_indicator) { D(p->level--); return NULL; @@ -7193,7 +7193,7 @@ lambda_params_rule(Parser *p) } arguments_ty _res = NULL; int _mark = p->mark; - { // invalid_lambda_parameters + if (p->call_invalid_rules) { // invalid_lambda_parameters if (p->error_indicator) { D(p->level--); return NULL; @@ -7670,7 +7670,7 @@ lambda_star_etc_rule(Parser *p) D(fprintf(stderr, "%*c%s lambda_star_etc[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "lambda_kwds")); } - { // invalid_lambda_star_etc + if (p->call_invalid_rules) { // invalid_lambda_star_etc if (p->error_indicator) { D(p->level--); return NULL; @@ -10330,7 +10330,7 @@ primary_raw(Parser *p) UNUSED(_start_lineno); // Only used by EXTRA macro int _start_col_offset = p->tokens[_mark]->col_offset; UNUSED(_start_col_offset); // Only used by EXTRA macro - { // invalid_primary + if (p->call_invalid_rules) { // invalid_primary if (p->error_indicator) { D(p->level--); return NULL; @@ -11209,7 +11209,7 @@ listcomp_rule(Parser *p) return NULL; } } - { // invalid_comprehension + if (p->call_invalid_rules) { // invalid_comprehension if (p->error_indicator) { D(p->level--); return NULL; @@ -11340,7 +11340,7 @@ group_rule(Parser *p) D(fprintf(stderr, "%*c%s group[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'(' (yield_expr | named_expression) ')'")); } - { // invalid_group + if (p->call_invalid_rules) { // invalid_group if (p->error_indicator) { D(p->level--); return NULL; @@ -11434,7 +11434,7 @@ genexp_rule(Parser *p) return NULL; } } - { // invalid_comprehension + if (p->call_invalid_rules) { // invalid_comprehension if (p->error_indicator) { D(p->level--); return NULL; @@ -11593,7 +11593,7 @@ setcomp_rule(Parser *p) return NULL; } } - { // invalid_comprehension + if (p->call_invalid_rules) { // invalid_comprehension if (p->error_indicator) { D(p->level--); return NULL; @@ -11745,7 +11745,7 @@ dictcomp_rule(Parser *p) D(fprintf(stderr, "%*c%s dictcomp[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'{' kvpair for_if_clauses '}'")); } - { // invalid_dict_comprehension + if (p->call_invalid_rules) { // invalid_dict_comprehension if (p->error_indicator) { D(p->level--); return NULL; @@ -12064,7 +12064,7 @@ for_if_clause_rule(Parser *p) return NULL; } } - { // invalid_for_target + if (p->call_invalid_rules) { // invalid_for_target if (p->error_indicator) { D(p->level--); return NULL; @@ -12190,7 +12190,7 @@ yield_expr_rule(Parser *p) return _res; } -// arguments: args ','? &')' | incorrect_arguments +// arguments: args ','? &')' | invalid_arguments static expr_ty arguments_rule(Parser *p) { @@ -12235,24 +12235,24 @@ arguments_rule(Parser *p) D(fprintf(stderr, "%*c%s arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "args ','? &')'")); } - { // incorrect_arguments + if (p->call_invalid_rules) { // invalid_arguments if (p->error_indicator) { D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "incorrect_arguments")); - void *incorrect_arguments_var; + D(fprintf(stderr, "%*c> arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "invalid_arguments")); + void *invalid_arguments_var; if ( - (incorrect_arguments_var = incorrect_arguments_rule(p)) // incorrect_arguments + (invalid_arguments_var = invalid_arguments_rule(p)) // invalid_arguments ) { - D(fprintf(stderr, "%*c+ arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "incorrect_arguments")); - _res = incorrect_arguments_var; + D(fprintf(stderr, "%*c+ arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "invalid_arguments")); + _res = invalid_arguments_var; goto done; } p->mark = _mark; D(fprintf(stderr, "%*c%s arguments[%d-%d]: %s failed!\n", p->level, ' ', - p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "incorrect_arguments")); + p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "invalid_arguments")); } _res = NULL; done: @@ -12589,7 +12589,7 @@ kwarg_or_starred_rule(Parser *p) D(fprintf(stderr, "%*c%s kwarg_or_starred[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "starred_expression")); } - { // invalid_kwarg + if (p->call_invalid_rules) { // invalid_kwarg if (p->error_indicator) { D(p->level--); return NULL; @@ -12709,7 +12709,7 @@ kwarg_or_double_starred_rule(Parser *p) D(fprintf(stderr, "%*c%s kwarg_or_double_starred[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'**' expression")); } - { // invalid_kwarg + if (p->call_invalid_rules) { // invalid_kwarg if (p->error_indicator) { D(p->level--); return NULL; @@ -14421,14 +14421,14 @@ t_atom_rule(Parser *p) return _res; } -// incorrect_arguments: +// invalid_arguments: // | args ',' '*' // | expression for_if_clauses ',' [args | expression for_if_clauses] // | args for_if_clauses // | args ',' expression for_if_clauses // | args ',' args static void * -incorrect_arguments_rule(Parser *p) +invalid_arguments_rule(Parser *p) { D(p->level++); if (p->error_indicator) { @@ -14442,7 +14442,7 @@ incorrect_arguments_rule(Parser *p) D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> incorrect_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' '*'")); + D(fprintf(stderr, "%*c> invalid_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' '*'")); Token * _literal; Token * _literal_1; expr_ty args_var; @@ -14454,7 +14454,7 @@ incorrect_arguments_rule(Parser *p) (_literal_1 = _PyPegen_expect_token(p, 16)) // token='*' ) { - D(fprintf(stderr, "%*c+ incorrect_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' '*'")); + D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' '*'")); _res = RAISE_SYNTAX_ERROR ( "iterable argument unpacking follows keyword argument unpacking" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; @@ -14464,7 +14464,7 @@ incorrect_arguments_rule(Parser *p) goto done; } p->mark = _mark; - D(fprintf(stderr, "%*c%s incorrect_arguments[%d-%d]: %s failed!\n", p->level, ' ', + D(fprintf(stderr, "%*c%s invalid_arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "args ',' '*'")); } { // expression for_if_clauses ',' [args | expression for_if_clauses] @@ -14472,7 +14472,7 @@ incorrect_arguments_rule(Parser *p) D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> incorrect_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); + D(fprintf(stderr, "%*c> invalid_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); Token * _literal; void *_opt_var; UNUSED(_opt_var); // Silence compiler warnings @@ -14488,7 +14488,7 @@ incorrect_arguments_rule(Parser *p) (_opt_var = _tmp_127_rule(p), 1) // [args | expression for_if_clauses] ) { - D(fprintf(stderr, "%*c+ incorrect_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); + D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); _res = RAISE_SYNTAX_ERROR_KNOWN_LOCATION ( a , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; @@ -14498,7 +14498,7 @@ incorrect_arguments_rule(Parser *p) goto done; } p->mark = _mark; - D(fprintf(stderr, "%*c%s incorrect_arguments[%d-%d]: %s failed!\n", p->level, ' ', + D(fprintf(stderr, "%*c%s invalid_arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); } { // args for_if_clauses @@ -14506,7 +14506,7 @@ incorrect_arguments_rule(Parser *p) D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> incorrect_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args for_if_clauses")); + D(fprintf(stderr, "%*c> invalid_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args for_if_clauses")); expr_ty a; asdl_seq* for_if_clauses_var; if ( @@ -14515,7 +14515,7 @@ incorrect_arguments_rule(Parser *p) (for_if_clauses_var = for_if_clauses_rule(p)) // for_if_clauses ) { - D(fprintf(stderr, "%*c+ incorrect_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args for_if_clauses")); + D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args for_if_clauses")); _res = _PyPegen_nonparen_genexp_in_call ( p , a ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; @@ -14525,7 +14525,7 @@ incorrect_arguments_rule(Parser *p) goto done; } p->mark = _mark; - D(fprintf(stderr, "%*c%s incorrect_arguments[%d-%d]: %s failed!\n", p->level, ' ', + D(fprintf(stderr, "%*c%s invalid_arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "args for_if_clauses")); } { // args ',' expression for_if_clauses @@ -14533,7 +14533,7 @@ incorrect_arguments_rule(Parser *p) D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> incorrect_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); + D(fprintf(stderr, "%*c> invalid_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); Token * _literal; expr_ty a; expr_ty args_var; @@ -14548,7 +14548,7 @@ incorrect_arguments_rule(Parser *p) (for_if_clauses_var = for_if_clauses_rule(p)) // for_if_clauses ) { - D(fprintf(stderr, "%*c+ incorrect_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); + D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); _res = RAISE_SYNTAX_ERROR_KNOWN_LOCATION ( a , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; @@ -14558,7 +14558,7 @@ incorrect_arguments_rule(Parser *p) goto done; } p->mark = _mark; - D(fprintf(stderr, "%*c%s incorrect_arguments[%d-%d]: %s failed!\n", p->level, ' ', + D(fprintf(stderr, "%*c%s invalid_arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "args ',' expression for_if_clauses")); } { // args ',' args @@ -14566,7 +14566,7 @@ incorrect_arguments_rule(Parser *p) D(p->level--); return NULL; } - D(fprintf(stderr, "%*c> incorrect_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' args")); + D(fprintf(stderr, "%*c> invalid_arguments[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "args ',' args")); Token * _literal; expr_ty a; expr_ty args_var; @@ -14578,7 +14578,7 @@ incorrect_arguments_rule(Parser *p) (args_var = args_rule(p)) // args ) { - D(fprintf(stderr, "%*c+ incorrect_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' args")); + D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' args")); _res = _PyPegen_arguments_parsing_error ( p , a ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; @@ -14588,7 +14588,7 @@ incorrect_arguments_rule(Parser *p) goto done; } p->mark = _mark; - D(fprintf(stderr, "%*c%s incorrect_arguments[%d-%d]: %s failed!\n", p->level, ' ', + D(fprintf(stderr, "%*c%s invalid_arguments[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "args ',' args")); } _res = NULL; diff --git a/Parser/pegen/pegen.c b/Parser/pegen/pegen.c index a7643fc3a6ad9..78891af825007 100644 --- a/Parser/pegen/pegen.c +++ b/Parser/pegen/pegen.c @@ -1101,15 +1101,28 @@ _PyPegen_Parser_New(struct tok_state *tok, int start_rule, int flags, p->feature_version = feature_version; p->known_err_token = NULL; p->level = 0; + p->call_invalid_rules = 0; return p; } +static void +reset_parser_state(Parser *p) +{ + for (int i = 0; i < p->fill; i++) { + p->tokens[i]->memo = NULL; + } + p->mark = 0; + p->call_invalid_rules = 1; +} + void * _PyPegen_run_parser(Parser *p) { void *res = _PyPegen_parse(p); if (res == NULL) { + reset_parser_state(p); + _PyPegen_parse(p); if (PyErr_Occurred()) { return NULL; } diff --git a/Parser/pegen/pegen.h b/Parser/pegen/pegen.h index bd35d4ff71ae8..2fea84fd22ab4 100644 --- a/Parser/pegen/pegen.h +++ b/Parser/pegen/pegen.h @@ -73,6 +73,7 @@ typedef struct { growable_comment_array type_ignore_comments; Token *known_err_token; int level; + int call_invalid_rules; } Parser; typedef struct { diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index aee668c3f329a..d0abc12b4026a 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -730,7 +730,10 @@ def handle_alt_loop(self, node: Alt, is_gather: bool, rulename: Optional[str]) - def visit_Alt( self, node: Alt, is_loop: bool, is_gather: bool, rulename: Optional[str] ) -> None: - self.print(f"{{ // {node}") + if len(node.items) == 1 and str(node.items[0]).startswith('invalid_'): + self.print(f"if (p->call_invalid_rules) {{ // {node}") + else: + self.print(f"{{ // {node}") with self.indent(): self._check_for_errors() node_str = str(node).replace('"', '\\"')

1 0

[3.9] bpo-41659: Disallow curly brace directly after primary (GH-22996) (#23006)
by lysnikolaou Oct. 27, 2020

Oct. 27, 2020

https://github.com/python/cpython/commit/c4b58cea4771afc0ddfdb857b0fb5115b9… commit: c4b58cea4771afc0ddfdb857b0fb5115b9f4bc9f branch: 3.9 author: Lysandros Nikolaou <lisandrosnik(a)gmail.com> committer: lysnikolaou <lisandrosnik(a)gmail.com> date: 2020-10-28T00:38:42+02:00 summary: [3.9] bpo-41659: Disallow curly brace directly after primary (GH-22996) (#23006) (cherry picked from commit 15acc4eaba8519d7d5f2acaffde65446b44dcf79) files: A Misc/NEWS.d/next/Core and Builtins/2020-10-27-18-32-49.bpo-41659.d4a-8o.rst M Grammar/python.gram M Lib/test/test_exceptions.py M Lib/test/test_syntax.py M Parser/pegen/parse.c diff --git a/Grammar/python.gram b/Grammar/python.gram index 8f4d250dc3ae4..6c36b6acb5352 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -460,6 +460,7 @@ await_primary[expr_ty] (memo): | AWAIT a=primary { CHECK_VERSION(5, "Await expressions are", _Py_Await(a, EXTRA)) } | primary primary[expr_ty]: + | invalid_primary # must be before 'primay genexp' because of invalid_genexp | a=primary '.' b=NAME { _Py_Attribute(a, b->v.Name.id, Load, EXTRA) } | a=primary b=genexp { _Py_Call(a, CHECK(_PyPegen_singleton_seq(p, b)), NULL, EXTRA) } | a=primary '(' b=[arguments] ')' { @@ -664,6 +665,8 @@ invalid_del_stmt: RAISE_SYNTAX_ERROR_INVALID_TARGET(DEL_TARGETS, a) } invalid_block: | NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block") } +invalid_primary: + | primary a='{' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "invalid syntax") } invalid_comprehension: | ('[' | '(' | '{') a=starred_expression for_if_clauses { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable unpacking cannot be used in comprehension") } diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 47f37114f0259..8d125b57ad6d5 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -205,6 +205,7 @@ def testSyntaxErrorOffset(self): check(b'Python = "\xcf\xb3\xf2\xee\xed" +', 1, 18) check('x = "a', 1, 7) check('lambda x: x = 2', 1, 1) + check('f{a + b + c}', 1, 2) # Errors thrown by compile.c check('class foo:return 1', 1, 11) diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index d6583b9d30c27..b0527e6cd85ba 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -802,6 +802,9 @@ def _check_error(self, code, errtext, else: self.fail("compile() did not raise SyntaxError") + def test_curly_brace_after_primary_raises_immediately(self): + self._check_error("f{", "invalid syntax", mode="single") + def test_assign_call(self): self._check_error("f() = 1", "assign") diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-27-18-32-49.bpo-41659.d4a-8o.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-18-32-49.bpo-41659.d4a-8o.rst new file mode 100644 index 0000000000000..038749a7b16c9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-27-18-32-49.bpo-41659.d4a-8o.rst @@ -0,0 +1,3 @@ +Fix a bug in the parser, where a curly brace following a `primary` didn't fail immediately. +This led to invalid expressions like `a {b}` to throw a :exc:`SyntaxError` with a wrong offset, +or invalid expressions ending with a curly brace like `a {` to not fail immediately in the REPL. \ No newline at end of file diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c index 1a4df757eae33..2eb9e03d52819 100644 --- a/Parser/pegen/parse.c +++ b/Parser/pegen/parse.c @@ -225,173 +225,174 @@ static KeywordToken *reserved_keywords[] = { #define invalid_ann_assign_target_type 1147 #define invalid_del_stmt_type 1148 #define invalid_block_type 1149 -#define invalid_comprehension_type 1150 -#define invalid_dict_comprehension_type 1151 -#define invalid_parameters_type 1152 -#define invalid_lambda_parameters_type 1153 -#define invalid_star_etc_type 1154 -#define invalid_lambda_star_etc_type 1155 -#define invalid_double_type_comments_type 1156 -#define invalid_with_item_type 1157 -#define invalid_for_target_type 1158 -#define invalid_group_type 1159 -#define invalid_import_from_targets_type 1160 -#define _loop0_1_type 1161 -#define _loop0_2_type 1162 -#define _loop0_4_type 1163 -#define _gather_3_type 1164 -#define _loop0_6_type 1165 -#define _gather_5_type 1166 -#define _loop0_8_type 1167 -#define _gather_7_type 1168 -#define _loop0_10_type 1169 -#define _gather_9_type 1170 -#define _loop1_11_type 1171 -#define _loop0_13_type 1172 -#define _gather_12_type 1173 -#define _tmp_14_type 1174 -#define _tmp_15_type 1175 -#define _tmp_16_type 1176 -#define _tmp_17_type 1177 -#define _tmp_18_type 1178 -#define _tmp_19_type 1179 -#define _tmp_20_type 1180 -#define _tmp_21_type 1181 -#define _loop1_22_type 1182 -#define _tmp_23_type 1183 -#define _tmp_24_type 1184 -#define _loop0_26_type 1185 -#define _gather_25_type 1186 -#define _loop0_28_type 1187 -#define _gather_27_type 1188 -#define _tmp_29_type 1189 -#define _tmp_30_type 1190 -#define _loop0_31_type 1191 -#define _loop1_32_type 1192 -#define _loop0_34_type 1193 -#define _gather_33_type 1194 -#define _tmp_35_type 1195 -#define _loop0_37_type 1196 -#define _gather_36_type 1197 -#define _tmp_38_type 1198 -#define _loop0_40_type 1199 -#define _gather_39_type 1200 -#define _loop0_42_type 1201 -#define _gather_41_type 1202 -#define _loop0_44_type 1203 -#define _gather_43_type 1204 -#define _loop0_46_type 1205 -#define _gather_45_type 1206 -#define _tmp_47_type 1207 -#define _loop1_48_type 1208 -#define _tmp_49_type 1209 -#define _tmp_50_type 1210 -#define _tmp_51_type 1211 -#define _tmp_52_type 1212 -#define _tmp_53_type 1213 -#define _loop0_54_type 1214 -#define _loop0_55_type 1215 -#define _loop0_56_type 1216 -#define _loop1_57_type 1217 -#define _loop0_58_type 1218 -#define _loop1_59_type 1219 -#define _loop1_60_type 1220 -#define _loop1_61_type 1221 -#define _loop0_62_type 1222 -#define _loop1_63_type 1223 -#define _loop0_64_type 1224 -#define _loop1_65_type 1225 -#define _loop0_66_type 1226 -#define _loop1_67_type 1227 -#define _loop1_68_type 1228 -#define _tmp_69_type 1229 -#define _loop0_71_type 1230 -#define _gather_70_type 1231 -#define _loop1_72_type 1232 -#define _loop0_74_type 1233 -#define _gather_73_type 1234 -#define _loop1_75_type 1235 -#define _loop0_76_type 1236 -#define _loop0_77_type 1237 -#define _loop0_78_type 1238 -#define _loop1_79_type 1239 -#define _loop0_80_type 1240 -#define _loop1_81_type 1241 -#define _loop1_82_type 1242 -#define _loop1_83_type 1243 -#define _loop0_84_type 1244 -#define _loop1_85_type 1245 -#define _loop0_86_type 1246 -#define _loop1_87_type 1247 -#define _loop0_88_type 1248 -#define _loop1_89_type 1249 -#define _loop1_90_type 1250 -#define _loop1_91_type 1251 -#define _loop1_92_type 1252 -#define _tmp_93_type 1253 -#define _loop0_95_type 1254 -#define _gather_94_type 1255 -#define _tmp_96_type 1256 -#define _tmp_97_type 1257 -#define _tmp_98_type 1258 -#define _tmp_99_type 1259 -#define _loop1_100_type 1260 -#define _tmp_101_type 1261 -#define _tmp_102_type 1262 -#define _loop0_104_type 1263 -#define _gather_103_type 1264 -#define _loop1_105_type 1265 -#define _loop0_106_type 1266 -#define _loop0_107_type 1267 -#define _loop0_109_type 1268 -#define _gather_108_type 1269 -#define _tmp_110_type 1270 -#define _loop0_112_type 1271 -#define _gather_111_type 1272 -#define _loop0_114_type 1273 -#define _gather_113_type 1274 -#define _loop0_116_type 1275 -#define _gather_115_type 1276 -#define _loop0_118_type 1277 -#define _gather_117_type 1278 -#define _loop0_119_type 1279 -#define _loop0_121_type 1280 -#define _gather_120_type 1281 -#define _tmp_122_type 1282 -#define _loop0_124_type 1283 -#define _gather_123_type 1284 -#define _loop0_126_type 1285 -#define _gather_125_type 1286 -#define _tmp_127_type 1287 -#define _loop0_128_type 1288 -#define _loop0_129_type 1289 -#define _loop0_130_type 1290 -#define _tmp_131_type 1291 -#define _tmp_132_type 1292 -#define _loop0_133_type 1293 -#define _tmp_134_type 1294 -#define _loop0_135_type 1295 -#define _tmp_136_type 1296 -#define _tmp_137_type 1297 -#define _tmp_138_type 1298 -#define _tmp_139_type 1299 -#define _tmp_140_type 1300 -#define _tmp_141_type 1301 -#define _tmp_142_type 1302 -#define _tmp_143_type 1303 -#define _tmp_144_type 1304 -#define _tmp_145_type 1305 -#define _tmp_146_type 1306 -#define _tmp_147_type 1307 -#define _tmp_148_type 1308 -#define _tmp_149_type 1309 -#define _tmp_150_type 1310 -#define _tmp_151_type 1311 -#define _tmp_152_type 1312 -#define _loop1_153_type 1313 -#define _loop1_154_type 1314 -#define _tmp_155_type 1315 -#define _tmp_156_type 1316 +#define invalid_primary_type 1150 // Left-recursive +#define invalid_comprehension_type 1151 +#define invalid_dict_comprehension_type 1152 +#define invalid_parameters_type 1153 +#define invalid_lambda_parameters_type 1154 +#define invalid_star_etc_type 1155 +#define invalid_lambda_star_etc_type 1156 +#define invalid_double_type_comments_type 1157 +#define invalid_with_item_type 1158 +#define invalid_for_target_type 1159 +#define invalid_group_type 1160 +#define invalid_import_from_targets_type 1161 +#define _loop0_1_type 1162 +#define _loop0_2_type 1163 +#define _loop0_4_type 1164 +#define _gather_3_type 1165 +#define _loop0_6_type 1166 +#define _gather_5_type 1167 +#define _loop0_8_type 1168 +#define _gather_7_type 1169 +#define _loop0_10_type 1170 +#define _gather_9_type 1171 +#define _loop1_11_type 1172 +#define _loop0_13_type 1173 +#define _gather_12_type 1174 +#define _tmp_14_type 1175 +#define _tmp_15_type 1176 +#define _tmp_16_type 1177 +#define _tmp_17_type 1178 +#define _tmp_18_type 1179 +#define _tmp_19_type 1180 +#define _tmp_20_type 1181 +#define _tmp_21_type 1182 +#define _loop1_22_type 1183 +#define _tmp_23_type 1184 +#define _tmp_24_type 1185 +#define _loop0_26_type 1186 +#define _gather_25_type 1187 +#define _loop0_28_type 1188 +#define _gather_27_type 1189 +#define _tmp_29_type 1190 +#define _tmp_30_type 1191 +#define _loop0_31_type 1192 +#define _loop1_32_type 1193 +#define _loop0_34_type 1194 +#define _gather_33_type 1195 +#define _tmp_35_type 1196 +#define _loop0_37_type 1197 +#define _gather_36_type 1198 +#define _tmp_38_type 1199 +#define _loop0_40_type 1200 +#define _gather_39_type 1201 +#define _loop0_42_type 1202 +#define _gather_41_type 1203 +#define _loop0_44_type 1204 +#define _gather_43_type 1205 +#define _loop0_46_type 1206 +#define _gather_45_type 1207 +#define _tmp_47_type 1208 +#define _loop1_48_type 1209 +#define _tmp_49_type 1210 +#define _tmp_50_type 1211 +#define _tmp_51_type 1212 +#define _tmp_52_type 1213 +#define _tmp_53_type 1214 +#define _loop0_54_type 1215 +#define _loop0_55_type 1216 +#define _loop0_56_type 1217 +#define _loop1_57_type 1218 +#define _loop0_58_type 1219 +#define _loop1_59_type 1220 +#define _loop1_60_type 1221 +#define _loop1_61_type 1222 +#define _loop0_62_type 1223 +#define _loop1_63_type 1224 +#define _loop0_64_type 1225 +#define _loop1_65_type 1226 +#define _loop0_66_type 1227 +#define _loop1_67_type 1228 +#define _loop1_68_type 1229 +#define _tmp_69_type 1230 +#define _loop0_71_type 1231 +#define _gather_70_type 1232 +#define _loop1_72_type 1233 +#define _loop0_74_type 1234 +#define _gather_73_type 1235 +#define _loop1_75_type 1236 +#define _loop0_76_type 1237 +#define _loop0_77_type 1238 +#define _loop0_78_type 1239 +#define _loop1_79_type 1240 +#define _loop0_80_type 1241 +#define _loop1_81_type 1242 +#define _loop1_82_type 1243 +#define _loop1_83_type 1244 +#define _loop0_84_type 1245 +#define _loop1_85_type 1246 +#define _loop0_86_type 1247 +#define _loop1_87_type 1248 +#define _loop0_88_type 1249 +#define _loop1_89_type 1250 +#define _loop1_90_type 1251 +#define _loop1_91_type 1252 +#define _loop1_92_type 1253 +#define _tmp_93_type 1254 +#define _loop0_95_type 1255 +#define _gather_94_type 1256 +#define _tmp_96_type 1257 +#define _tmp_97_type 1258 +#define _tmp_98_type 1259 +#define _tmp_99_type 1260 +#define _loop1_100_type 1261 +#define _tmp_101_type 1262 +#define _tmp_102_type 1263 +#define _loop0_104_type 1264 +#define _gather_103_type 1265 +#define _loop1_105_type 1266 +#define _loop0_106_type 1267 +#define _loop0_107_type 1268 +#define _loop0_109_type 1269 +#define _gather_108_type 1270 +#define _tmp_110_type 1271 +#define _loop0_112_type 1272 +#define _gather_111_type 1273 +#define _loop0_114_type 1274 +#define _gather_113_type 1275 +#define _loop0_116_type 1276 +#define _gather_115_type 1277 +#define _loop0_118_type 1278 +#define _gather_117_type 1279 +#define _loop0_119_type 1280 +#define _loop0_121_type 1281 +#define _gather_120_type 1282 +#define _tmp_122_type 1283 +#define _loop0_124_type 1284 +#define _gather_123_type 1285 +#define _loop0_126_type 1286 +#define _gather_125_type 1287 +#define _tmp_127_type 1288 +#define _loop0_128_type 1289 +#define _loop0_129_type 1290 +#define _loop0_130_type 1291 +#define _tmp_131_type 1292 +#define _tmp_132_type 1293 +#define _loop0_133_type 1294 +#define _tmp_134_type 1295 +#define _loop0_135_type 1296 +#define _tmp_136_type 1297 +#define _tmp_137_type 1298 +#define _tmp_138_type 1299 +#define _tmp_139_type 1300 +#define _tmp_140_type 1301 +#define _tmp_141_type 1302 +#define _tmp_142_type 1303 +#define _tmp_143_type 1304 +#define _tmp_144_type 1305 +#define _tmp_145_type 1306 +#define _tmp_146_type 1307 +#define _tmp_147_type 1308 +#define _tmp_148_type 1309 +#define _tmp_149_type 1310 +#define _tmp_150_type 1311 +#define _tmp_151_type 1312 +#define _tmp_152_type 1313 +#define _loop1_153_type 1314 +#define _loop1_154_type 1315 +#define _tmp_155_type 1316 +#define _tmp_156_type 1317 static mod_ty file_rule(Parser *p); static mod_ty interactive_rule(Parser *p); @@ -543,6 +544,7 @@ static void *invalid_assignment_rule(Parser *p); static expr_ty invalid_ann_assign_target_rule(Parser *p); static void *invalid_del_stmt_rule(Parser *p); static void *invalid_block_rule(Parser *p); +static void *invalid_primary_rule(Parser *p); static void *invalid_comprehension_rule(Parser *p); static void *invalid_dict_comprehension_rule(Parser *p); static void *invalid_parameters_rule(Parser *p); @@ -10274,6 +10276,7 @@ await_primary_rule(Parser *p) // Left-recursive // primary: +// | invalid_primary // | primary '.' NAME // | primary genexp // | primary '(' arguments? ')' @@ -10327,6 +10330,25 @@ primary_raw(Parser *p) UNUSED(_start_lineno); // Only used by EXTRA macro int _start_col_offset = p->tokens[_mark]->col_offset; UNUSED(_start_col_offset); // Only used by EXTRA macro + { // invalid_primary + if (p->error_indicator) { + D(p->level--); + return NULL; + } + D(fprintf(stderr, "%*c> primary[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "invalid_primary")); + void *invalid_primary_var; + if ( + (invalid_primary_var = invalid_primary_rule(p)) // invalid_primary + ) + { + D(fprintf(stderr, "%*c+ primary[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "invalid_primary")); + _res = invalid_primary_var; + goto done; + } + p->mark = _mark; + D(fprintf(stderr, "%*c%s primary[%d-%d]: %s failed!\n", p->level, ' ', + p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "invalid_primary")); + } { // primary '.' NAME if (p->error_indicator) { D(p->level--); @@ -15047,6 +15069,51 @@ invalid_block_rule(Parser *p) return _res; } +// Left-recursive +// invalid_primary: primary '{' +static void * +invalid_primary_rule(Parser *p) +{ + D(p->level++); + if (p->error_indicator) { + D(p->level--); + return NULL; + } + void * _res = NULL; + int _mark = p->mark; + { // primary '{' + if (p->error_indicator) { + D(p->level--); + return NULL; + } + D(fprintf(stderr, "%*c> invalid_primary[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "primary '{'")); + Token * a; + expr_ty primary_var; + if ( + (primary_var = primary_rule(p)) // primary + && + (a = _PyPegen_expect_token(p, 25)) // token='{' + ) + { + D(fprintf(stderr, "%*c+ invalid_primary[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "primary '{'")); + _res = RAISE_SYNTAX_ERROR_KNOWN_LOCATION ( a , "invalid syntax" ); + if (_res == NULL && PyErr_Occurred()) { + p->error_indicator = 1; + D(p->level--); + return NULL; + } + goto done; + } + p->mark = _mark; + D(fprintf(stderr, "%*c%s invalid_primary[%d-%d]: %s failed!\n", p->level, ' ', + p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "primary '{'")); + } + _res = NULL; + done: + D(p->level--); + return _res; +} + // invalid_comprehension: ('[' | '(' | '{') starred_expression for_if_clauses static void * invalid_comprehension_rule(Parser *p)

1 0

[3.9] Remove git conflict lines from test_parse_directory in peg_generator (GH-23007)
by lysnikolaou Oct. 27, 2020

Oct. 27, 2020

https://github.com/python/cpython/commit/8aee019d71b9ed44107bdc8a0c9c132059… commit: 8aee019d71b9ed44107bdc8a0c9c132059c0e1ef branch: 3.9 author: Lysandros Nikolaou <lisandrosnik(a)gmail.com> committer: lysnikolaou <lisandrosnik(a)gmail.com> date: 2020-10-28T00:37:36+02:00 summary: [3.9] Remove git conflict lines from test_parse_directory in peg_generator (GH-23007) files: M Tools/peg_generator/scripts/test_parse_directory.py diff --git a/Tools/peg_generator/scripts/test_parse_directory.py b/Tools/peg_generator/scripts/test_parse_directory.py index 100db1d424c7c..6dfdbb9170e6e 100755 --- a/Tools/peg_generator/scripts/test_parse_directory.py +++ b/Tools/peg_generator/scripts/test_parse_directory.py @@ -5,14 +5,9 @@ import os import sys import time -import traceback import tokenize -<<<<<<< HEAD import _peg_parser -from glob import glob -======= from glob import glob, escape ->>>>>>> 9355868458... bpo-41043: Escape literal part of the path for glob(). (GH-20994) from pathlib import PurePath from typing import List, Optional, Any, Tuple

1 0

bpo-41805: Documentation for PEP 585 (GH-22615)
by gvanrossum Oct. 27, 2020

Oct. 27, 2020

https://github.com/python/cpython/commit/4173320920706b49a004bdddd8d7108e89… commit: 4173320920706b49a004bdddd8d7108e8984e3fc branch: master author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: gvanrossum <guido(a)python.org> date: 2020-10-27T14:37:18-07:00 summary: bpo-41805: Documentation for PEP 585 (GH-22615) files: A Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst M Doc/glossary.rst M Doc/library/stdtypes.rst M Doc/library/types.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 847500e556056..4fd01e0160c26 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -483,6 +483,13 @@ Glossary See also the :term:`single dispatch` glossary entry, the :func:`functools.singledispatch` decorator, and :pep:`443`. + generic type + A :term:`type` that can be parameterized; typically a container like + :class:`list`. Used for :term:`type hints <type hint>` and + :term:`annotations <annotation>`. + + See :pep:`483` for more details, and :mod:`typing` or + :ref:`generic alias type <types-genericalias>` for its uses. GIL See :term:`global interpreter lock`. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index c74d164463601..3fd94ea1bd310 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4763,6 +4763,200 @@ define these methods must provide them as a normal Python accessible method. Compared to the overhead of setting up the runtime context, the overhead of a single class dictionary lookup is negligible. + +.. _types-genericalias: + +Generic Alias Type +================== + +.. index:: + object: GenericAlias + pair: Generic; Alias + +``GenericAlias`` objects are created by subscripting a class (usually a +container), such as ``list[int]``. They are intended primarily for +:term:`type annotations <annotation>`. + +Usually, the :ref:`subscription <subscriptions>` of container objects calls the +method :meth:`__getitem__` of the object. However, the subscription of some +containers' classes may call the classmethod :meth:`__class_getitem__` of the +class instead. The classmethod :meth:`__class_getitem__` should return a +``GenericAlias`` object. + +.. note:: + If the :meth:`__getitem__` of the class' metaclass is present, it will take + precedence over the :meth:`__class_getitem__` defined in the class (see + :pep:`560` for more details). + +The ``GenericAlias`` object acts as a proxy for :term:`generic types +<generic type>`, implementing *parameterized generics* - a specific instance +of a generic which provides the types for container elements. + +The user-exposed type for the ``GenericAlias`` object can be accessed from +:data:`types.GenericAlias` and used for :func:`isinstance` checks. + +.. describe:: T[X, Y, ...] + + Creates a ``GenericAlias`` representing a type ``T`` containing elements + of types *X*, *Y*, and more depending on the ``T`` used. + For example, a function expecting a :class:`list` containing + :class:`float` elements:: + + def average(values: list[float]) -> float: + return sum(values) / len(values) + + Another example for :term:`mapping` objects, using a :class:`dict`, which + is a generic type expecting two type parameters representing the key type + and the value type. In this example, the function expects a ``dict`` with + keys of type :class:`str` and values of type :class:`int`:: + + def send_post_request(url: str, body: dict[str, int]) -> None: + ... + +The builtin functions :func:`isinstance` and :func:`issubclass` do not accept +``GenericAlias`` types for their second argument:: + + >>> isinstance([1, 2], list[str]) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: isinstance() argument 2 cannot be a parameterized generic + +The Python runtime does not enforce :term:`type annotations <annotation>`. +This extends to generic types and their type parameters. When creating +an object from a ``GenericAlias``, container elements are not checked +against their type. For example, the following code is discouraged, but will +run without errors:: + + >>> t = list[str] + >>> t([1, 2, 3]) + [1, 2, 3] + +Furthermore, parameterized generics erase type parameters during object +creation:: + + >>> t = list[str] + >>> type(t) + <class 'types.GenericAlias'> + + >>> l = t() + >>> type(l) + <class 'list'> + +Calling :func:`repr` or :func:`str` on a generic shows the parameterized type:: + + >>> repr(list[int]) + 'list[int]' + + >>> str(list[int]) + 'list[int]' + +The :meth:`__getitem__` method of generics will raise an exception to disallow +mistakes like ``dict[str][str]``:: + + >>> dict[str][str] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: There are no type variables left in dict[str] + +However, such expressions are valid when :ref:`type variables <generics>` are +used. The index must have as many elements as there are type variable items +in the ``GenericAlias`` object's :attr:`__args__ <genericalias.__args__>`. :: + + >>> from typing import TypeVar + >>> Y = TypeVar('Y') + >>> dict[str, Y][int] + dict[str, int] + + +Standard Generic Collections +---------------------------- + +These standard library collections support parameterized generics. + +* :class:`tuple` +* :class:`list` +* :class:`dict` +* :class:`set` +* :class:`frozenset` +* :class:`type` +* :class:`collections.deque` +* :class:`collections.defaultdict` +* :class:`collections.OrderedDict` +* :class:`collections.Counter` +* :class:`collections.ChainMap` +* :class:`collections.abc.Awaitable` +* :class:`collections.abc.Coroutine` +* :class:`collections.abc.AsyncIterable` +* :class:`collections.abc.AsyncIterator` +* :class:`collections.abc.AsyncGenerator` +* :class:`collections.abc.Iterable` +* :class:`collections.abc.Iterator` +* :class:`collections.abc.Generator` +* :class:`collections.abc.Reversible` +* :class:`collections.abc.Container` +* :class:`collections.abc.Collection` +* :class:`collections.abc.Callable` +* :class:`collections.abc.Set` +* :class:`collections.abc.MutableSet` +* :class:`collections.abc.Mapping` +* :class:`collections.abc.MutableMapping` +* :class:`collections.abc.Sequence` +* :class:`collections.abc.MutableSequence` +* :class:`collections.abc.ByteString` +* :class:`collections.abc.MappingView` +* :class:`collections.abc.KeysView` +* :class:`collections.abc.ItemsView` +* :class:`collections.abc.ValuesView` +* :class:`contextlib.AbstractContextManager` +* :class:`contextlib.AbstractAsyncContextManager` +* :ref:`re.Pattern <re-objects>` +* :ref:`re.Match <match-objects>` + + +Special Attributes of Generic Alias +----------------------------------- + +All parameterized generics implement special read-only attributes. + +.. attribute:: genericalias.__origin__ + + This attribute points at the non-parameterized generic class:: + + >>> list[int].__origin__ + <class 'list'> + + +.. attribute:: genericalias.__args__ + + This attribute is a :class:`tuple` (possibly of length 1) of generic + types passed to the original :meth:`__class_getitem__` + of the generic container:: + + >>> dict[str, list[int]].__args__ + (<class 'str'>, list[int]) + + +.. attribute:: genericalias.__parameters__ + + This attribute is a lazily computed tuple (possibly empty) of unique type + variables found in ``__args__``:: + + >>> from typing import TypeVar + + >>> T = TypeVar('T') + >>> list[T].__parameters__ + (~T,) + + +.. seealso:: + + * :pep:`585` -- "Type Hinting Generics In Standard Collections" + * :meth:`__class_getitem__` -- Used to implement parameterized generics. + * :ref:`generics` -- Generics in the :mod:`typing` module. + +.. versionadded:: 3.9 + + .. _types-union: Union Type diff --git a/Doc/library/types.rst b/Doc/library/types.rst index a5cf9ab1344a3..00720559d0a4a 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -262,6 +262,13 @@ Standard names are defined for the following types: .. versionadded:: 3.10 +.. data:: GenericAlias + + The type of :ref:`parameterized generics <types-genericalias>` such as + ``list[int]``. + + .. versionadded:: 3.9 + .. data:: Union The type of :ref:`union type expressions<types-union>`. diff --git a/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst b/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst new file mode 100644 index 0000000000000..9c9134350a317 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-10-10-01-36-37.bpo-41805.l-CGv5.rst @@ -0,0 +1,3 @@ +Documented :ref:`generic alias type <types-genericalias>` and +:data:`types.GenericAlias`. Also added an entry in glossary for +:term:`generic types <generic type>`.

1 0

bpo-42161: Micro-optimize _collections._count_elements() (GH-23008)
by vstinner Oct. 27, 2020

Oct. 27, 2020

https://github.com/python/cpython/commit/35b95aaf21534e4a8e3370dfd6f7482265… commit: 35b95aaf21534e4a8e3370dfd6f7482265316c9e branch: master author: Victor Stinner <vstinner(a)python.org> committer: vstinner <vstinner(a)python.org> date: 2020-10-27T22:24:33+01:00 summary: bpo-42161: Micro-optimize _collections._count_elements() (GH-23008) Move the _PyLong_GetOne() call outside the fast-path loop. files: M Modules/_collectionsmodule.c diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 8990071f519ea..157875067635a 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -2278,6 +2278,7 @@ _collections__count_elements_impl(PyObject *module, PyObject *mapping, PyObject *dict_get; PyObject *mapping_setitem; PyObject *dict_setitem; + PyObject *one = _PyLong_GetOne(); // borrowed reference it = PyObject_GetIter(iterable); if (it == NULL) @@ -2324,10 +2325,10 @@ _collections__count_elements_impl(PyObject *module, PyObject *mapping, if (oldval == NULL) { if (PyErr_Occurred()) goto done; - if (_PyDict_SetItem_KnownHash(mapping, key, _PyLong_GetOne(), hash) < 0) + if (_PyDict_SetItem_KnownHash(mapping, key, one, hash) < 0) goto done; } else { - newval = PyNumber_Add(oldval, _PyLong_GetOne()); + newval = PyNumber_Add(oldval, one); if (newval == NULL) goto done; if (_PyDict_SetItem_KnownHash(mapping, key, newval, hash) < 0) @@ -2336,13 +2337,13 @@ _collections__count_elements_impl(PyObject *module, PyObject *mapping, } Py_DECREF(key); } - } else { + } + else { bound_get = _PyObject_GetAttrId(mapping, &PyId_get); if (bound_get == NULL) goto done; PyObject *zero = _PyLong_GetZero(); // borrowed reference - PyObject *one = _PyLong_GetOne(); // borrowed reference while (1) { key = PyIter_Next(it); if (key == NULL)

1 0