Mailman 3 October 2020 - Python-checkins

bpo-42198: New section in stdtypes for type annotation types (GH-23063)
by gvanrossum Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/3bf0d02f2817c48b6ee61a95b52a6d76ad… commit: 3bf0d02f2817c48b6ee61a95b52a6d76ad543be9 branch: master author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: gvanrossum <guido(a)python.org> date: 2020-10-31T13:25:44-07:00 summary: bpo-42198: New section in stdtypes for type annotation types (GH-23063) files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 8a9cadde7d5dc..84d5a3a59f0b4 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4764,10 +4764,20 @@ Compared to the overhead of setting up the runtime context, the overhead of a single class dictionary lookup is negligible. +Type Annotation Types --- :ref:`Generic Alias <types-genericalias>`, :ref:`Union <types-union>` +=============================================================================================== + +.. index:: + single: annotation; type annotation; type hint + +The core built-in types for :term:`type annotations <annotation>` are +:ref:`Generic Alias <types-genericalias>` and :ref:`Union <types-union>`. + + .. _types-genericalias: Generic Alias Type -================== +------------------ .. index:: object: GenericAlias @@ -4870,7 +4880,7 @@ in the ``GenericAlias`` object's :attr:`__args__ <genericalias.__args__>`. :: Standard Generic Collections ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ These standard library collections support parameterized generics. @@ -4915,7 +4925,7 @@ These standard library collections support parameterized generics. Special Attributes of Generic Alias ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ All parameterized generics implement special read-only attributes. @@ -4961,7 +4971,7 @@ All parameterized generics implement special read-only attributes. .. _types-union: Union Type -========== +---------- .. index:: object: Union

1 0

[3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) (GH-23066)
by miss-islington Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/cfcb952e30e01d7cce430829af8edc7afc… commit: cfcb952e30e01d7cce430829af8edc7afc94e0b1 branch: 3.9 author: Lysandros Nikolaou <lisandrosnik(a)gmail.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-31T12:06:03-07:00 summary: [3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) (GH-23066) Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> (cherry picked from commit 02cdfc93f82fecdb7eae97a868d4ee222b9875d9) Automerge-Triggered-By: GH:lysnikolaou files: A Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst M Lib/test/test_syntax.py M Parser/pegen/parse.c M Tools/peg_generator/pegen/c_generator.py diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index a95992d869e5e..1336231fbbfbf 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -964,6 +964,14 @@ def func2(): """ self._check_error(code, "invalid syntax") + def test_invalid_line_continuation_left_recursive(self): + # Check bpo-42218: SyntaxErrors following left-recursive rules + # (t_primary_raw in this case) need to be tested explicitly + self._check_error("A.\u018a\\ ", + "unexpected character after line continuation character") + self._check_error("A.\u03bc\\\n", + "unexpected EOF while parsing") + def test_main(): support.run_unittest(SyntaxTestCase) from test import test_syntax diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst new file mode 100644 index 0000000000000..a38a310e4b45b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst @@ -0,0 +1,3 @@ +Fixed a bug in the PEG parser that was causing crashes in debug mode. Now errors are checked +in left-recursive rules to avoid cases where such errors do not get handled in time and appear +as long-distance crashes in other places. diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c index bae9463e274a3..97cefa9c2a3d4 100644 --- a/Parser/pegen/parse.c +++ b/Parser/pegen/parse.c @@ -3460,6 +3460,8 @@ dotted_name_rule(Parser *p) } p->mark = _mark; void *_raw = dotted_name_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9044,6 +9046,8 @@ bitwise_or_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_or_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9158,6 +9162,8 @@ bitwise_xor_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_xor_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9272,6 +9278,8 @@ bitwise_and_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_and_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9386,6 +9394,8 @@ shift_expr_rule(Parser *p) } p->mark = _mark; void *_raw = shift_expr_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9539,6 +9549,8 @@ sum_rule(Parser *p) } p->mark = _mark; void *_raw = sum_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9698,6 +9710,8 @@ term_rule(Parser *p) } p->mark = _mark; void *_raw = term_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -10302,6 +10316,8 @@ primary_rule(Parser *p) } p->mark = _mark; void *_raw = primary_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -13962,6 +13978,8 @@ t_primary_rule(Parser *p) } p->mark = _mark; void *_raw = t_primary_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index d0abc12b4026a..b4d6a0bab51f4 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -496,6 +496,9 @@ def _set_up_rule_memoization(self, node: Rule, result_type: str) -> None: ) self.print("p->mark = _mark;") self.print(f"void *_raw = {node.name}_raw(p);") + self.print("if (p->error_indicator)") + with self.indent(): + self.print("return NULL;") self.print("if (_raw == NULL || p->mark <= _resmark)") with self.indent(): self.print("break;")

1 0

bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
by lysnikolaou Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/02cdfc93f82fecdb7eae97a868d4ee222b… commit: 02cdfc93f82fecdb7eae97a868d4ee222b9875d9 branch: master author: Lysandros Nikolaou <lisandrosnik(a)gmail.com> committer: lysnikolaou <lisandrosnik(a)gmail.com> date: 2020-10-31T20:31:41+02:00 summary: bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst M Lib/test/test_syntax.py M Parser/parser.c M Tools/peg_generator/pegen/c_generator.py diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index e89d9401f2c39..91ca1db43a74f 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -972,6 +972,14 @@ def func2(): """ self._check_error(code, "invalid syntax") + def test_invalid_line_continuation_left_recursive(self): + # Check bpo-42218: SyntaxErrors following left-recursive rules + # (t_primary_raw in this case) need to be tested explicitly + self._check_error("A.\u018a\\ ", + "unexpected character after line continuation character") + self._check_error("A.\u03bc\\\n", + "unexpected EOF while parsing") + def test_main(): support.run_unittest(SyntaxTestCase) from test import test_syntax diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst new file mode 100644 index 0000000000000..a38a310e4b45b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-31-17-50-23.bpo-42218.Dp_Z3v.rst @@ -0,0 +1,3 @@ +Fixed a bug in the PEG parser that was causing crashes in debug mode. Now errors are checked +in left-recursive rules to avoid cases where such errors do not get handled in time and appear +as long-distance crashes in other places. diff --git a/Parser/parser.c b/Parser/parser.c index a882a81344cc6..48ebfe65aedaf 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -3461,6 +3461,8 @@ dotted_name_rule(Parser *p) } p->mark = _mark; void *_raw = dotted_name_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9045,6 +9047,8 @@ bitwise_or_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_or_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9159,6 +9163,8 @@ bitwise_xor_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_xor_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9273,6 +9279,8 @@ bitwise_and_rule(Parser *p) } p->mark = _mark; void *_raw = bitwise_and_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9387,6 +9395,8 @@ shift_expr_rule(Parser *p) } p->mark = _mark; void *_raw = shift_expr_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9540,6 +9550,8 @@ sum_rule(Parser *p) } p->mark = _mark; void *_raw = sum_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -9699,6 +9711,8 @@ term_rule(Parser *p) } p->mark = _mark; void *_raw = term_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -10303,6 +10317,8 @@ primary_rule(Parser *p) } p->mark = _mark; void *_raw = primary_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; @@ -13943,6 +13959,8 @@ t_primary_rule(Parser *p) } p->mark = _mark; void *_raw = t_primary_raw(p); + if (p->error_indicator) + return NULL; if (_raw == NULL || p->mark <= _resmark) break; _resmark = p->mark; diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index 52bdb844e6bdd..6af0d3f7a2a14 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -502,6 +502,9 @@ def _set_up_rule_memoization(self, node: Rule, result_type: str) -> None: ) self.print("p->mark = _mark;") self.print(f"void *_raw = {node.name}_raw(p);") + self.print("if (p->error_indicator)") + with self.indent(): + self.print("return NULL;") self.print("if (_raw == NULL || p->mark <= _resmark)") with self.indent(): self.print("break;")

1 0

bpo-42198: Improve consistency of Union docs (GH-23029)
by miss-islington Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/d21cb2d5ee56b8975d92e2ee094aba81f0… commit: d21cb2d5ee56b8975d92e2ee094aba81f0801be5 branch: master author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-31T08:08:17-07:00 summary: bpo-42198: Improve consistency of Union docs (GH-23029) No backport is required since union is only in 3.10. This addresses "3. Consistency nitpicks for Union's docs" in the bpo. Please skip news. Thank you. files: M Doc/library/stdtypes.rst M Doc/whatsnew/3.10.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 09477e63786c5..8a9cadde7d5dc 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4968,15 +4968,16 @@ Union Type pair: union; type A union object holds the value of the ``|`` (bitwise or) operation on -multiple :ref:`type objects<bltin-type-objects>`. These types are intended -primarily for type annotations. The union type expression enables cleaner -type hinting syntax compared to :data:`typing.Union`. +multiple :ref:`type objects <bltin-type-objects>`. These types are intended +primarily for :term:`type annotations <annotation>`. The union type expression +enables cleaner type hinting syntax compared to :data:`typing.Union`. .. describe:: X | Y | ... Defines a union object which holds types *X*, *Y*, and so forth. ``X | Y`` means either X or Y. It is equivalent to ``typing.Union[X, Y]``. - Example:: + For example, the following function expects an argument of type + :class:`int` or :class:`float`:: def square(number: int | float) -> int | float: return number ** 2 @@ -4985,15 +4986,15 @@ type hinting syntax compared to :data:`typing.Union`. Union objects can be tested for equality with other union objects. Details: - * Unions of unions are flattened, e.g.:: + * Unions of unions are flattened:: (int | str) | float == int | str | float - * Redundant types are removed, e.g.:: + * Redundant types are removed:: int | str | int == int | str - * When comparing unions, the order is ignored, e.g.:: + * When comparing unions, the order is ignored:: int | str == str | int @@ -5012,14 +5013,8 @@ type hinting syntax compared to :data:`typing.Union`. >>> isinstance("", int | str) True - .. - At the time of writing this, there is no documentation for parameterized - generics or PEP 585. Thus the link currently points to PEP 585 itself. - Please change the link for parameterized generics to reference the correct - documentation once documentation for PEP 585 becomes available. - - However, union objects containing `parameterized generics - <https://www.python.org/dev/peps/pep-0585/>`_ cannot be used:: + However, union objects containing :ref:`parameterized generics + <types-genericalias>` cannot be used:: >>> isinstance(1, int | list[int]) Traceback (most recent call last): @@ -5033,20 +5028,16 @@ type hinting syntax compared to :data:`typing.Union`. >>> issubclass(bool, int | str) True - .. - Once again, please change the link below for parameterized generics to - reference the correct documentation once documentation for PEP 585 - becomes available. - - However, union objects containing `parameterized generics - <https://www.python.org/dev/peps/pep-0585/>`_ cannot be used:: + However, union objects containing :ref:`parameterized generics + <types-genericalias>` cannot be used:: >>> issubclass(bool, bool | list[str]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: issubclass() argument 2 cannot contain a parameterized generic -The type of a union object is :data:`types.Union`. An object cannot be +The user-exposed type for the union object can be accessed from +:data:`types.Union` and used for :func:`isinstance` checks. An object cannot be instantiated from the type:: >>> import types diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index f4d72908486dd..60dee0c6bd165 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -134,7 +134,7 @@ arguments of multiple types, :data:`typing.Union` was used:: return number ** 2 -Now, type hints can be written in a more succinct manner:: +Type hints can now be written in a more succinct manner:: def square(number: int | float) -> int | float: return number ** 2

1 0

[3.9] bpo-42198: Document __new__ for types.GenericAlias (GH-23039) (GH-23061)
by miss-islington Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/dbaa07db671c4e5842629c8be912f2b037… commit: dbaa07db671c4e5842629c8be912f2b0370be794 branch: 3.9 author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-31T08:05:36-07:00 summary: [3.9] bpo-42198: Document __new__ for types.GenericAlias (GH-23039) (GH-23061) (cherry picked from commit bcbf758476c1148993ddf4b54d3f6169b973ee1c) files: M Doc/library/stdtypes.rst M Doc/library/types.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index e19e76f8311c8..a48cfa1327791 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4760,7 +4760,8 @@ The ``GenericAlias`` object acts as a proxy for :term:`generic types of a generic which provides the types for container elements. The user-exposed type for the ``GenericAlias`` object can be accessed from -:data:`types.GenericAlias` and used for :func:`isinstance` checks. +:class:`types.GenericAlias` and used for :func:`isinstance` checks. It can +also be used to create ``GenericAlias`` objects directly. .. describe:: T[X, Y, ...] diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 1573e460f2f2b..5d68c6852678c 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -242,11 +242,22 @@ Standard names are defined for the following types: Defaults to ``None``. Previously the attribute was optional. -.. data:: GenericAlias +.. class:: GenericAlias(t_origin, t_args) The type of :ref:`parameterized generics <types-genericalias>` such as ``list[int]``. + ``t_origin`` should be a non-parameterized generic class, such as ``list``, + ``tuple`` or ``dict``. ``t_args`` should be a :class:`tuple` (possibly of + length 1) of types which parameterize ``t_origin``:: + + >>> from types import GenericAlias + + >>> list[int] == GenericAlias(list, (int,)) + True + >>> dict[str, int] == GenericAlias(dict, (str, int)) + True + .. versionadded:: 3.9

1 0

bpo-40956: Convert _sqlite3 module level functions to Argument Clinic (GH-22484)
by corona10 Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/7d210271579ae31f43b32f73c2aff5bc4f… commit: 7d210271579ae31f43b32f73c2aff5bc4fe0d27f branch: master author: Erlend Egeberg Aasland <erlend.aasland(a)innova.no> committer: corona10 <donghee.na92(a)gmail.com> date: 2020-10-31T15:07:44+09:00 summary: bpo-40956: Convert _sqlite3 module level functions to Argument Clinic (GH-22484) files: A Modules/_sqlite/clinic/module.c.h M Modules/_sqlite/microprotocols.c M Modules/_sqlite/microprotocols.h M Modules/_sqlite/module.c diff --git a/Modules/_sqlite/clinic/module.c.h b/Modules/_sqlite/clinic/module.c.h new file mode 100644 index 0000000000000..fb1e1187b209f --- /dev/null +++ b/Modules/_sqlite/clinic/module.c.h @@ -0,0 +1,222 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(pysqlite_complete_statement__doc__, +"complete_statement($module, /, statement)\n" +"--\n" +"\n" +"Checks if a string contains a complete SQL statement. Non-standard."); + +#define PYSQLITE_COMPLETE_STATEMENT_METHODDEF \ + {"complete_statement", (PyCFunction)(void(*)(void))pysqlite_complete_statement, METH_FASTCALL|METH_KEYWORDS, pysqlite_complete_statement__doc__}, + +static PyObject * +pysqlite_complete_statement_impl(PyObject *module, const char *statement); + +static PyObject * +pysqlite_complete_statement(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"statement", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "complete_statement", 0}; + PyObject *argsbuf[1]; + const char *statement; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("complete_statement", "argument 'statement'", "str", args[0]); + goto exit; + } + Py_ssize_t statement_length; + statement = PyUnicode_AsUTF8AndSize(args[0], &statement_length); + if (statement == NULL) { + goto exit; + } + if (strlen(statement) != (size_t)statement_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + return_value = pysqlite_complete_statement_impl(module, statement); + +exit: + return return_value; +} + +PyDoc_STRVAR(pysqlite_enable_shared_cache__doc__, +"enable_shared_cache($module, /, do_enable)\n" +"--\n" +"\n" +"Enable or disable shared cache mode for the calling thread.\n" +"\n" +"Experimental/Non-standard."); + +#define PYSQLITE_ENABLE_SHARED_CACHE_METHODDEF \ + {"enable_shared_cache", (PyCFunction)(void(*)(void))pysqlite_enable_shared_cache, METH_FASTCALL|METH_KEYWORDS, pysqlite_enable_shared_cache__doc__}, + +static PyObject * +pysqlite_enable_shared_cache_impl(PyObject *module, int do_enable); + +static PyObject * +pysqlite_enable_shared_cache(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"do_enable", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "enable_shared_cache", 0}; + PyObject *argsbuf[1]; + int do_enable; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + do_enable = _PyLong_AsInt(args[0]); + if (do_enable == -1 && PyErr_Occurred()) { + goto exit; + } + return_value = pysqlite_enable_shared_cache_impl(module, do_enable); + +exit: + return return_value; +} + +PyDoc_STRVAR(pysqlite_register_adapter__doc__, +"register_adapter($module, type, caster, /)\n" +"--\n" +"\n" +"Registers an adapter with pysqlite\'s adapter registry. Non-standard."); + +#define PYSQLITE_REGISTER_ADAPTER_METHODDEF \ + {"register_adapter", (PyCFunction)(void(*)(void))pysqlite_register_adapter, METH_FASTCALL, pysqlite_register_adapter__doc__}, + +static PyObject * +pysqlite_register_adapter_impl(PyObject *module, PyTypeObject *type, + PyObject *caster); + +static PyObject * +pysqlite_register_adapter(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyTypeObject *type; + PyObject *caster; + + if (!_PyArg_CheckPositional("register_adapter", nargs, 2, 2)) { + goto exit; + } + type = (PyTypeObject *)args[0]; + caster = args[1]; + return_value = pysqlite_register_adapter_impl(module, type, caster); + +exit: + return return_value; +} + +PyDoc_STRVAR(pysqlite_register_converter__doc__, +"register_converter($module, name, converter, /)\n" +"--\n" +"\n" +"Registers a converter with pysqlite. Non-standard."); + +#define PYSQLITE_REGISTER_CONVERTER_METHODDEF \ + {"register_converter", (PyCFunction)(void(*)(void))pysqlite_register_converter, METH_FASTCALL, pysqlite_register_converter__doc__}, + +static PyObject * +pysqlite_register_converter_impl(PyObject *module, PyObject *orig_name, + PyObject *callable); + +static PyObject * +pysqlite_register_converter(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *orig_name; + PyObject *callable; + + if (!_PyArg_CheckPositional("register_converter", nargs, 2, 2)) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("register_converter", "argument 1", "str", args[0]); + goto exit; + } + if (PyUnicode_READY(args[0]) == -1) { + goto exit; + } + orig_name = args[0]; + callable = args[1]; + return_value = pysqlite_register_converter_impl(module, orig_name, callable); + +exit: + return return_value; +} + +PyDoc_STRVAR(pysqlite_enable_callback_trace__doc__, +"enable_callback_tracebacks($module, enable, /)\n" +"--\n" +"\n" +"Enable or disable callback functions throwing errors to stderr."); + +#define PYSQLITE_ENABLE_CALLBACK_TRACE_METHODDEF \ + {"enable_callback_tracebacks", (PyCFunction)pysqlite_enable_callback_trace, METH_O, pysqlite_enable_callback_trace__doc__}, + +static PyObject * +pysqlite_enable_callback_trace_impl(PyObject *module, int enable); + +static PyObject * +pysqlite_enable_callback_trace(PyObject *module, PyObject *arg) +{ + PyObject *return_value = NULL; + int enable; + + enable = _PyLong_AsInt(arg); + if (enable == -1 && PyErr_Occurred()) { + goto exit; + } + return_value = pysqlite_enable_callback_trace_impl(module, enable); + +exit: + return return_value; +} + +PyDoc_STRVAR(pysqlite_adapt__doc__, +"adapt($module, obj, proto=PrepareProtocolType, alt=<unrepresentable>, /)\n" +"--\n" +"\n" +"Adapt given object to given protocol. Non-standard."); + +#define PYSQLITE_ADAPT_METHODDEF \ + {"adapt", (PyCFunction)(void(*)(void))pysqlite_adapt, METH_FASTCALL, pysqlite_adapt__doc__}, + +static PyObject * +pysqlite_adapt_impl(PyObject *module, PyObject *obj, PyObject *proto, + PyObject *alt); + +static PyObject * +pysqlite_adapt(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *obj; + PyObject *proto = (PyObject*)pysqlite_PrepareProtocolType; + PyObject *alt = NULL; + + if (!_PyArg_CheckPositional("adapt", nargs, 1, 3)) { + goto exit; + } + obj = args[0]; + if (nargs < 2) { + goto skip_optional; + } + proto = args[1]; + if (nargs < 3) { + goto skip_optional; + } + alt = args[2]; +skip_optional: + return_value = pysqlite_adapt_impl(module, obj, proto, alt); + +exit: + return return_value; +} +/*[clinic end generated code: output=d87990f941c209fa input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/microprotocols.c b/Modules/_sqlite/microprotocols.c index ddc30e8a89b46..cf1fefd671851 100644 --- a/Modules/_sqlite/microprotocols.c +++ b/Modules/_sqlite/microprotocols.c @@ -29,7 +29,6 @@ #include "microprotocols.h" #include "prepare_protocol.h" - /** the adapters registry **/ static PyObject *psyco_adapters = NULL; @@ -150,15 +149,3 @@ pysqlite_microprotocols_adapt(PyObject *obj, PyObject *proto, PyObject *alt) PyErr_SetString(pysqlite_ProgrammingError, "can't adapt"); return NULL; } - -/** module-level functions **/ - -PyObject * -pysqlite_adapt(pysqlite_Cursor *self, PyObject *args) -{ - PyObject *obj, *alt = NULL; - PyObject *proto = (PyObject*)pysqlite_PrepareProtocolType; - - if (!PyArg_ParseTuple(args, "O|OO", &obj, &proto, &alt)) return NULL; - return pysqlite_microprotocols_adapt(obj, proto, alt); -} diff --git a/Modules/_sqlite/microprotocols.h b/Modules/_sqlite/microprotocols.h index 87df6bac55797..e9adef916e724 100644 --- a/Modules/_sqlite/microprotocols.h +++ b/Modules/_sqlite/microprotocols.h @@ -44,9 +44,4 @@ extern int pysqlite_microprotocols_add( extern PyObject *pysqlite_microprotocols_adapt( PyObject *obj, PyObject *proto, PyObject *alt); -extern PyObject * - pysqlite_adapt(pysqlite_Cursor* self, PyObject *args); -#define pysqlite_adapt_doc \ - "adapt(obj, protocol, alternate) -> adapt obj to given protocol. Non-standard." - #endif /* !defined(PSYCOPG_MICROPROTOCOLS_H) */ diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index 0297e2fab292e..33324402385f4 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -33,6 +33,12 @@ #error "SQLite 3.7.3 or higher required" #endif +#include "clinic/module.c.h" +/*[clinic input] +module _sqlite3 +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=81e330492d57488e]*/ + /* static objects at module-level */ PyObject *pysqlite_Error = NULL; @@ -102,18 +108,19 @@ Opens a connection to the SQLite database file *database*. You can use\n\ \":memory:\" to open a database connection to a database that resides in\n\ RAM instead of on disk."); -static PyObject* module_complete(PyObject* self, PyObject* args, PyObject* - kwargs) -{ - static char *kwlist[] = {"statement", NULL}; - char* statement; +/*[clinic input] +_sqlite3.complete_statement as pysqlite_complete_statement - PyObject* result; + statement: str - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s", kwlist, &statement)) - { - return NULL; - } +Checks if a string contains a complete SQL statement. Non-standard. +[clinic start generated code]*/ + +static PyObject * +pysqlite_complete_statement_impl(PyObject *module, const char *statement) +/*[clinic end generated code: output=e55f1ff1952df558 input=f6b24996b31c5c33]*/ +{ + PyObject* result; if (sqlite3_complete(statement)) { result = Py_True; @@ -126,23 +133,22 @@ static PyObject* module_complete(PyObject* self, PyObject* args, PyObject* return result; } -PyDoc_STRVAR(module_complete_doc, -"complete_statement(sql)\n\ -\n\ -Checks if a string contains a complete SQL statement. Non-standard."); +/*[clinic input] +_sqlite3.enable_shared_cache as pysqlite_enable_shared_cache -static PyObject* module_enable_shared_cache(PyObject* self, PyObject* args, PyObject* - kwargs) + do_enable: int + +Enable or disable shared cache mode for the calling thread. + +Experimental/Non-standard. +[clinic start generated code]*/ + +static PyObject * +pysqlite_enable_shared_cache_impl(PyObject *module, int do_enable) +/*[clinic end generated code: output=259c74eedee1516b input=8400e41bc58b6b24]*/ { - static char *kwlist[] = {"do_enable", NULL}; - int do_enable; int rc; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "i", kwlist, &do_enable)) - { - return NULL; - } - rc = sqlite3_enable_shared_cache(do_enable); if (rc != SQLITE_OK) { @@ -153,22 +159,23 @@ static PyObject* module_enable_shared_cache(PyObject* self, PyObject* args, PyOb } } -PyDoc_STRVAR(module_enable_shared_cache_doc, -"enable_shared_cache(do_enable)\n\ -\n\ -Enable or disable shared cache mode for the calling thread.\n\ -Experimental/Non-standard."); +/*[clinic input] +_sqlite3.register_adapter as pysqlite_register_adapter + + type: object(type='PyTypeObject *') + caster: object + / -static PyObject* module_register_adapter(PyObject* self, PyObject* args) +Registers an adapter with pysqlite's adapter registry. Non-standard. +[clinic start generated code]*/ + +static PyObject * +pysqlite_register_adapter_impl(PyObject *module, PyTypeObject *type, + PyObject *caster) +/*[clinic end generated code: output=a287e8db18e8af23 input=839dad90e2492725]*/ { - PyTypeObject* type; - PyObject* caster; int rc; - if (!PyArg_ParseTuple(args, "OO", &type, &caster)) { - return NULL; - } - /* a basic type is adapted; there's a performance optimization if that's not the case * (99 % of all usages) */ if (type == &PyLong_Type || type == &PyFloat_Type @@ -183,23 +190,25 @@ static PyObject* module_register_adapter(PyObject* self, PyObject* args) Py_RETURN_NONE; } -PyDoc_STRVAR(module_register_adapter_doc, -"register_adapter(type, callable)\n\ -\n\ -Registers an adapter with pysqlite's adapter registry. Non-standard."); +/*[clinic input] +_sqlite3.register_converter as pysqlite_register_converter + + name as orig_name: unicode + converter as callable: object + / -static PyObject* module_register_converter(PyObject* self, PyObject* args) +Registers a converter with pysqlite. Non-standard. +[clinic start generated code]*/ + +static PyObject * +pysqlite_register_converter_impl(PyObject *module, PyObject *orig_name, + PyObject *callable) +/*[clinic end generated code: output=a2f2bfeed7230062 input=e074cf7f4890544f]*/ { - PyObject* orig_name; PyObject* name = NULL; - PyObject* callable; PyObject* retval = NULL; _Py_IDENTIFIER(upper); - if (!PyArg_ParseTuple(args, "UO", &orig_name, &callable)) { - return NULL; - } - /* convert the name to upper case */ name = _PyObject_CallMethodIdNoArgs(orig_name, &PyId_upper); if (!name) { @@ -217,24 +226,42 @@ static PyObject* module_register_converter(PyObject* self, PyObject* args) return retval; } -PyDoc_STRVAR(module_register_converter_doc, -"register_converter(typename, callable)\n\ -\n\ -Registers a converter with pysqlite. Non-standard."); +/*[clinic input] +_sqlite3.enable_callback_tracebacks as pysqlite_enable_callback_trace -static PyObject* enable_callback_tracebacks(PyObject* self, PyObject* args) + enable: int + / + +Enable or disable callback functions throwing errors to stderr. +[clinic start generated code]*/ + +static PyObject * +pysqlite_enable_callback_trace_impl(PyObject *module, int enable) +/*[clinic end generated code: output=4ff1d051c698f194 input=cb79d3581eb77c40]*/ { - if (!PyArg_ParseTuple(args, "i", &_pysqlite_enable_callback_tracebacks)) { - return NULL; - } + _pysqlite_enable_callback_tracebacks = enable; Py_RETURN_NONE; } -PyDoc_STRVAR(enable_callback_tracebacks_doc, -"enable_callback_tracebacks(flag)\n\ -\n\ -Enable or disable callback functions throwing errors to stderr."); +/*[clinic input] +_sqlite3.adapt as pysqlite_adapt + + obj: object + proto: object(c_default='(PyObject*)pysqlite_PrepareProtocolType') = PrepareProtocolType + alt: object = NULL + / + +Adapt given object to given protocol. Non-standard. +[clinic start generated code]*/ + +static PyObject * +pysqlite_adapt_impl(PyObject *module, PyObject *obj, PyObject *proto, + PyObject *alt) +/*[clinic end generated code: output=0c3927c5fcd23dd9 input=a58ab77fb5ae22dd]*/ +{ + return pysqlite_microprotocols_adapt(obj, proto, alt); +} static void converters_init(PyObject* module) { @@ -252,18 +279,12 @@ static void converters_init(PyObject* module) static PyMethodDef module_methods[] = { {"connect", (PyCFunction)(void(*)(void))module_connect, METH_VARARGS | METH_KEYWORDS, module_connect_doc}, - {"complete_statement", (PyCFunction)(void(*)(void))module_complete, - METH_VARARGS | METH_KEYWORDS, module_complete_doc}, - {"enable_shared_cache", (PyCFunction)(void(*)(void))module_enable_shared_cache, - METH_VARARGS | METH_KEYWORDS, module_enable_shared_cache_doc}, - {"register_adapter", (PyCFunction)module_register_adapter, - METH_VARARGS, module_register_adapter_doc}, - {"register_converter", (PyCFunction)module_register_converter, - METH_VARARGS, module_register_converter_doc}, - {"adapt", (PyCFunction)pysqlite_adapt, METH_VARARGS, - pysqlite_adapt_doc}, - {"enable_callback_tracebacks", (PyCFunction)enable_callback_tracebacks, - METH_VARARGS, enable_callback_tracebacks_doc}, + PYSQLITE_ADAPT_METHODDEF + PYSQLITE_COMPLETE_STATEMENT_METHODDEF + PYSQLITE_ENABLE_CALLBACK_TRACE_METHODDEF + PYSQLITE_ENABLE_SHARED_CACHE_METHODDEF + PYSQLITE_REGISTER_ADAPTER_METHODDEF + PYSQLITE_REGISTER_CONVERTER_METHODDEF {NULL, NULL} };

1 0

bpo-42198: Document __new__ for types.GenericAlias (GH-23039)
by miss-islington Oct. 31, 2020

Oct. 31, 2020

https://github.com/python/cpython/commit/bcbf758476c1148993ddf4b54d3f6169b9… commit: bcbf758476c1148993ddf4b54d3f6169b973ee1c branch: master author: kj <28750310+Fidget-Spinner(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2020-10-30T23:02:38-07:00 summary: bpo-42198: Document __new__ for types.GenericAlias (GH-23039) files: M Doc/library/stdtypes.rst M Doc/library/types.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 3fd94ea1bd310..09477e63786c5 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4793,7 +4793,8 @@ The ``GenericAlias`` object acts as a proxy for :term:`generic types of a generic which provides the types for container elements. The user-exposed type for the ``GenericAlias`` object can be accessed from -:data:`types.GenericAlias` and used for :func:`isinstance` checks. +:class:`types.GenericAlias` and used for :func:`isinstance` checks. It can +also be used to create ``GenericAlias`` objects directly. .. describe:: T[X, Y, ...] diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 00720559d0a4a..ad40a9fbf3273 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -262,11 +262,22 @@ Standard names are defined for the following types: .. versionadded:: 3.10 -.. data:: GenericAlias +.. class:: GenericAlias(t_origin, t_args) The type of :ref:`parameterized generics <types-genericalias>` such as ``list[int]``. + ``t_origin`` should be a non-parameterized generic class, such as ``list``, + ``tuple`` or ``dict``. ``t_args`` should be a :class:`tuple` (possibly of + length 1) of types which parameterize ``t_origin``:: + + >>> from types import GenericAlias + + >>> list[int] == GenericAlias(list, (int,)) + True + >>> dict[str, int] == GenericAlias(dict, (str, int)) + True + .. versionadded:: 3.9 .. data:: Union

1 0

Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)"
by methane Oct. 30, 2020

Oct. 30, 2020

https://github.com/python/cpython/commit/43ca084c88d1e46a44199f347c72e26db8… commit: 43ca084c88d1e46a44199f347c72e26db84903c9 branch: master author: Inada Naoki <songofacandy(a)gmail.com> committer: methane <songofacandy(a)gmail.com> date: 2020-10-31T11:15:38+09:00 summary: Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)" `_RandomNameSequence` is not true singleton so using `os.register_at_fork` doesn't make sense unlike `random._inst`. This reverts commit 8e409cebad42032bb7d0f2cadd8b1e36081d98af. files: D Misc/NEWS.d/next/Library/2020-10-27-00-42-09.bpo-42160.eiLOCi.rst M Lib/tempfile.py M Lib/test/test_tempfile.py diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 1bc5c71fd0349..770f72c25295c 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -129,22 +129,24 @@ class _RandomNameSequence: _RandomNameSequence is an iterator.""" - def __init__(self, characters="abcdefghijklmnopqrstuvwxyz0123456789_", length=8, rng=None): - if rng is None: - rng = _Random() - if hasattr(_os, "fork"): - # prevent same state after fork - _os.register_at_fork(after_in_child=rng.seed) - self.rng = rng - self.characters = characters - self.length = length + characters = "abcdefghijklmnopqrstuvwxyz0123456789_" + + @property + def rng(self): + cur_pid = _os.getpid() + if cur_pid != getattr(self, '_rng_pid', None): + self._rng = _Random() + self._rng_pid = cur_pid + return self._rng def __iter__(self): return self def __next__(self): c = self.characters - return ''.join(self.rng.choices(c, k=self.length)) + choose = self.rng.choice + letters = [choose(c) for dummy in range(8)] + return ''.join(letters) def _candidate_tempdir_list(): """Generate a list of candidate temporary directories which diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 77d710efaf107..8ace883d74bb2 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -153,8 +153,8 @@ def setUp(self): self.r = tempfile._RandomNameSequence() super().setUp() - def test_get_eight_char_str(self): - # _RandomNameSequence returns a eight-character string + def test_get_six_char_str(self): + # _RandomNameSequence returns a six-character string s = next(self.r) self.nameCheck(s, '', '', '') diff --git a/Misc/NEWS.d/next/Library/2020-10-27-00-42-09.bpo-42160.eiLOCi.rst b/Misc/NEWS.d/next/Library/2020-10-27-00-42-09.bpo-42160.eiLOCi.rst deleted file mode 100644 index c5f3091283a87..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-27-00-42-09.bpo-42160.eiLOCi.rst +++ /dev/null @@ -1 +0,0 @@ -Replaced pid check in ``tempfile._RandomNameSequence`` with ``os.register_at_fork`` to reduce overhead.

1 0

[3.9] bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) (GH-23051)
by pablogsal Oct. 30, 2020

Oct. 30, 2020

https://github.com/python/cpython/commit/ddcd57e3ea75ab0ad370bbaaa6b76338ed… commit: ddcd57e3ea75ab0ad370bbaaa6b76338edbca395 branch: 3.9 author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2020-10-31T00:40:42Z summary: [3.9] bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) (GH-23051) (cherry picked from commit 06f8c3328dcd81c84d1ee2b3a57b5381dcb38482) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-10-30-22-16-30.bpo-42214.lXskM_.rst M Grammar/python.gram M Lib/test/test_syntax.py M Parser/pegen/parse.c M Parser/pegen/pegen.c M Parser/pegen/pegen.h diff --git a/Grammar/python.gram b/Grammar/python.gram index b709d3d5c3214..60eeb367e95ba 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -413,7 +413,7 @@ compare_op_bitwise_or_pair[CmpopExprPair*]: | is_bitwise_or eq_bitwise_or[CmpopExprPair*]: '==' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Eq, a) } noteq_bitwise_or[CmpopExprPair*]: - | (tok='!=' {_PyPegen_check_barry_as_flufl(p) ? NULL : tok}) a=bitwise_or {_PyPegen_cmpop_expr_pair(p, NotEq, a) } + | (tok='!=' { _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok}) a=bitwise_or {_PyPegen_cmpop_expr_pair(p, NotEq, a) } lte_bitwise_or[CmpopExprPair*]: '<=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, LtE, a) } lt_bitwise_or[CmpopExprPair*]: '<' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Lt, a) } gte_bitwise_or[CmpopExprPair*]: '>=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, GtE, a) } diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index b0527e6cd85ba..a95992d869e5e 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -947,6 +947,23 @@ def test_empty_line_after_linecont(self): self.fail("Empty line after a line continuation character is valid.") + def test_barry_as_flufl_with_syntax_errors(self): + # The "barry_as_flufl" rule can produce some "bugs-at-a-distance" if + # is reading the wrong token in the presence of syntax errors later + # in the file. See bpo-42214 for more information. + code = """ +def func1(): + if a != b: + raise ValueError + +def func2(): + try + return 1 + finally: + pass +""" + self._check_error(code, "invalid syntax") + def test_main(): support.run_unittest(SyntaxTestCase) from test import test_syntax diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-30-22-16-30.bpo-42214.lXskM_.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-30-22-16-30.bpo-42214.lXskM_.rst new file mode 100644 index 0000000000000..3f85bbe83901a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-30-22-16-30.bpo-42214.lXskM_.rst @@ -0,0 +1,2 @@ +Fixed a possible crash in the PEG parser when checking for the '!=' token in +the ``barry_as_flufl`` rule. Patch by Pablo Galindo. diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c index 48a2443474684..bae9463e274a3 100644 --- a/Parser/pegen/parse.c +++ b/Parser/pegen/parse.c @@ -21307,7 +21307,7 @@ _tmp_93_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ _tmp_93[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'!='")); - _res = _PyPegen_check_barry_as_flufl ( p ) ? NULL : tok; + _res = _PyPegen_check_barry_as_flufl ( p , tok ) ? NULL : tok; if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; D(p->level--); diff --git a/Parser/pegen/pegen.c b/Parser/pegen/pegen.c index 78891af825007..4e742a5ec7175 100644 --- a/Parser/pegen/pegen.c +++ b/Parser/pegen/pegen.c @@ -62,8 +62,7 @@ init_normalization(Parser *p) /* Checks if the NOTEQUAL token is valid given the current parser flags 0 indicates success and nonzero indicates failure (an exception may be set) */ int -_PyPegen_check_barry_as_flufl(Parser *p) { - Token *t = p->tokens[p->fill - 1]; +_PyPegen_check_barry_as_flufl(Parser *p, Token* t) { assert(t->bytes != NULL); assert(t->type == NOTEQUAL); diff --git a/Parser/pegen/pegen.h b/Parser/pegen/pegen.h index 2fea84fd22ab4..a2f524a597d23 100644 --- a/Parser/pegen/pegen.h +++ b/Parser/pegen/pegen.h @@ -263,7 +263,7 @@ expr_ty _PyPegen_collect_call_seqs(Parser *, asdl_seq *, asdl_seq *, int end_col_offset, PyArena *arena); expr_ty _PyPegen_concatenate_strings(Parser *p, asdl_seq *); asdl_seq *_PyPegen_join_sequences(Parser *, asdl_seq *, asdl_seq *); -int _PyPegen_check_barry_as_flufl(Parser *); +int _PyPegen_check_barry_as_flufl(Parser *, Token *); mod_ty _PyPegen_make_module(Parser *, asdl_seq *); // Error reporting helpers

1 0

bpo-42208: Add _locale._get_locale_encoding() (GH-23052)
by vstinner Oct. 30, 2020

Oct. 30, 2020

https://github.com/python/cpython/commit/b62bdf71ea0cd52041d49691d8ae3dc645… commit: b62bdf71ea0cd52041d49691d8ae3dc645bd48e1 branch: master author: Victor Stinner <vstinner(a)python.org> committer: vstinner <vstinner(a)python.org> date: 2020-10-31T01:32:11+01:00 summary: bpo-42208: Add _locale._get_locale_encoding() (GH-23052) * Add a new _locale._get_locale_encoding() function to get the current locale encoding. * Modify locale.getpreferredencoding() to use it. * Remove the _bootlocale module. files: D Lib/_bootlocale.py M Lib/locale.py M Lib/test/test_mimetypes.py M Modules/_localemodule.c M Modules/clinic/_localemodule.c.h M PCbuild/lib.pyproj diff --git a/Lib/_bootlocale.py b/Lib/_bootlocale.py deleted file mode 100644 index 3273a3b42252b..0000000000000 --- a/Lib/_bootlocale.py +++ /dev/null @@ -1,46 +0,0 @@ -"""A minimal subset of the locale module used at interpreter startup -(imported by the _io module), in order to reduce startup time. - -Don't import directly from third-party code; use the `locale` module instead! -""" - -import sys -import _locale - -if sys.platform.startswith("win"): - def getpreferredencoding(do_setlocale=True): - if sys.flags.utf8_mode: - return 'UTF-8' - return _locale._getdefaultlocale()[1] -else: - try: - _locale.CODESET - except AttributeError: - if hasattr(sys, 'getandroidapilevel'): - # On Android langinfo.h and CODESET are missing, and UTF-8 is - # always used in mbstowcs() and wcstombs(). - def getpreferredencoding(do_setlocale=True): - return 'UTF-8' - else: - def getpreferredencoding(do_setlocale=True): - if sys.flags.utf8_mode: - return 'UTF-8' - # This path for legacy systems needs the more complex - # getdefaultlocale() function, import the full locale module. - import locale - return locale.getpreferredencoding(do_setlocale) - else: - def getpreferredencoding(do_setlocale=True): - assert not do_setlocale - if sys.flags.utf8_mode: - return 'UTF-8' - result = _locale.nl_langinfo(_locale.CODESET) - if not result and sys.platform == 'darwin': - # nl_langinfo can return an empty string - # when the setting has an invalid value. - # Default to UTF-8 in that case because - # UTF-8 is the default charset on OSX and - # returning nothing will crash the - # interpreter. - result = 'UTF-8' - return result diff --git a/Lib/locale.py b/Lib/locale.py index 1a4e9f694f309..ee841e8b8655e 100644 --- a/Lib/locale.py +++ b/Lib/locale.py @@ -619,53 +619,49 @@ def resetlocale(category=LC_ALL): """ _setlocale(category, _build_localename(getdefaultlocale())) -if sys.platform.startswith("win"): - # On Win32, this will return the ANSI code page - def getpreferredencoding(do_setlocale = True): - """Return the charset that the user is likely using.""" + +try: + from _locale import _get_locale_encoding +except ImportError: + def _get_locale_encoding(): + if hasattr(sys, 'getandroidapilevel'): + # On Android langinfo.h and CODESET are missing, and UTF-8 is + # always used in mbstowcs() and wcstombs(). + return 'UTF-8' if sys.flags.utf8_mode: return 'UTF-8' - import _bootlocale - return _bootlocale.getpreferredencoding(False) + encoding = getdefaultlocale()[1] + if encoding is None: + # LANG not set, default conservatively to ASCII + encoding = 'ascii' + return encoding + +try: + CODESET +except NameError: + def getpreferredencoding(do_setlocale=True): + """Return the charset that the user is likely using.""" + return _get_locale_encoding() else: # On Unix, if CODESET is available, use that. - try: - CODESET - except NameError: - if hasattr(sys, 'getandroidapilevel'): - # On Android langinfo.h and CODESET are missing, and UTF-8 is - # always used in mbstowcs() and wcstombs(). - def getpreferredencoding(do_setlocale = True): - return 'UTF-8' - else: - # Fall back to parsing environment variables :-( - def getpreferredencoding(do_setlocale = True): - """Return the charset that the user is likely using, - by looking at environment variables.""" - if sys.flags.utf8_mode: - return 'UTF-8' - res = getdefaultlocale()[1] - if res is None: - # LANG not set, default conservatively to ASCII - res = 'ascii' - return res - else: - def getpreferredencoding(do_setlocale = True): - """Return the charset that the user is likely using, - according to the system configuration.""" - if sys.flags.utf8_mode: - return 'UTF-8' - import _bootlocale - if do_setlocale: - oldloc = setlocale(LC_CTYPE) - try: - setlocale(LC_CTYPE, "") - except Error: - pass - result = _bootlocale.getpreferredencoding(False) - if do_setlocale: - setlocale(LC_CTYPE, oldloc) - return result + def getpreferredencoding(do_setlocale=True): + """Return the charset that the user is likely using, + according to the system configuration.""" + if sys.flags.utf8_mode: + return 'UTF-8' + + if not do_setlocale: + return _get_locale_encoding() + + old_loc = setlocale(LC_CTYPE) + try: + try: + setlocale(LC_CTYPE, "") + except Error: + pass + return _get_locale_encoding() + finally: + setlocale(LC_CTYPE, old_loc) ### Database diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index ddeae38e1372f..d63f6b66e10c9 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -3,7 +3,7 @@ import mimetypes import pathlib import sys -import unittest +import unittest.mock from test import support from test.support import os_helper @@ -71,14 +71,14 @@ def test_read_mime_types(self): # bpo-41048: read_mime_types should read the rule file with 'utf-8' encoding. # Not with locale encoding. _bootlocale has been imported because io.open(...) # uses it. - with os_helper.temp_dir() as directory: - data = "application/no-mans-land Fran\u00E7ais" - file = pathlib.Path(directory, "sample.mimetype") - file.write_text(data, encoding='utf-8') - import _bootlocale - with support.swap_attr(_bootlocale, 'getpreferredencoding', lambda do_setlocale=True: 'ASCII'): - mime_dict = mimetypes.read_mime_types(file) - eq(mime_dict[".Français"], "application/no-mans-land") + data = "application/no-mans-land Fran\u00E7ais" + filename = "filename" + fp = io.StringIO(data) + with unittest.mock.patch.object(mimetypes, 'open', + return_value=fp) as mock_open: + mime_dict = mimetypes.read_mime_types(filename) + mock_open.assert_called_with(filename, encoding='utf-8') + eq(mime_dict[".Français"], "application/no-mans-land") def test_non_standard_types(self): eq = self.assertEqual diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c index 9c7ce876e4059..359deb7544043 100644 --- a/Modules/_localemodule.c +++ b/Modules/_localemodule.c @@ -768,9 +768,24 @@ _locale_bind_textdomain_codeset_impl(PyObject *module, const char *domain, } Py_RETURN_NONE; } -#endif +#endif // HAVE_BIND_TEXTDOMAIN_CODESET + +#endif // HAVE_LIBINTL_H + + +/*[clinic input] +_locale._get_locale_encoding + +Get the current locale encoding. +[clinic start generated code]*/ + +static PyObject * +_locale__get_locale_encoding_impl(PyObject *module) +/*[clinic end generated code: output=e8e2f6f6f184591a input=513d9961d2f45c76]*/ +{ + return _Py_GetLocaleEncoding(); +} -#endif static struct PyMethodDef PyLocale_Methods[] = { _LOCALE_SETLOCALE_METHODDEF @@ -797,6 +812,7 @@ static struct PyMethodDef PyLocale_Methods[] = { _LOCALE_BIND_TEXTDOMAIN_CODESET_METHODDEF #endif #endif + _LOCALE__GET_LOCALE_ENCODING_METHODDEF {NULL, NULL} }; diff --git a/Modules/clinic/_localemodule.c.h b/Modules/clinic/_localemodule.c.h index 5d1db3ece796d..703d034c32e80 100644 --- a/Modules/clinic/_localemodule.c.h +++ b/Modules/clinic/_localemodule.c.h @@ -545,6 +545,24 @@ _locale_bind_textdomain_codeset(PyObject *module, PyObject *const *args, Py_ssiz #endif /* defined(HAVE_LIBINTL_H) && defined(HAVE_BIND_TEXTDOMAIN_CODESET) */ +PyDoc_STRVAR(_locale__get_locale_encoding__doc__, +"_get_locale_encoding($module, /)\n" +"--\n" +"\n" +"Get the current locale encoding."); + +#define _LOCALE__GET_LOCALE_ENCODING_METHODDEF \ + {"_get_locale_encoding", (PyCFunction)_locale__get_locale_encoding, METH_NOARGS, _locale__get_locale_encoding__doc__}, + +static PyObject * +_locale__get_locale_encoding_impl(PyObject *module); + +static PyObject * +_locale__get_locale_encoding(PyObject *module, PyObject *Py_UNUSED(ignored)) +{ + return _locale__get_locale_encoding_impl(module); +} + #ifndef _LOCALE_STRCOLL_METHODDEF #define _LOCALE_STRCOLL_METHODDEF #endif /* !defined(_LOCALE_STRCOLL_METHODDEF) */ @@ -584,4 +602,4 @@ _locale_bind_textdomain_codeset(PyObject *module, PyObject *const *args, Py_ssiz #ifndef _LOCALE_BIND_TEXTDOMAIN_CODESET_METHODDEF #define _LOCALE_BIND_TEXTDOMAIN_CODESET_METHODDEF #endif /* !defined(_LOCALE_BIND_TEXTDOMAIN_CODESET_METHODDEF) */ -/*[clinic end generated code: output=fe944779cd572d8e input=a9049054013a1b77]*/ +/*[clinic end generated code: output=cd703c8a3a75fcf4 input=a9049054013a1b77]*/ diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index f0c51edb9d1ca..a15165d92ef12 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -1572,7 +1572,6 @@ <Compile Include="zoneinfo\__init__.py" /> <Compile Include="zoneinfo\_tzpath.py" /> <Compile Include="zoneinfo\_zoneinfo.py" /> - <Compile Include="_bootlocale.py" /> <Compile Include="_collections_abc.py" /> <Compile Include="_compat_pickle.py" /> <Compile Include="_compression.py" />

1 0