Mailman 3 December 2017 - Python-checkins

Don't hide unexpected errors in PyErr_WarnExplicitObject(). (#4585)
by Serhiy Storchaka 01 Dec '17

01 Dec '17

https://github.com/python/cpython/commit/a561862048555d555fa4850eaf832ae547… commit: a561862048555d555fa4850eaf832ae5474c7e1f branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-12-01T08:40:23+02:00 summary: Don't hide unexpected errors in PyErr_WarnExplicitObject(). (#4585) files: M Python/ast.c diff --git a/Python/ast.c b/Python/ast.c index e44ce516617..e2092f0f854 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -4160,18 +4160,19 @@ warn_invalid_escape_sequence(struct compiling *c, const node *n, } if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, msg, c->c_filename, LINENO(n), - NULL, NULL) < 0 && - PyErr_ExceptionMatches(PyExc_DeprecationWarning)) + NULL, NULL) < 0) { - const char *s; + if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) { + const char *s; - /* Replace the DeprecationWarning exception with a SyntaxError - to get a more accurate error report */ - PyErr_Clear(); + /* Replace the DeprecationWarning exception with a SyntaxError + to get a more accurate error report */ + PyErr_Clear(); - s = PyUnicode_AsUTF8(msg); - if (s != NULL) { - ast_error(c, n, s); + s = PyUnicode_AsUTF8(msg); + if (s != NULL) { + ast_error(c, n, s); + } } Py_DECREF(msg); return -1;

1 0

bpo-32186: Release the GIL during lseek and fstat (GH-4652) (#4661)
by Victor Stinner 01 Dec '17

01 Dec '17

https://github.com/python/cpython/commit/8bcd41040a5f1f9b48a86d0e21f196e4b1… commit: 8bcd41040a5f1f9b48a86d0e21f196e4b1f90e4b branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Victor Stinner <victor.stinner(a)gmail.com> date: 2017-12-01T07:26:31+01:00 summary: bpo-32186: Release the GIL during lseek and fstat (GH-4652) (#4661) In _io_FileIO_readall_impl(), lseek() and _Py_fstat_noraise() were called without releasing the GIL. This can cause all threads to hang for unlimited time when calling FileIO.read() and the NFS server is not accessible. (cherry picked from commit 6a89481680b921e7b317c29877bdda9a6031e5ad) files: A Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst M Modules/_io/fileio.c diff --git a/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst b/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst new file mode 100644 index 00000000000..ea696c6098a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst @@ -0,0 +1,3 @@ +io.FileIO.readall() and io.FileIO.read() now release the GIL when +getting the file size. Fixed hang of all threads with inaccessible NFS +server. Patch by Nir Soffer. diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index 918fa577758..52cbb94b24b 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -691,10 +691,12 @@ _io_FileIO_readall_impl(fileio *self) Py_ssize_t bytes_read = 0; Py_ssize_t n; size_t bufsize; + int fstat_result; if (self->fd < 0) return err_closed(); + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH #ifdef MS_WINDOWS pos = _lseeki64(self->fd, 0L, SEEK_CUR); @@ -702,8 +704,10 @@ _io_FileIO_readall_impl(fileio *self) pos = lseek(self->fd, 0L, SEEK_CUR); #endif _Py_END_SUPPRESS_IPH + fstat_result = _Py_fstat_noraise(self->fd, &status); + Py_END_ALLOW_THREADS - if (_Py_fstat_noraise(self->fd, &status) == 0) + if (fstat_result == 0) end = status.st_size; else end = (Py_off_t)-1;

1 0

bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)
by Nick Coghlan 01 Dec '17

01 Dec '17

https://github.com/python/cpython/commit/73a7e9b10b2ec9636e3c6396cf7b3695f8… commit: 73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856 branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: Nick Coghlan <ncoghlan(a)gmail.com> date: 2017-12-01T14:54:17+10:00 summary: bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579) The current behaviour of yield expressions inside comprehensions and generator expressions is essentially an accident of implementation - it arises implicitly from the way the compiler handles yield expressions inside nested functions and generators. Since the current behaviour wasn't deliberately designed, and is inherently confusing, we're deprecating it, with no current plans to reintroduce it. Instead, our advice will be to use a named nested generator definition for cases where this behaviour is desired. files: A Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst M Doc/reference/expressions.rst M Doc/whatsnew/3.7.rst M Lib/test/test_generators.py M Lib/test/test_grammar.py M Python/symtable.c diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 1cff8a52df9..fb92ad0f07c 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -183,8 +183,21 @@ by considering each of the :keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. -Note that the comprehension is executed in a separate scope, so names assigned -to in the target list don't "leak" into the enclosing scope. +However, aside from the iterable expression in the leftmost :keyword:`for` clause, +the comprehension is executed in a separate implicitly nested scope. This ensures +that names assigned to in the target list don't "leak" into the enclosing scope. + +The iterable expression in the leftmost :keyword:`for` clause is evaluated +directly in the enclosing scope and then passed as an argument to the implictly +nested scope. Subsequent :keyword:`for` clauses and any filter condition in the +leftmost :keyword:`for` clause cannot be evaluated in the enclosing scope as +they may depend on the values obtained from the leftmost iterable. For example: +``[x*y for x in range(10) for y in range(x, x+10)]``. + +To ensure the comprehension always results in a container of the appropriate +type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly +nested scope (in Python 3.7, such expressions emit :exc:`DeprecationWarning` +when compiled, in Python 3.8+ they will emit :exc:`SyntaxError`). Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for` clause may be used to iterate over a :term:`asynchronous iterator`. @@ -198,6 +211,13 @@ or :keyword:`await` expressions it is called an suspend the execution of the coroutine function in which it appears. See also :pep:`530`. +.. versionadded:: 3.6 + Asynchronous comprehensions were introduced. + +.. deprecated:: 3.7 + ``yield`` and ``yield from`` deprecated in the implicitly nested scope. + + .. _lists: List displays @@ -316,27 +336,42 @@ brackets or curly braces. Variables used in the generator expression are evaluated lazily when the :meth:`~generator.__next__` method is called for the generator object (in the same -fashion as normal generators). However, the leftmost :keyword:`for` clause is -immediately evaluated, so that an error produced by it can be seen before any -other possible error in the code that handles the generator expression. -Subsequent :keyword:`for` clauses cannot be evaluated immediately since they -may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in -range(10) for y in bar(x))``. +fashion as normal generators). However, the iterable expression in the +leftmost :keyword:`for` clause is immediately evaluated, so that an error +produced by it will be emitted at the point where the generator expression +is defined, rather than at the point where the first value is retrieved. +Subsequent :keyword:`for` clauses and any filter condition in the leftmost +:keyword:`for` clause cannot be evaluated in the enclosing scope as they may +depend on the values obtained from the leftmost iterable. For example: +``(x*y for x in range(10) for y in range(x, x+10))``. The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for details. +To avoid interfering with the expected operation of the generator expression +itself, ``yield`` and ``yield from`` expressions are prohibited in the +implicitly defined generator (in Python 3.7, such expressions emit +:exc:`DeprecationWarning` when compiled, in Python 3.8+ they will emit +:exc:`SyntaxError`). + If a generator expression contains either :keyword:`async for` clauses or :keyword:`await` expressions it is called an :dfn:`asynchronous generator expression`. An asynchronous generator expression returns a new asynchronous generator object, which is an asynchronous iterator (see :ref:`async-iterators`). +.. versionadded:: 3.6 + Asynchronous generator expressions were introduced. + .. versionchanged:: 3.7 Prior to Python 3.7, asynchronous generator expressions could only appear in :keyword:`async def` coroutines. Starting with 3.7, any function can use asynchronous generator expressions. +.. deprecated:: 3.7 + ``yield`` and ``yield from`` deprecated in the implicitly nested scope. + + .. _yieldexpr: Yield expressions @@ -364,6 +399,16 @@ coroutine function to be an asynchronous generator. For example:: async def agen(): # defines an asynchronous generator function (PEP 525) yield 123 +Due to their side effects on the containing scope, ``yield`` expressions +are not permitted as part of the implicitly defined scopes used to +implement comprehensions and generator expressions (in Python 3.7, such +expressions emit :exc:`DeprecationWarning` when compiled, in Python 3.8+ +they will emit :exc:`SyntaxError`).. + +.. deprecated:: 3.7 + Yield expressions deprecated in the implicitly nested scopes used to + implement comprehensions and generator expressions. + Generator functions are described below, while asynchronous generator functions are described separately in section :ref:`asynchronous-generator-functions`. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 6545a18a991..b6dad4eab6b 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -570,6 +570,18 @@ Other CPython Implementation Changes Deprecated ========== +* Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated + in comprehensions and generator expressions (aside from the iterable expression + in the leftmost :keyword:`for` clause). This ensures that comprehensions + always immediately return a container of the appropriate type (rather than + potentially returning a :term:`generator iterator` object), while generator + expressions won't attempt to interleave their implicit output with the output + from any explicit yield expressions. + + In Python 3.7, such expressions emit :exc:`DeprecationWarning` when compiled, + in Python 3.8+ they will emit :exc:`SyntaxError`. (Contributed by Serhiy + Storchaka in :issue:`10544`.) + - Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with a macro if ``Py_LIMITED_API`` is not set or set to the value between ``0x03050400`` and ``0x03060000`` (not including) or ``0x03060100`` or diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index 7eac9d076be..f88c762581d 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -1830,13 +1830,7 @@ def printsolution(self, x): [None] - -An obscene abuse of a yield expression within a generator expression: - ->>> list((yield 21) for i in range(4)) -[21, None, 21, None, 21, None, 21, None] - -And a more sane, but still weird usage: +Yield is allowed only in the outermost iterable in generator expression: >>> def f(): list(i for i in [(yield 26)]) >>> type(f()) @@ -2106,10 +2100,6 @@ def printsolution(self, x): >>> type(f()) <class 'generator'> ->>> def f(): x=(i for i in (yield) if (yield)) ->>> type(f()) -<class 'generator'> - >>> def f(d): d[(yield "a")] = d[(yield "b")] = 27 >>> data = [1,2] >>> g = f(data) diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index 65e26bfd383..823315f8cd0 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -841,6 +841,41 @@ def g(): f((yield from ()), 1) # Check annotation refleak on SyntaxError check_syntax_error(self, "def g(a:(yield)): pass") + def test_yield_in_comprehensions(self): + # Check yield in comprehensions + def g(): [x for x in [(yield 1)]] + def g(): [x for x in [(yield from ())]] + + def check(code, warntext): + with self.assertWarnsRegex(DeprecationWarning, warntext): + compile(code, '<test string>', 'exec') + import warnings + with warnings.catch_warnings(): + warnings.filterwarnings('error', category=DeprecationWarning) + with self.assertRaisesRegex(SyntaxError, warntext): + compile(code, '<test string>', 'exec') + + check("def g(): [(yield x) for x in ()]", + "'yield' inside list comprehension") + check("def g(): [x for x in () if not (yield x)]", + "'yield' inside list comprehension") + check("def g(): [y for x in () for y in [(yield x)]]", + "'yield' inside list comprehension") + check("def g(): {(yield x) for x in ()}", + "'yield' inside set comprehension") + check("def g(): {(yield x): x for x in ()}", + "'yield' inside dict comprehension") + check("def g(): {x: (yield x) for x in ()}", + "'yield' inside dict comprehension") + check("def g(): ((yield x) for x in ())", + "'yield' inside generator expression") + check("def g(): [(yield from x) for x in ()]", + "'yield' inside list comprehension") + check("class C: [(yield x) for x in ()]", + "'yield' inside list comprehension") + check("[(yield x) for x in ()]", + "'yield' inside list comprehension") + def test_raise(self): # 'raise' test [',' test] try: raise RuntimeError('just testing') diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst new file mode 100644 index 00000000000..555c94ebaee --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst @@ -0,0 +1,3 @@ +Yield expressions are now deprecated in comprehensions and generator +expressions. They are still permitted in the definition of the outermost +iterable, as that is evaluated directly in the enclosing scope. diff --git a/Python/symtable.c b/Python/symtable.c index 55815c91cc9..bbac25cf376 100644 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -1734,7 +1734,6 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e, e->lineno, e->col_offset)) { return 0; } - st->st_cur->ste_generator = is_generator; if (outermost->is_async) { st->st_cur->ste_coroutine = 1; } @@ -1754,6 +1753,36 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e, if (value) VISIT(st, expr, value); VISIT(st, expr, elt); + if (st->st_cur->ste_generator) { + PyObject *msg = PyUnicode_FromString( + (e->kind == ListComp_kind) ? "'yield' inside list comprehension" : + (e->kind == SetComp_kind) ? "'yield' inside set comprehension" : + (e->kind == DictComp_kind) ? "'yield' inside dict comprehension" : + "'yield' inside generator expression"); + if (msg == NULL) { + symtable_exit_block(st, (void *)e); + return 0; + } + if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, + msg, st->st_filename, st->st_cur->ste_lineno, + NULL, NULL) == -1) + { + if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) { + /* Replace the DeprecationWarning exception with a SyntaxError + to get a more accurate error report */ + PyErr_Clear(); + PyErr_SetObject(PyExc_SyntaxError, msg); + PyErr_SyntaxLocationObject(st->st_filename, + st->st_cur->ste_lineno, + st->st_cur->ste_col_offset); + } + Py_DECREF(msg); + symtable_exit_block(st, (void *)e); + return 0; + } + Py_DECREF(msg); + } + st->st_cur->ste_generator |= is_generator; return symtable_exit_block(st, (void *)e); }

1 0

[2 down, 63 flat] Results for Python (master branch) 2017-11-30
by lp_benchmark_robot＠intel.com 01 Dec '17

01 Dec '17

Results for project python/master, build date: 2017-11-30 03:02:54-08:00. - commit: 5e3806f - previous commit: ef83806 - revision date: 2017-11-30 11:40:24+01:00 - environment: Broadwell-EP - cpu: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz 2x22 cores, stepping 1, LLC 55 MB - mem: 128 GB - os: Ubuntu 16.04.2 LTS - kernel: 4.4.0-62-generic x86_64 GNU/Linux Baseline results were generated using release v3.6.0, with hash 5c4568a from 2016-12-22 23:38:47+00:00. +-----+------------------------+--------+------------+------------+------------+ | | |relative|change since|change since|current rev | | | benchmark|std_dev*| last run | baseline |run with PGO| +-----+------------------------+--------+------------+------------+------------+ | :-| | 2to3| 0.916% | -0.409% | +5.102% | +8.082% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method| 0.762% | +1.685% | +21.474% | +12.290% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method_slots| 1.967% | +0.768% | +21.960% | +12.611% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method_unknown| 1.098% | +2.416% | +20.705% | +10.536% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_simple| 3.488% | -1.404% | +5.459% | +10.667% | +-----+------------------------+--------+------------+------------+------------+ | :-| | chameleon| 1.589% | -2.449% | +8.309% | +13.216% | +-----+------------------------+--------+------------+------------+------------+ | :-| | chaos| 1.147% | -1.390% | +6.055% | +12.058% | +-----+------------------------+--------+------------+------------+------------+ | :-| | crypto_pyaes| 0.442% | -0.222% | +1.015% | +9.049% | +-----+------------------------+--------+------------+------------+------------+ | :-| | deltablue| 3.505% | -2.362% | +6.036% | +21.181% | +-----+------------------------+--------+------------+------------+------------+ | :-| | django_template| 3.681% | +3.133% | +14.618% | +11.245% | +-----+------------------------+--------+------------+------------+------------+ | :-| | dulwich_log| 0.940% | +1.083% | +5.205% | +6.847% | +-----+------------------------+--------+------------+------------+------------+ | :-| | fannkuch| 0.548% | -0.106% | +5.765% | +3.798% | +-----+------------------------+--------+------------+------------+------------+ | :-| | float| 0.661% | -0.725% | +2.008% | +8.320% | +-----+------------------------+--------+------------+------------+------------+ | :-| | genshi_text| 1.114% | +0.616% | +13.703% | +8.743% | +-----+------------------------+--------+------------+------------+------------+ | :-| | genshi_xml| 1.665% | -0.044% | +8.726% | +10.790% | +-----+------------------------+--------+------------+------------+------------+ | :-| | go| 0.853% | +1.106% | +5.703% | +12.986% | +-----+------------------------+--------+------------+------------+------------+ | :-| | hexiom| 0.674% | -0.203% | +10.836% | +11.775% | +-----+------------------------+--------+------------+------------+------------+ | :-| | html5lib| 2.813% | +0.794% | +9.502% | +7.087% | +-----+------------------------+--------+------------+------------+------------+ | :-| | json_dumps| 1.728% | +0.680% | +2.876% | +10.346% | +-----+------------------------+--------+------------+------------+------------+ | :-| | json_loads| 1.318% | -0.288% | -0.491% | +13.395% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_format| 1.862% | -0.007% | +7.453% | +11.248% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_silent| 2.332% | +1.224% | +47.578% | +11.560% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_simple| 1.679% | +0.637% | +7.466% | +12.927% | +-----+------------------------+--------+------------+------------+------------+ | :-| | mako| 0.546% | +0.212% | +17.008% | +14.204% | +-----+------------------------+--------+------------+------------+------------+ | :-| | mdp| 2.636% | -1.214% | +7.639% | +9.745% | +-----+------------------------+--------+------------+------------+------------+ | :-| | meteor_contest| 1.359% | -3.370% | +3.221% | +6.219% | +-----+------------------------+--------+------------+------------+------------+ | :-| | nbody| 0.407% | +0.426% | +0.587% | +0.566% | +-----+------------------------+--------+------------+------------+------------+ | :-| | nqueens| 0.499% | -2.089% | +3.861% | +9.438% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pathlib| 1.527% | +1.535% | +5.113% | +10.219% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle| 0.919% | +2.065% | +1.811% | +21.245% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_dict| 0.320% | -0.477% | -2.985% | +26.985% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_list| 0.695% | +0.605% | +1.587% | +23.852% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_pure_python| 3.164% | +1.713% | +12.074% | +11.079% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pidigits| 0.153% | -0.355% | +0.097% | +10.284% | +-----+------------------------+--------+------------+------------+------------+ | :-| | python_startup| 0.133% | +0.188% | +10.815% | +4.906% | +-----+------------------------+--------+------------+------------+------------+ | :-| | python_startup_no_site| 0.102% | +0.260% | +2.140% | +5.017% | +-----+------------------------+--------+------------+------------+------------+ | :-| | raytrace| 1.031% | -0.932% | +10.093% | +13.697% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_compile| 5.984% | +2.723% | +1.123% | +9.618% | +-----+------------------------+--------+------------+------------+------------+ | :-( | regex_dna| 0.222% | -2.184% | -1.323% | +11.633% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_effbot| 2.611% | -1.148% | -6.653% | +10.110% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_v8| 1.740% | -0.280% | +4.988% | +9.628% | +-----+------------------------+--------+------------+------------+------------+ | :-| | richards| 1.284% | -0.758% | +7.778% | +15.737% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_fft| 3.399% | +0.106% | -1.663% | +4.744% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_lu| 1.653% | +1.165% | +27.828% | +8.128% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_monte_carlo| 2.215% | +0.556% | +5.018% | +6.420% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_sor| 0.595% | +0.215% | +13.660% | +11.800% | +-----+------------------------+--------+------------+------------+------------+ | :-( | scimark_sparse_mat_mult| 0.798% | -2.438% | -1.095% | -1.534% | +-----+------------------------+--------+------------+------------+------------+ | :-| | spectral_norm| 0.420% | -0.489% | +4.527% | +2.576% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlalchemy_declarative| 1.148% | -0.286% | +6.583% | +6.610% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlalchemy_imperative| 3.167% | +0.594% | +6.690% | +3.779% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlite_synth| 4.248% | -2.020% | +17.379% | +8.979% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_expand| 2.721% | +0.903% | +11.881% | +7.195% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_integrate| 1.884% | -0.420% | +9.251% | +6.365% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_str| 4.119% | -0.196% | +11.483% | +8.351% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_sum| 5.311% | +0.923% | +10.915% | +8.949% | +-----+------------------------+--------+------------+------------+------------+ | :-| | telco| 5.150% | +0.471% | +21.959% | +9.949% | +-----+------------------------+--------+------------+------------+------------+ | :-| | tornado_http| 1.009% | +0.225% | +5.949% | +6.401% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpack_sequence| 0.886% | +0.219% | +2.095% | +0.691% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle| 5.465% | +1.071% | +9.481% | +18.129% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle_list| 1.747% | -1.580% | +0.114% | +16.484% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle_pure_python| 3.519% | -0.984% | +6.017% | +9.617% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_generate| 0.953% | +1.453% | +6.091% | +9.021% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_iterparse| 2.506% | +1.278% | +3.581% | +7.705% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_parse| 2.292% | +0.304% | -7.052% | +13.223% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_process| 1.329% | +0.590% | +7.095% | +9.182% | +-----+------------------------+--------+------------+------------+------------+ * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/2-down-63-flat-results-for-python-mas… Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

bpo-32186: Release the GIL during lseek and fstat (#4652)
by Victor Stinner 01 Dec '17

01 Dec '17

https://github.com/python/cpython/commit/6a89481680b921e7b317c29877bdda9a60… commit: 6a89481680b921e7b317c29877bdda9a6031e5ad branch: master author: Nir Soffer <nirsof(a)gmail.com> committer: Victor Stinner <victor.stinner(a)gmail.com> date: 2017-12-01T02:18:58+01:00 summary: bpo-32186: Release the GIL during lseek and fstat (#4652) In _io_FileIO_readall_impl(), lseek() and _Py_fstat_noraise() were called without releasing the GIL. This can cause all threads to hang for unlimited time when calling FileIO.read() and the NFS server is not accessible. files: A Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst M Modules/_io/fileio.c diff --git a/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst b/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst new file mode 100644 index 00000000000..ea696c6098a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-11-30-20-38-16.bpo-32186.O42bVe.rst @@ -0,0 +1,3 @@ +io.FileIO.readall() and io.FileIO.read() now release the GIL when +getting the file size. Fixed hang of all threads with inaccessible NFS +server. Patch by Nir Soffer. diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index b6755b851dc..269142cfee5 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -683,10 +683,12 @@ _io_FileIO_readall_impl(fileio *self) Py_ssize_t bytes_read = 0; Py_ssize_t n; size_t bufsize; + int fstat_result; if (self->fd < 0) return err_closed(); + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH #ifdef MS_WINDOWS pos = _lseeki64(self->fd, 0L, SEEK_CUR); @@ -694,8 +696,10 @@ _io_FileIO_readall_impl(fileio *self) pos = lseek(self->fd, 0L, SEEK_CUR); #endif _Py_END_SUPPRESS_IPH + fstat_result = _Py_fstat_noraise(self->fd, &status); + Py_END_ALLOW_THREADS - if (_Py_fstat_noraise(self->fd, &status) == 0) + if (fstat_result == 0) end = status.st_size; else end = (Py_off_t)-1;

1 0