Mailman 3 July 2019 - Python-checkins

Don't skip pickle check_frame_opcodes() (GH-15027)
by Neil Schemenauer 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/0f9efbcf9a051198dc9e226528d5c5d24a… commit: 0f9efbcf9a051198dc9e226528d5c5d24a52d23a branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Neil Schemenauer <nas-github(a)arctrix.com> date: 2019-07-30T13:10:15-07:00 summary: Don't skip pickle check_frame_opcodes() (GH-15027) This looks like the only place that proto 4 framing gets exercised so leave it as part of the PGO task. (cherry picked from commit eca7ffc61cf925eae5def8c64b05d47f24e60e1f) Co-authored-by: Neil Schemenauer <nas-github(a)arctrix.com> files: M Lib/test/pickletester.py diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 5d402a4711bf..c9f374678ae3 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2281,7 +2281,6 @@ def test_setitems_on_non_dicts(self): FRAME_SIZE_MIN = 4 FRAME_SIZE_TARGET = 64 * 1024 - @support.skip_if_pgo_task def check_frame_opcodes(self, pickled): """ Check the arguments of FRAME opcodes in a protocol 4+ pickle.

1 0

Don't skip pickle check_frame_opcodes() (GH-15025)
by Neil Schemenauer 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/eca7ffc61cf925eae5def8c64b05d47f24… commit: eca7ffc61cf925eae5def8c64b05d47f24e60e1f branch: master author: Neil Schemenauer <nas-github(a)arctrix.com> committer: GitHub <noreply(a)github.com> date: 2019-07-30T12:08:17-07:00 summary: Don't skip pickle check_frame_opcodes() (GH-15025) This looks like the only place that proto 4 framing gets exercised so leave it as part of the PGO task. files: M Lib/test/pickletester.py diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 5d402a4711bf..c9f374678ae3 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2281,7 +2281,6 @@ def test_setitems_on_non_dicts(self): FRAME_SIZE_MIN = 4 FRAME_SIZE_TARGET = 64 * 1024 - @support.skip_if_pgo_task def check_frame_opcodes(self, pickled): """ Check the arguments of FRAME opcodes in a protocol 4+ pickle.

1 0

bpo-37707: Exclude expensive unit tests from PGO task (GH-15009) (#15024)
by Neil Schemenauer 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/382cb85401bb010ead411c0532499ffe16… commit: 382cb85401bb010ead411c0532499ffe16c3cf27 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Neil Schemenauer <nas-github(a)arctrix.com> date: 2019-07-30T11:34:32-07:00 summary: bpo-37707: Exclude expensive unit tests from PGO task (GH-15009) (#15024) Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable. (cherry picked from commit 52a48e62c6a94577152f9301bbe5f3bc806cfcf1) Co-authored-by: Neil Schemenauer <nas-github(a)arctrix.com> files: A Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst M Lib/test/libregrtest/main.py M Lib/test/pickletester.py M Lib/test/support/__init__.py M Lib/test/test_bz2.py M Lib/test/test_itertools.py M Lib/test/test_lzma.py M Lib/test/test_statistics.py diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index 2b6607df15dd..87278d0c76a2 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -643,6 +643,7 @@ def _main(self, tests, kwargs): input("Press any key to continue...") support.PGO = self.ns.pgo + support.PGO_EXTENDED = self.ns.pgo_extended setup_tests(self.ns) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 2bc99e6becfd..5d402a4711bf 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2281,6 +2281,7 @@ def test_setitems_on_non_dicts(self): FRAME_SIZE_MIN = 4 FRAME_SIZE_TARGET = 64 * 1024 + @support.skip_if_pgo_task def check_frame_opcodes(self, pickled): """ Check the arguments of FRAME opcodes in a protocol 4+ pickle. @@ -2328,6 +2329,7 @@ def check_frame_opcodes(self, pickled): elif frameless_start is not None: self.assertLess(pos - frameless_start, self.FRAME_SIZE_MIN) + @support.skip_if_pgo_task def test_framing_many_objects(self): obj = list(range(10**5)) for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): @@ -2417,6 +2419,7 @@ def remove_frames(pickled, keep_frame=None): count_opcode(pickle.FRAME, pickled)) self.assertEqual(obj, self.loads(some_frames_pickle)) + @support.skip_if_pgo_task def test_framed_write_sizes_with_delayed_writer(self): class ChunkAccumulator: """Accumulate pickler output in a list of raw chunks.""" diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 1ada1f927d9c..b88eb5804a96 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -971,6 +971,10 @@ def dec(*args, **kwargs): # useful for PGO PGO = False +# Set by libregrtest/main.py if we are running the extended (time consuming) +# PGO task. If this is True, PGO is also True. +PGO_EXTENDED = False + @contextlib.contextmanager def temp_dir(path=None, quiet=False): """Return a context manager that creates a temporary directory. @@ -2636,6 +2640,12 @@ def skip_unless_xattr(test): msg = "no non-broken extended attribute support" return test if ok else unittest.skip(msg)(test) +def skip_if_pgo_task(test): + """Skip decorator for tests not run in (non-extended) PGO task""" + ok = not PGO or PGO_EXTENDED + msg = "Not run for (non-extended) PGO task" + return test if ok else unittest.skip(msg)(test) + _bind_nix_socket_error = None def skip_unless_bind_unix_socket(test): """Decorator for tests requiring a functional bind() for unix sockets.""" diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py index f158b901b9c9..eb2f72ee4a5d 100644 --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -643,6 +643,7 @@ def testCompressChunks10(self): data += bz2c.flush() self.assertEqual(ext_decompress(data), self.TEXT) + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=2) def testCompress4G(self, size): # "Test BZ2Compressor.compress()/flush() with >4GiB input" @@ -701,6 +702,7 @@ def testEOFError(self): self.assertRaises(EOFError, bz2d.decompress, b"anything") self.assertRaises(EOFError, bz2d.decompress, b"") + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=3.3) def testDecompress4G(self, size): # "Test BZ2Decompressor.decompress() with >4GiB input" diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index ea060a98a5ee..573739fde14c 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -2062,6 +2062,7 @@ def gen2(x): self.assertRaises(AssertionError, list, cycle(gen1())) self.assertEqual(hist, [0,1]) + @support.skip_if_pgo_task def test_long_chain_of_empty_iterables(self): # Make sure itertools.chain doesn't run into recursion limits when # dealing with long chains of empty iterables. Even with a high diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py index 3dc2c1e7e3b7..117de0a1b554 100644 --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -333,6 +333,7 @@ def test_decompressor_multistream(self): # Test with inputs larger than 4GiB. + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=2) def test_compressor_bigmem(self, size): lzc = LZMACompressor() @@ -344,6 +345,7 @@ def test_compressor_bigmem(self, size): finally: ddata = None + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=3) def test_decompressor_bigmem(self, size): lzd = LZMADecompressor() diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index ed2f6579b0b9..104718ed7db6 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -14,6 +14,7 @@ import random import sys import unittest +from test import support from decimal import Decimal from fractions import Fraction @@ -2462,6 +2463,7 @@ def test_cdf(self): self.assertEqual(X.cdf(float('Inf')), 1.0) self.assertTrue(math.isnan(X.cdf(float('NaN')))) + @support.skip_if_pgo_task def test_inv_cdf(self): NormalDist = statistics.NormalDist diff --git a/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst b/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst new file mode 100644 index 000000000000..c0d58ab747b2 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst @@ -0,0 +1,3 @@ +Mark some individual tests to skip when --pgo is used. The tests marked +increase the PGO task time significantly and likely don't help improve +optimization of the final executable.

1 0

bpo-37707: Exclude expensive unit tests from PGO task (GH-15009)
by Neil Schemenauer 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/52a48e62c6a94577152f9301bbe5f3bc80… commit: 52a48e62c6a94577152f9301bbe5f3bc806cfcf1 branch: master author: Neil Schemenauer <nas-github(a)arctrix.com> committer: GitHub <noreply(a)github.com> date: 2019-07-30T11:08:18-07:00 summary: bpo-37707: Exclude expensive unit tests from PGO task (GH-15009) Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable. files: A Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst M Lib/test/libregrtest/main.py M Lib/test/pickletester.py M Lib/test/support/__init__.py M Lib/test/test_bz2.py M Lib/test/test_itertools.py M Lib/test/test_lzma.py M Lib/test/test_statistics.py diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index 2b6607df15dd..87278d0c76a2 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -643,6 +643,7 @@ def _main(self, tests, kwargs): input("Press any key to continue...") support.PGO = self.ns.pgo + support.PGO_EXTENDED = self.ns.pgo_extended setup_tests(self.ns) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 2bc99e6becfd..5d402a4711bf 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2281,6 +2281,7 @@ def test_setitems_on_non_dicts(self): FRAME_SIZE_MIN = 4 FRAME_SIZE_TARGET = 64 * 1024 + @support.skip_if_pgo_task def check_frame_opcodes(self, pickled): """ Check the arguments of FRAME opcodes in a protocol 4+ pickle. @@ -2328,6 +2329,7 @@ def check_frame_opcodes(self, pickled): elif frameless_start is not None: self.assertLess(pos - frameless_start, self.FRAME_SIZE_MIN) + @support.skip_if_pgo_task def test_framing_many_objects(self): obj = list(range(10**5)) for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): @@ -2417,6 +2419,7 @@ def remove_frames(pickled, keep_frame=None): count_opcode(pickle.FRAME, pickled)) self.assertEqual(obj, self.loads(some_frames_pickle)) + @support.skip_if_pgo_task def test_framed_write_sizes_with_delayed_writer(self): class ChunkAccumulator: """Accumulate pickler output in a list of raw chunks.""" diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 9efb3d91705f..dbbbdb0ee339 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -973,6 +973,10 @@ def dec(*args, **kwargs): # useful for PGO PGO = False +# Set by libregrtest/main.py if we are running the extended (time consuming) +# PGO task. If this is True, PGO is also True. +PGO_EXTENDED = False + @contextlib.contextmanager def temp_dir(path=None, quiet=False): """Return a context manager that creates a temporary directory. @@ -2638,6 +2642,12 @@ def skip_unless_xattr(test): msg = "no non-broken extended attribute support" return test if ok else unittest.skip(msg)(test) +def skip_if_pgo_task(test): + """Skip decorator for tests not run in (non-extended) PGO task""" + ok = not PGO or PGO_EXTENDED + msg = "Not run for (non-extended) PGO task" + return test if ok else unittest.skip(msg)(test) + _bind_nix_socket_error = None def skip_unless_bind_unix_socket(test): """Decorator for tests requiring a functional bind() for unix sockets.""" diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py index f158b901b9c9..eb2f72ee4a5d 100644 --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -643,6 +643,7 @@ def testCompressChunks10(self): data += bz2c.flush() self.assertEqual(ext_decompress(data), self.TEXT) + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=2) def testCompress4G(self, size): # "Test BZ2Compressor.compress()/flush() with >4GiB input" @@ -701,6 +702,7 @@ def testEOFError(self): self.assertRaises(EOFError, bz2d.decompress, b"anything") self.assertRaises(EOFError, bz2d.decompress, b"") + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=3.3) def testDecompress4G(self, size): # "Test BZ2Decompressor.decompress() with >4GiB input" diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index ea060a98a5ee..573739fde14c 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -2062,6 +2062,7 @@ def gen2(x): self.assertRaises(AssertionError, list, cycle(gen1())) self.assertEqual(hist, [0,1]) + @support.skip_if_pgo_task def test_long_chain_of_empty_iterables(self): # Make sure itertools.chain doesn't run into recursion limits when # dealing with long chains of empty iterables. Even with a high diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py index 3dc2c1e7e3b7..117de0a1b554 100644 --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -333,6 +333,7 @@ def test_decompressor_multistream(self): # Test with inputs larger than 4GiB. + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=2) def test_compressor_bigmem(self, size): lzc = LZMACompressor() @@ -344,6 +345,7 @@ def test_compressor_bigmem(self, size): finally: ddata = None + @support.skip_if_pgo_task @bigmemtest(size=_4G + 100, memuse=3) def test_decompressor_bigmem(self, size): lzd = LZMADecompressor() diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index ed2f6579b0b9..104718ed7db6 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -14,6 +14,7 @@ import random import sys import unittest +from test import support from decimal import Decimal from fractions import Fraction @@ -2462,6 +2463,7 @@ def test_cdf(self): self.assertEqual(X.cdf(float('Inf')), 1.0) self.assertTrue(math.isnan(X.cdf(float('NaN')))) + @support.skip_if_pgo_task def test_inv_cdf(self): NormalDist = statistics.NormalDist diff --git a/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst b/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst new file mode 100644 index 000000000000..c0d58ab747b2 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2019-07-29-11-36-16.bpo-37707.Sm-dGk.rst @@ -0,0 +1,3 @@ +Mark some individual tests to skip when --pgo is used. The tests marked +increase the PGO task time significantly and likely don't help improve +optimization of the final executable.

1 0

bpo-37704: Remove Tools/scripts/h2py.py (GH-15000)
by Victor Stinner 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/27eeaf0f2c9bd53a0fbdfdf38ee759e971… commit: 27eeaf0f2c9bd53a0fbdfdf38ee759e971221f8d branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-07-30T17:45:09+02:00 summary: bpo-37704: Remove Tools/scripts/h2py.py (GH-15000) Use cffi to access a C API in Python. files: A Misc/NEWS.d/next/Tools-Demos/2019-07-29-13-59-19.bpo-37704.xxGUz_.rst D Tools/scripts/h2py.py M Lib/test/test_tools/test_sundry.py M Tools/scripts/README diff --git a/Lib/test/test_tools/test_sundry.py b/Lib/test/test_tools/test_sundry.py index f5fed01491e3..10eb6941b3be 100644 --- a/Lib/test/test_tools/test_sundry.py +++ b/Lib/test/test_tools/test_sundry.py @@ -2,7 +2,7 @@ This file contains extremely basic regression tests for the scripts found in the Tools directory of a Python checkout or tarball which don't have separate -tests of their own, such as h2py.py. +tests of their own. """ import os diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-07-29-13-59-19.bpo-37704.xxGUz_.rst b/Misc/NEWS.d/next/Tools-Demos/2019-07-29-13-59-19.bpo-37704.xxGUz_.rst new file mode 100644 index 000000000000..52ca4895dee9 --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2019-07-29-13-59-19.bpo-37704.xxGUz_.rst @@ -0,0 +1 @@ +Remove ``Tools/scripts/h2py.py``: use cffi to access a C API in Python. diff --git a/Tools/scripts/README b/Tools/scripts/README index d4ac2ade25ef..bee5b0bd9b20 100644 --- a/Tools/scripts/README +++ b/Tools/scripts/README @@ -30,7 +30,6 @@ ftpmirror.py FTP mirror script get-remote-certificate.py Fetch the certificate that the server(s) are providing in PEM form google.py Open a webbrowser with Google gprof2html.py Transform gprof(1) output into useful HTML -h2py.py Translate #define's into Python assignments highlight.py Python syntax highlighting with HTML output idle3 Main program to start IDLE ifdef.py Remove #if(n)def groups from C sources diff --git a/Tools/scripts/h2py.py b/Tools/scripts/h2py.py deleted file mode 100755 index ea37c04d4c56..000000000000 --- a/Tools/scripts/h2py.py +++ /dev/null @@ -1,171 +0,0 @@ -#! /usr/bin/env python3 - -# Read #define's and translate to Python code. -# Handle #include statements. -# Handle #define macros with one argument. -# Anything that isn't recognized or doesn't translate into valid -# Python is ignored. - -# Without filename arguments, acts as a filter. -# If one or more filenames are given, output is written to corresponding -# filenames in the local directory, translated to all uppercase, with -# the extension replaced by ".py". - -# By passing one or more options of the form "-i regular_expression" -# you can specify additional strings to be ignored. This is useful -# e.g. to ignore casts to u_long: simply specify "-i '(u_long)'". - -# XXX To do: -# - turn trailing C comments into Python comments -# - turn C Boolean operators "&& || !" into Python "and or not" -# - what to do about #if(def)? -# - what to do about macros with multiple parameters? - -import sys, re, getopt, os - -p_define = re.compile(r'^[\t ]*#[\t ]*define[\t ]+([a-zA-Z0-9_]+)[\t ]+') - -p_macro = re.compile( - r'^[\t ]*#[\t ]*define[\t ]+' - r'([a-zA-Z0-9_]+)\(([_a-zA-Z][_a-zA-Z0-9]*)\)[\t ]+') - -p_include = re.compile(r'^[\t ]*#[\t ]*include[\t ]+<([^>\n]+)>') - -p_comment = re.compile(r'/\*([^*]+|\*+[^/])*(\*+/)?') -p_cpp_comment = re.compile('//.*') - -ignores = [p_comment, p_cpp_comment] - -p_char = re.compile(r"'(\\.[^\\]*|[^\\])'") - -p_hex = re.compile(r"0x([0-9a-fA-F]+)L?") - -filedict = {} -importable = {} - -try: - searchdirs=os.environ['include'].split(';') -except KeyError: - try: - searchdirs=os.environ['INCLUDE'].split(';') - except KeyError: - searchdirs=['/usr/include'] - try: - searchdirs.insert(0, os.path.join('/usr/include', - os.environ['MULTIARCH'])) - except KeyError: - pass - -def main(): - global filedict - opts, args = getopt.getopt(sys.argv[1:], 'i:') - for o, a in opts: - if o == '-i': - ignores.append(re.compile(a)) - if not args: - args = ['-'] - for filename in args: - if filename == '-': - sys.stdout.write('# Generated by h2py from stdin\n') - process(sys.stdin, sys.stdout) - else: - with open(filename) as fp: - outfile = os.path.basename(filename) - i = outfile.rfind('.') - if i > 0: outfile = outfile[:i] - modname = outfile.upper() - outfile = modname + '.py' - with open(outfile, 'w') as outfp: - outfp.write('# Generated by h2py from %s\n' % filename) - filedict = {} - for dir in searchdirs: - if filename[:len(dir)] == dir: - filedict[filename[len(dir)+1:]] = None # no '/' trailing - importable[filename[len(dir)+1:]] = modname - break - process(fp, outfp) - -def pytify(body): - # replace ignored patterns by spaces - for p in ignores: - body = p.sub(' ', body) - # replace char literals by ord(...) - body = p_char.sub("ord('\\1')", body) - # Compute negative hexadecimal constants - start = 0 - UMAX = 2*(sys.maxsize+1) - while 1: - m = p_hex.search(body, start) - if not m: break - s,e = m.span() - val = int(body[slice(*m.span(1))], 16) - if val > sys.maxsize: - val -= UMAX - body = body[:s] + "(" + str(val) + ")" + body[e:] - start = s + 1 - return body - -def process(fp, outfp, env = {}): - lineno = 0 - while 1: - line = fp.readline() - if not line: break - lineno = lineno + 1 - match = p_define.match(line) - if match: - # gobble up continuation lines - while line[-2:] == '\\\n': - nextline = fp.readline() - if not nextline: break - lineno = lineno + 1 - line = line + nextline - name = match.group(1) - body = line[match.end():] - body = pytify(body) - ok = 0 - stmt = '%s = %s\n' % (name, body.strip()) - try: - exec(stmt, env) - except: - sys.stderr.write('Skipping: %s' % stmt) - else: - outfp.write(stmt) - match = p_macro.match(line) - if match: - macro, arg = match.group(1, 2) - body = line[match.end():] - body = pytify(body) - stmt = 'def %s(%s): return %s\n' % (macro, arg, body) - try: - exec(stmt, env) - except: - sys.stderr.write('Skipping: %s' % stmt) - else: - outfp.write(stmt) - match = p_include.match(line) - if match: - regs = match.regs - a, b = regs[1] - filename = line[a:b] - if filename in importable: - outfp.write('from %s import *\n' % importable[filename]) - elif filename not in filedict: - filedict[filename] = None - inclfp = None - for dir in searchdirs: - try: - inclfp = open(dir + '/' + filename) - break - except IOError: - pass - if inclfp: - with inclfp: - outfp.write( - '\n# Included from %s\n' % filename) - process(inclfp, outfp, env) - else: - sys.stderr.write('Warning - could not find file %s\n' % - filename) - -if __name__ == '__main__': - main()

1 0

bpo-37587: Make json.loads faster for long strings (GH-14752)
by Miss Islington (bot) 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/9265a877426af4fa5c44cc8482e0198806… commit: 9265a877426af4fa5c44cc8482e0198806889350 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-07-30T07:37:28-07:00 summary: bpo-37587: Make json.loads faster for long strings (GH-14752) When scanning the string, most characters are valid, so checking for invalid characters first means never needing to check the value of strict on valid strings, and only needing to check it on invalid characters when doing non-strict parsing of invalid strings. This provides a measurable reduction in per-character processing time (~11% in the pre-merge patch testing). (cherry picked from commit 8a758f5b99c5fc3fd32edeac049d7d4a4b7cc163) Co-authored-by: Marco Paolini <mpaolini(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst M Modules/_json.c diff --git a/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst b/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst new file mode 100644 index 000000000000..80a89feab0ce --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst @@ -0,0 +1 @@ +Make json.loads faster for long strings. (Patch by Marco Paolini) diff --git a/Modules/_json.c b/Modules/_json.c index e3aa997598fc..048a9654ce18 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -439,7 +439,7 @@ scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next if (c == '"' || c == '\\') { break; } - else if (strict && c <= 0x1f) { + else if (c <= 0x1f && strict) { raise_errmsg("Invalid control character at", pystr, next); goto bail; }

1 0

bpo-37587: Make json.loads faster for long strings (GH-14752)
by Nick Coghlan 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/8a758f5b99c5fc3fd32edeac049d7d4a4b… commit: 8a758f5b99c5fc3fd32edeac049d7d4a4b7cc163 branch: master author: Marco Paolini <mpaolini(a)users.noreply.github.com> committer: Nick Coghlan <ncoghlan(a)gmail.com> date: 2019-07-31T00:16:34+10:00 summary: bpo-37587: Make json.loads faster for long strings (GH-14752) When scanning the string, most characters are valid, so checking for invalid characters first means never needing to check the value of strict on valid strings, and only needing to check it on invalid characters when doing non-strict parsing of invalid strings. This provides a measurable reduction in per-character processing time (~11% in the pre-merge patch testing). files: A Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst M Modules/_json.c diff --git a/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst b/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst new file mode 100644 index 000000000000..80a89feab0ce --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-07-13-16-02-48.bpo-37587.fd-1aF.rst @@ -0,0 +1 @@ +Make json.loads faster for long strings. (Patch by Marco Paolini) diff --git a/Modules/_json.c b/Modules/_json.c index 38beb6f50d2e..76da1d345e9d 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -439,7 +439,7 @@ scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next if (c == '"' || c == '\\') { break; } - else if (strict && c <= 0x1f) { + else if (c <= 0x1f && strict) { raise_errmsg("Invalid control character at", pystr, next); goto bail; }

1 0

bpo-37268: Add deprecation notice and a DeprecationWarning for the parser module (GH-15017)
by Miss Islington (bot) 30 Jul '19

30 Jul '19

https://github.com/python/cpython/commit/9211e2fd81fe1db6f73ded70752b144cc9… commit: 9211e2fd81fe1db6f73ded70752b144cc9691ab6 branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> date: 2019-07-30T04:04:01-07:00 summary: bpo-37268: Add deprecation notice and a DeprecationWarning for the parser module (GH-15017) Deprecate the parser module and add a deprecation warning triggered on import and a warning block in the documentation. https://bugs.python.org/issue37268 Automerge-Triggered-By: @pablogsal files: A Misc/NEWS.d/next/Library/2019-07-30-01-27-29.bpo-37268.QDmA44.rst M Doc/library/parser.rst M Doc/whatsnew/3.9.rst M Lib/test/test_parser.py M Modules/parsermodule.c diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst index a302681eca05..c55cd653b371 100644 --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -25,11 +25,11 @@ from this. This is better than trying to parse and modify an arbitrary Python code fragment as a string because parsing is performed in a manner identical to the code forming the application. It is also faster. -.. note:: +.. warning:: - From Python 2.5 onward, it's much more convenient to cut in at the Abstract - Syntax Tree (AST) generation and compilation stage, using the :mod:`ast` - module. + The parser module is deprecated and will be removed in future versions of + Python. For the majority of use cases you can leverage the Abstract Syntax + Tree (AST) generation and compilation stage, using the :mod:`ast` module. There are a few things to note about this module which are important to making use of the data structures created. This is not a tutorial on editing the parse diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index 85e254f061eb..273fd2b50d59 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -142,6 +142,10 @@ Deprecated Python versions it will raise a :exc:`TypeError` for all floats. (Contributed by Serhiy Storchaka in :issue:`37315`.) +* The :mod:`parser` module is deprecated and will be removed in future versions + of Python. For the majority of use cases users can leverage the Abstract Syntax + Tree (AST) generation and compilation stage, using the :mod:`ast` module. + Removed ======= diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py index e5285c636022..ec1845d7fe9a 100644 --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -6,6 +6,7 @@ import struct from test import support from test.support.script_helper import assert_python_failure +from test.support.script_helper import assert_python_ok # # First, we test that we can generate trees from valid source fragments, @@ -987,5 +988,13 @@ def test_two_args_to_expr(self): with self.assertRaises(TypeError): parser.expr("a", "b") + +class TestDeprecation(unittest.TestCase): + def test_deprecation_message(self): + code = "def f():\n import parser\n\nf()" + rc, out, err = assert_python_ok('-c', code) + self.assertIn(b'<string>:2: DeprecationWarning', err) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2019-07-30-01-27-29.bpo-37268.QDmA44.rst b/Misc/NEWS.d/next/Library/2019-07-30-01-27-29.bpo-37268.QDmA44.rst new file mode 100644 index 000000000000..d5c7b1d2fc66 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-07-30-01-27-29.bpo-37268.QDmA44.rst @@ -0,0 +1,2 @@ +The :mod:`parser` module is deprecated and will be removed in future +versions of Python. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c index 079d00f32aa6..b2495fc8b322 100644 --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -1158,6 +1158,12 @@ PyInit_parser(void) { PyObject *module, *copyreg; + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "The parser module is deprecated and will be removed " + "in future versions of Python", 7) != 0) { + return NULL; + } + if (PyType_Ready(&PyST_Type) < 0) return NULL; module = PyModule_Create(&parsermodule);

1 0

bpo-34162: Update idlelib/news.txt for 3.7.5 (GH-15016)
by Terry Jan Reedy 29 Jul '19

29 Jul '19

https://github.com/python/cpython/commit/e941cbcf8c2fce59a9afa8043948f9a147… commit: e941cbcf8c2fce59a9afa8043948f9a147e09177 branch: 3.7 author: Terry Jan Reedy <tjreedy(a)udel.edu> committer: GitHub <noreply(a)github.com> date: 2019-07-29T19:21:11-04:00 summary: bpo-34162: Update idlelib/news.txt for 3.7.5 (GH-15016) files: M Lib/idlelib/NEWS.txt diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index fc9f0fdee2c3..2b223e1c8b7a 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,5 +1,5 @@ -What's New in IDLE 3.7.4 -Released on 2019-06-24? +What's New in IDLE 3.7.5 +Released on 2019-09-30? ====================================== @@ -27,6 +27,11 @@ of after a random delay. bpo-27452: Cleanup config.py by inlining RemoveFile and simplifying the handling of __file__ in CreateConfigHandlers/ + +What's New in IDLE 3.7.4 +Released on 2019-07-08 +====================================== + bpo-26806: To compensate for stack frames added by IDLE and avoid possible problems with low recursion limits, add 30 to limits in the user code execution process. Subtract 30 when reporting recursion

1 0

bpo-34162: Update idlelib/news.txt. (GH-15011)
by Miss Islington (bot) 29 Jul '19

29 Jul '19

https://github.com/python/cpython/commit/8b50e3e2729190d4b65ee9510d81f01bd3… commit: 8b50e3e2729190d4b65ee9510d81f01bd31f2f7c branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-07-29T15:39:49-07:00 summary: bpo-34162: Update idlelib/news.txt. (GH-15011) (cherry picked from commit f35c51d2eadd297bcf06d4f7c536bd1d8682b724) Co-authored-by: Terry Jan Reedy <tjreedy(a)udel.edu> files: M Lib/idlelib/NEWS.txt diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 5d4a936d63f5..109b2d523a6e 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,30 @@ Released on 2019-10-20? ====================================== +bpo-37692: Improve highlight config sample with example shell +interaction and better labels for shell elements. + +bpo-37628: Settings dialog no longer expands with font size. +The font and highlight sample boxes gain scrollbars instead. + +bpo-17535: Add optional line numbers for IDLE editor windows. + +bpo-37627: Initialize the Customize Run dialog with the command line +arguments most recently entered before. The user can optionally edit +before submitting them. + +bpo-33610: Code context always shows the correct context when toggled on. + +bpo-36390: Gather Format menu functions into format.py. Combine +paragraph.py, rstrip.py, and format methods from editor.py. + +bpo-37530: Optimize code context to reduce unneeded background activity. +Font and highlight changes now occur along with text changes instead +of after a random delay. + +bpo-27452: Cleanup config.py by inlining RemoveFile and simplifying +the handling of __file__ in CreateConfigHandlers/ + bpo-26806: To compensate for stack frames added by IDLE and avoid possible problems with low recursion limits, add 30 to limits in the user code execution process. Subtract 30 when reporting recursion

1 0