Mailman 3 January 2018 - Python-checkins

bpo-28134: Ignore proto in unknown socket test (GH-5435)
by Christian Heimes Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/2e0ecde8d74f5fc0e3e3e39216975cc70e… commit: 2e0ecde8d74f5fc0e3e3e39216975cc70efc4796 branch: master author: Christian Heimes <christian(a)python.org> committer: GitHub <noreply(a)github.com> date: 2018-01-30T08:55:46+01:00 summary: bpo-28134: Ignore proto in unknown socket test (GH-5435) Band-aid for macOS: Some platforms seem to ignore unknown protocols. Signed-off-by: Christian Heimes <christian(a)python.org> files: M Lib/test/… [View More]

1 0

bpo-31961: subprocess now accepts path-like args (GH-4329)
by Gregory P. Smith Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/dd42cb71f2cb02f3a32f016137b12a146b… commit: dd42cb71f2cb02f3a32f016137b12a146bc0d0e2 branch: master author: Anders Lorentsen <Phaqui(a)gmail.com> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-01-29T23:27:28-08:00 summary: bpo-31961: subprocess now accepts path-like args (GH-4329) Allow os.PathLike args in subprocess APIs. files: A Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst M Doc/library/subprocess.rst M … [View More]Lib/subprocess.py M Lib/test/test_subprocess.py diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 86f3e06318f4..27d4288f67b3 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -335,12 +335,12 @@ functions. the class uses the Windows ``CreateProcess()`` function. The arguments to :class:`Popen` are as follows. - *args* should be a sequence of program arguments or else a single string. - By default, the program to execute is the first item in *args* if *args* is - a sequence. If *args* is a string, the interpretation is - platform-dependent and described below. See the *shell* and *executable* - arguments for additional differences from the default behavior. Unless - otherwise stated, it is recommended to pass *args* as a sequence. + *args* should be a sequence of program arguments or else a single string or + :term:`path-like object`. By default, the program to execute is the first + item in *args* if *args* is a sequence. If *args* is a string, the + interpretation is platform-dependent and described below. See the *shell* + and *executable* arguments for additional differences from the default + behavior. Unless otherwise stated, it is recommended to pass *args* as a sequence. On POSIX, if *args* is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not @@ -551,6 +551,10 @@ functions. Popen destructor now emits a :exc:`ResourceWarning` warning if the child process is still running. + .. versionchanged:: 3.7 + *args*, or the first element of *args* if *args* is a sequence, can now + be a :term:`path-like object`. + Exceptions ^^^^^^^^^^ diff --git a/Lib/subprocess.py b/Lib/subprocess.py index 93635ee61f7e..2723bc9e4274 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1097,7 +1097,12 @@ def _execute_child(self, args, executable, preexec_fn, close_fds, assert not pass_fds, "pass_fds not supported on Windows." if not isinstance(args, str): - args = list2cmdline(args) + try: + args = os.fsdecode(args) # os.PathLike -> str + except TypeError: # not an os.PathLike, must be a sequence. + args = list(args) + args[0] = os.fsdecode(args[0]) # os.PathLike -> str + args = list2cmdline(args) # Process startup details if startupinfo is None: @@ -1369,7 +1374,10 @@ def _execute_child(self, args, executable, preexec_fn, close_fds, if isinstance(args, (str, bytes)): args = [args] else: - args = list(args) + try: + args = list(args) + except TypeError: # os.PathLike instead of a sequence? + args = [os.fsencode(args)] # os.PathLike -> [str] if shell: # On Android the default shell is at '/system/bin/sh'. diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index eee24bbe37cf..858a70131205 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1475,6 +1475,37 @@ def test_run_kwargs(self): env=newenv) self.assertEqual(cp.returncode, 33) + def test_run_with_pathlike_path(self): + # bpo-31961: test run(pathlike_object) + class Path: + def __fspath__(self): + # the name of a command that can be run without + # any argumenets that exit fast + return 'dir' if mswindows else 'ls' + + path = Path() + if mswindows: + res = subprocess.run(path, stdout=subprocess.DEVNULL, shell=True) + else: + res = subprocess.run(path, stdout=subprocess.DEVNULL) + + self.assertEqual(res.returncode, 0) + + def test_run_with_pathlike_path_and_arguments(self): + # bpo-31961: test run([pathlike_object, 'additional arguments']) + class Path: + def __fspath__(self): + # the name of a command that can be run without + # any argumenets that exits fast + return sys.executable + + path = Path() + + args = [path, '-c', 'import sys; sys.exit(57)'] + res = subprocess.run(args) + + self.assertEqual(res.returncode, 57) + def test_capture_output(self): cp = self.run_python(("import sys;" "sys.stdout.write('BDFL'); " diff --git a/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst b/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst new file mode 100644 index 000000000000..611f4e978f1a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst @@ -0,0 +1,4 @@ +The *args* argument of subprocess.Popen can now be a +:term:`path-like object`. If *args* is given as a +sequence, it's first element can now be a +:term:`path-like object` as well. [View Less]

1 0

closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
by Benjamin Peterson Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/14e976e00e65bf343ba0fca016c3c9132a… commit: 14e976e00e65bf343ba0fca016c3c9132a843daf branch: master author: Eric Appelt <eric.appelt(a)gmail.com> committer: Benjamin Peterson <benjamin(a)python.org> date: 2018-01-29T23:13:03-08:00 summary: closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242) Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test. Use the … [View More]same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM. For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change. files: M Lib/lib2to3/tests/support.py M Lib/lib2to3/tests/test_parser.py diff --git a/Lib/lib2to3/tests/support.py b/Lib/lib2to3/tests/support.py index ae7cfe8ee274..fe084e8903fc 100644 --- a/Lib/lib2to3/tests/support.py +++ b/Lib/lib2to3/tests/support.py @@ -15,7 +15,13 @@ proj_dir = os.path.normpath(os.path.join(test_dir, "..")) grammar_path = os.path.join(test_dir, "..", "Grammar.txt") grammar = pgen2_driver.load_grammar(grammar_path) +grammar_no_print_statement = pgen2_driver.load_grammar(grammar_path) +del grammar_no_print_statement.keywords["print"] driver = pgen2_driver.Driver(grammar, convert=pytree.convert) +driver_no_print_statement = pgen2_driver.Driver( + grammar_no_print_statement, + convert=pytree.convert +) def parse_string(string): return driver.parse_string(reformat(string), debug=True) diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py index ed1ef8364639..15f5b63f1f1e 100644 --- a/Lib/lib2to3/tests/test_parser.py +++ b/Lib/lib2to3/tests/test_parser.py @@ -8,7 +8,7 @@ # Testing imports from . import support -from .support import driver +from .support import driver, driver_no_print_statement from test.support import verbose # Python imports @@ -413,8 +413,6 @@ class TestParserIdempotency(support.TestCase): """A cut-down version of pytree_idempotency.py.""" - # Issue 13125 - @unittest.expectedFailure def test_all_project_files(self): for filepath in support.all_project_files(): with open(filepath, "rb") as fp: @@ -425,12 +423,13 @@ def test_all_project_files(self): source = fp.read() try: tree = driver.parse_string(source) - except ParseError as err: - if verbose > 0: - warnings.warn('ParseError on file %s (%s)' % (filepath, err)) - continue + except ParseError: + try: + tree = driver_no_print_statement.parse_string(source) + except ParseError as err: + self.fail('ParseError on file %s (%s)' % (filepath, err)) new = str(tree) - x = diff(filepath, new) + x = diff(filepath, new, encoding=encoding) if x: self.fail("Idempotency failed: %s" % filepath) @@ -481,9 +480,9 @@ def test_trailing_comma_after_generator_expression_argument_works(self): self.validate("set(x for x in [],)") -def diff(fn, result): +def diff(fn, result, encoding='utf-8'): try: - with open('@', 'w') as f: + with open('@', 'w', encoding=encoding, newline='\n') as f: f.write(str(result)) fn = fn.replace('"', '\\"') return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None)) [View Less]

1 0

closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441) (GH-5442)
by Benjamin Peterson Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/6b2bbcc4cca414f35f67caa4674f59f41f… commit: 6b2bbcc4cca414f35f67caa4674f59f41ff638ea branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Benjamin Peterson <benjamin(a)python.org> date: 2018-01-29T23:01:32-08:00 summary: closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441) (GH-5442) (cherry picked from commit 95441809ef77a8df5e14601ade6c054ef7114c02) Co-authored-by:… [View More]

1 0

bpo-32102 Add "capture_output=True" to subprocess.run (GH-5149)
by Gregory P. Smith Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/ce0f33d04528fcafc673a8707871f8430d… commit: ce0f33d04528fcafc673a8707871f8430d8f7ce8 branch: master author: Bo Bayles <bbayles(a)gmail.com> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-01-29T22:40:39-08:00 summary: bpo-32102 Add "capture_output=True" to subprocess.run (GH-5149) Add "capture_output=True" option to subprocess.run, this is equivalent to setting stdout=PIPE, stderr=PIPE but is much more readable. files: A Misc/… [View More]NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst M Doc/library/subprocess.rst M Lib/subprocess.py M Lib/test/test_subprocess.py M Misc/ACKS diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index af96f41ef40a..86f3e06318f4 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -47,12 +47,14 @@ compatibility with older versions, see the :ref:`call-function-trio` section. The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - apart from *timeout*, - *input* and *check*, all the arguments to this function are passed through to - that interface. + same as that of the :class:`Popen` constructor - most of the arguments to + this function are passed through to that interface. (*timeout*, *input*, + *check*, and *capture_output* are not.) - This does not capture stdout or stderr by default. To do so, pass - :data:`PIPE` for the *stdout* and/or *stderr* arguments. + If *capture_output* is true, stdout and stderr will be captured. + When used, the internal :class:`Popen` object is automatically created with + ``stdout=PIPE`` and ``stderr=PIPE``. The *stdout* and *stderr* arguments may + not be used as well. The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout expires, the child process will be killed and waited for. The @@ -86,9 +88,9 @@ compatibility with older versions, see the :ref:`call-function-trio` section. ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + >>> subprocess.run(["ls", "-l", "/dev/null"], capture_output=True) CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, - stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr=b'') .. versionadded:: 3.5 @@ -98,7 +100,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. versionchanged:: 3.7 - Added the *text* parameter, as a more understandable alias of *universal_newlines* + Added the *text* parameter, as a more understandable alias of *universal_newlines*. + Added the *capture_output* parameter. .. class:: CompletedProcess diff --git a/Lib/subprocess.py b/Lib/subprocess.py index f69159e3aadc..93635ee61f7e 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -409,7 +409,8 @@ def check_returncode(self): self.stderr) -def run(*popenargs, input=None, timeout=None, check=False, **kwargs): +def run(*popenargs, + input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and @@ -442,6 +443,13 @@ def run(*popenargs, input=None, timeout=None, check=False, **kwargs): raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE + if capture_output: + if ('stdout' in kwargs) or ('stderr' in kwargs): + raise ValueError('stdout and stderr arguments may not be used ' + 'with capture_output.') + kwargs['stdout'] = PIPE + kwargs['stderr'] = PIPE + with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index dd63818f3b2f..eee24bbe37cf 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1475,6 +1475,38 @@ def test_run_kwargs(self): env=newenv) self.assertEqual(cp.returncode, 33) + def test_capture_output(self): + cp = self.run_python(("import sys;" + "sys.stdout.write('BDFL'); " + "sys.stderr.write('FLUFL')"), + capture_output=True) + self.assertIn(b'BDFL', cp.stdout) + self.assertIn(b'FLUFL', cp.stderr) + + def test_stdout_with_capture_output_arg(self): + # run() refuses to accept 'stdout' with 'capture_output' + tf = tempfile.TemporaryFile() + self.addCleanup(tf.close) + with self.assertRaises(ValueError, + msg=("Expected ValueError when stdout and capture_output " + "args supplied.")) as c: + output = self.run_python("print('will not be run')", + capture_output=True, stdout=tf) + self.assertIn('stdout', c.exception.args[0]) + self.assertIn('capture_output', c.exception.args[0]) + + def test_stderr_with_capture_output_arg(self): + # run() refuses to accept 'stderr' with 'capture_output' + tf = tempfile.TemporaryFile() + self.addCleanup(tf.close) + with self.assertRaises(ValueError, + msg=("Expected ValueError when stderr and capture_output " + "args supplied.")) as c: + output = self.run_python("print('will not be run')", + capture_output=True, stderr=tf) + self.assertIn('stderr', c.exception.args[0]) + self.assertIn('capture_output', c.exception.args[0]) + @unittest.skipIf(mswindows, "POSIX specific tests") class POSIXProcessTestCase(BaseTestCase): diff --git a/Misc/ACKS b/Misc/ACKS index bfcdd33b8ed7..3fdfa3041f87 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -108,6 +108,7 @@ Michael R Bax Anthony Baxter Mike Bayer Samuel L. Bayer +Bo Bayles Tommy Beadle Donald Beaudry David Beazley diff --git a/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst b/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst new file mode 100644 index 000000000000..cd4d0b5a8bd6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst @@ -0,0 +1 @@ +New argument ``capture_output`` for subprocess.run [View Less]

1 0

closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441)
by Benjamin Peterson Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/95441809ef77a8df5e14601ade6c054ef7… commit: 95441809ef77a8df5e14601ade6c054ef7114c02 branch: master author: Benjamin Peterson <benjamin(a)python.org> committer: GitHub <noreply(a)github.com> date: 2018-01-29T22:14:17-08:00 summary: closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441) files: A Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst M Lib/test/test_hashlib.py diff --git a/Lib/test/test_hashlib.… [View More]

1 0

bpo-32681: Fix an uninitialized variable in the C implementation of os.dup2 (GH-5346)
by Gregory P. Smith Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/3d86e484de6334fe16cbab512744597bd0… commit: 3d86e484de6334fe16cbab512744597bd0de4e80 branch: master author: Stéphane Wirtel <stephane(a)wirtel.be> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-01-29T22:04:36-08:00 summary: bpo-32681: Fix an uninitialized variable in the C implementation of os.dup2 (GH-5346) See https://bugs.python.org/issue32441 for where this was introduced. files: A Misc/NEWS.d/next/C API/2018-01-26-17-29-29.… [View More]

1 0

[3.6] replace dynamic import with 'exec' with importlib.import_module (GH-5433) (GH-5440)
by Benjamin Peterson Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/88fa79a10a340439ed021fbc4988b6f3e7… commit: 88fa79a10a340439ed021fbc4988b6f3e73b2391 branch: 3.6 author: Benjamin Peterson <benjamin(a)python.org> committer: GitHub <noreply(a)github.com> date: 2018-01-29T22:02:09-08:00 summary: [3.6] replace dynamic import with 'exec' with importlib.import_module (GH-5433) (GH-5440) (cherry picked from commit 77526f05fa788d6fb12f2121fe6b96c130d9b717) files: M Lib/test/test_hashlib.py diff --git a/Lib/… [View More]

1 0

bpo-21417: Add compresslevel= to the zipfile module (GH-5385)
by Gregory P. Smith Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/ce237c7d58ba207575cdfb0195a58a6407… commit: ce237c7d58ba207575cdfb0195a58a6407fbf717 branch: master author: Bo Bayles <bbayles(a)gmail.com> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-01-29T21:54:07-08:00 summary: bpo-21417: Add compresslevel= to the zipfile module (GH-5385) This allows the compression level to be specified when writing zipfiles (for the entire file *and* overridden on a per-file basis). Contributed by Bo … [View More]Bayles files: A Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst M Doc/library/zipfile.rst M Lib/test/test_zipfile.py M Lib/zipfile.py diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 5b8c776ed648..d58efe0b4175 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -130,10 +130,12 @@ ZipFile Objects --------------- -.. class:: ZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=True) +.. class:: ZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=True, \ + compresslevel=None) Open a ZIP file, where *file* can be a path to a file (a string), a file-like object or a :term:`path-like object`. + The *mode* parameter should be ``'r'`` to read an existing file, ``'w'`` to truncate and write a new file, ``'a'`` to append to an existing file, or ``'x'`` to exclusively create and write a new file. @@ -145,16 +147,27 @@ ZipFile Objects adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``'a'`` and the file does not exist at all, it is created. If *mode* is ``'r'`` or ``'a'``, the file should be seekable. + *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized - values will cause :exc:`NotImplementedError` to be raised. If :const:`ZIP_DEFLATED`, - :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponding module - (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available, :exc:`RuntimeError` - is raised. The default is :const:`ZIP_STORED`. If *allowZip64* is - ``True`` (the default) zipfile will create ZIP files that use the ZIP64 - extensions when the zipfile is larger than 4 GiB. If it is false :mod:`zipfile` - will raise an exception when the ZIP file would require ZIP64 extensions. + values will cause :exc:`NotImplementedError` to be raised. If + :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified + but the corresponding module (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not + available, :exc:`RuntimeError` is raised. The default is :const:`ZIP_STORED`. + + If *allowZip64* is ``True`` (the default) zipfile will create ZIP files that + use the ZIP64 extensions when the zipfile is larger than 4 GiB. If it is + ``false`` :mod:`zipfile` will raise an exception when the ZIP file would + require ZIP64 extensions. + + The *compresslevel* parameter controls the compression level to use when + writing files to the archive. + When using :const:`ZIP_STORED` or :const:`ZIP_LZMA` it has no effect. + When using :const:`ZIP_DEFLATED` integers ``0`` through ``9`` are accepted + (see :class:`zlib <zlib.compressobj>` for more information). + When using :const:`ZIP_BZIP2` integers ``1`` through ``9`` are accepted + (see :class:`bz2 <bz2.BZ2File>` for more information). If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate @@ -187,6 +200,9 @@ ZipFile Objects .. versionchanged:: 3.6.2 The *file* parameter accepts a :term:`path-like object`. + .. versionchanged:: 3.7 + Add the *compresslevel* parameter. + .. method:: ZipFile.close() @@ -351,13 +367,15 @@ ZipFile Objects :exc:`ValueError`. Previously, a :exc:`RuntimeError` was raised. -.. method:: ZipFile.write(filename, arcname=None, compress_type=None) +.. method:: ZipFile.write(filename, arcname=None, compress_type=None, \ + compresslevel=None) Write the file named *filename* to the archive, giving it the archive name *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. + the new entry. Similarly, *compresslevel* will override the constructor if + given. The archive must be open with mode ``'w'``, ``'x'`` or ``'a'``. .. note:: @@ -383,7 +401,8 @@ ZipFile Objects a :exc:`RuntimeError` was raised. -.. method:: ZipFile.writestr(zinfo_or_arcname, data[, compress_type]) +.. method:: ZipFile.writestr(zinfo_or_arcname, data, compress_type=None, \ + compresslevel=None) Write the string *data* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's @@ -393,7 +412,8 @@ ZipFile Objects If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* - (if that is a :class:`ZipInfo` instance). + (if that is a :class:`ZipInfo` instance). Similarly, *compresslevel* will + override the constructor if given. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 3bc867ea51c9..94db858a1517 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -53,9 +53,10 @@ def setUp(self): with open(TESTFN, "wb") as fp: fp.write(self.data) - def make_test_archive(self, f, compression): + def make_test_archive(self, f, compression, compresslevel=None): + kwargs = {'compression': compression, 'compresslevel': compresslevel} # Create the ZIP archive - with zipfile.ZipFile(f, "w", compression) as zipfp: + with zipfile.ZipFile(f, "w", **kwargs) as zipfp: zipfp.write(TESTFN, "another.name") zipfp.write(TESTFN, TESTFN) zipfp.writestr("strfile", self.data) @@ -63,8 +64,8 @@ def make_test_archive(self, f, compression): for line in self.line_gen: f.write(line) - def zip_test(self, f, compression): - self.make_test_archive(f, compression) + def zip_test(self, f, compression, compresslevel=None): + self.make_test_archive(f, compression, compresslevel) # Read the ZIP archive with zipfile.ZipFile(f, "r", compression) as zipfp: @@ -297,6 +298,22 @@ def test_writestr_compression(self): info = zipfp.getinfo('b.txt') self.assertEqual(info.compress_type, self.compression) + def test_writestr_compresslevel(self): + zipfp = zipfile.ZipFile(TESTFN2, "w", compresslevel=1) + zipfp.writestr("a.txt", "hello world", compress_type=self.compression) + zipfp.writestr("b.txt", "hello world", compress_type=self.compression, + compresslevel=2) + + # Compression level follows the constructor. + a_info = zipfp.getinfo('a.txt') + self.assertEqual(a_info.compress_type, self.compression) + self.assertEqual(a_info._compresslevel, 1) + + # Compression level is overridden. + b_info = zipfp.getinfo('b.txt') + self.assertEqual(b_info.compress_type, self.compression) + self.assertEqual(b_info._compresslevel, 2) + def test_read_return_size(self): # Issue #9837: ZipExtFile.read() shouldn't return more bytes # than requested. @@ -370,6 +387,21 @@ def test_repr(self): self.assertIn('[closed]', repr(zipopen)) self.assertIn('[closed]', repr(zipfp)) + def test_compresslevel_basic(self): + for f in get_files(self): + self.zip_test(f, self.compression, compresslevel=9) + + def test_per_file_compresslevel(self): + """Check that files within a Zip archive can have different + compression levels.""" + with zipfile.ZipFile(TESTFN2, "w", compresslevel=1) as zipfp: + zipfp.write(TESTFN, 'compress_1') + zipfp.write(TESTFN, 'compress_9', compresslevel=9) + one_info = zipfp.getinfo('compress_1') + nine_info = zipfp.getinfo('compress_9') + self.assertEqual(one_info._compresslevel, 1) + self.assertEqual(nine_info._compresslevel, 9) + def tearDown(self): unlink(TESTFN) unlink(TESTFN2) diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 37ce3281e092..f9db45f58a2b 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -295,6 +295,7 @@ class ZipInfo (object): 'filename', 'date_time', 'compress_type', + '_compresslevel', 'comment', 'extra', 'create_system', @@ -334,6 +335,7 @@ def __init__(self, filename="NoName", date_time=(1980,1,1,0,0,0)): # Standard values: self.compress_type = ZIP_STORED # Type of compression for the file + self._compresslevel = None # Level for the compressor self.comment = b"" # Comment for each file self.extra = b"" # ZIP extra data if sys.platform == 'win32': @@ -654,12 +656,16 @@ def _check_compression(compression): raise NotImplementedError("That compression method is not supported") -def _get_compressor(compress_type): +def _get_compressor(compress_type, compresslevel=None): if compress_type == ZIP_DEFLATED: - return zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, - zlib.DEFLATED, -15) + if compresslevel is not None: + return zlib.compressobj(compresslevel, zlib.DEFLATED, -15) + return zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, -15) elif compress_type == ZIP_BZIP2: + if compresslevel is not None: + return bz2.BZ2Compressor(compresslevel) return bz2.BZ2Compressor() + # compresslevel is ignored for ZIP_LZMA elif compress_type == ZIP_LZMA: return LZMACompressor() else: @@ -963,7 +969,8 @@ def __init__(self, zf, zinfo, zip64): self._zinfo = zinfo self._zip64 = zip64 self._zipfile = zf - self._compressor = _get_compressor(zinfo.compress_type) + self._compressor = _get_compressor(zinfo.compress_type, + zinfo._compresslevel) self._file_size = 0 self._compress_size = 0 self._crc = 0 @@ -1035,7 +1042,8 @@ def close(self): class ZipFile: """ Class with methods to open, read, write, close, list zip files. - z = ZipFile(file, mode="r", compression=ZIP_STORED, allowZip64=True) + z = ZipFile(file, mode="r", compression=ZIP_STORED, allowZip64=True, + compresslevel=None) file: Either the path to the file, or a file-like object. If it is a path, the file will be opened and closed by ZipFile. @@ -1046,13 +1054,19 @@ class ZipFile: allowZip64: if True ZipFile will create files with ZIP64 extensions when needed, otherwise it will raise an exception when this would be necessary. + compresslevel: None (default for the given compression type) or an integer + specifying the level to pass to the compressor. + When using ZIP_STORED or ZIP_LZMA this keyword has no effect. + When using ZIP_DEFLATED integers 0 through 9 are accepted. + When using ZIP_BZIP2 integers 1 through 9 are accepted. """ fp = None # Set here since __del__ checks it _windows_illegal_name_trans_table = None - def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True): + def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True, + compresslevel=None): """Open the ZIP file with mode read 'r', write 'w', exclusive create 'x', or append 'a'.""" if mode not in ('r', 'w', 'x', 'a'): @@ -1066,6 +1080,7 @@ def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True): self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression + self.compresslevel = compresslevel self.mode = mode self.pwd = None self._comment = b'' @@ -1342,6 +1357,7 @@ def open(self, name, mode="r", pwd=None, *, force_zip64=False): elif mode == 'w': zinfo = ZipInfo(name) zinfo.compress_type = self.compression + zinfo._compresslevel = self.compresslevel else: # Get info object for name zinfo = self.getinfo(name) @@ -1575,7 +1591,8 @@ def _writecheck(self, zinfo): raise LargeZipFile(requires_zip64 + " would require ZIP64 extensions") - def write(self, filename, arcname=None, compress_type=None): + def write(self, filename, arcname=None, + compress_type=None, compresslevel=None): """Put the bytes from filename into the archive under the name arcname.""" if not self.fp: @@ -1597,6 +1614,11 @@ def write(self, filename, arcname=None, compress_type=None): else: zinfo.compress_type = self.compression + if compresslevel is not None: + zinfo._compresslevel = compresslevel + else: + zinfo._compresslevel = self.compresslevel + if zinfo.is_dir(): with self._lock: if self._seekable: @@ -1617,7 +1639,8 @@ def write(self, filename, arcname=None, compress_type=None): with open(filename, "rb") as src, self.open(zinfo, 'w') as dest: shutil.copyfileobj(src, dest, 1024*8) - def writestr(self, zinfo_or_arcname, data, compress_type=None): + def writestr(self, zinfo_or_arcname, data, + compress_type=None, compresslevel=None): """Write a file into the archive. The contents is 'data', which may be either a 'str' or a 'bytes' instance; if it is a 'str', it is encoded as UTF-8 first. @@ -1629,6 +1652,7 @@ def writestr(self, zinfo_or_arcname, data, compress_type=None): zinfo = ZipInfo(filename=zinfo_or_arcname, date_time=time.localtime(time.time())[:6]) zinfo.compress_type = self.compression + zinfo._compresslevel = self.compresslevel if zinfo.filename[-1] == '/': zinfo.external_attr = 0o40775 << 16 # drwxrwxr-x zinfo.external_attr |= 0x10 # MS-DOS directory flag @@ -1648,6 +1672,9 @@ def writestr(self, zinfo_or_arcname, data, compress_type=None): if compress_type is not None: zinfo.compress_type = compress_type + if compresslevel is not None: + zinfo._compresslevel = compresslevel + zinfo.file_size = len(data) # Uncompressed size with self._lock: with self.open(zinfo, mode='w') as dest: diff --git a/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst b/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst new file mode 100644 index 000000000000..50207a0e4c33 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst @@ -0,0 +1 @@ +Added support for setting the compression level for zipfile.ZipFile. [View Less]

1 0

[3.6] Revert "bpo-32690: Preserve order of locals() (GH-5379) (#5390)"
by Nick Coghlan Jan. 30, 2018

Jan. 30, 2018

https://github.com/python/cpython/commit/05f91a42cd9575ef338a67bd04855b44f6… commit: 05f91a42cd9575ef338a67bd04855b44f6ac20c2 branch: 3.6 author: Nick Coghlan <ncoghlan(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2018-01-30T15:27:41+10:00 summary: [3.6] Revert "bpo-32690: Preserve order of locals() (GH-5379) (#5390)" This reverts commit 9105879bfd7133ecbac67f3e9c0bacf6e477de5a in order to keep the behaviour of locals() consistent between 3.6.4 and 3.6.5+. files: D … [View More]

1 0