Mailman 3 November 2011 - Python-checkins

cpython: Issue #6715: Add module for compression using the LZMA algorithm.
by nadeem.vawda Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/74d182cf0187 changeset: 73794:74d182cf0187 user: Nadeem Vawda <nadeem.vawda(a)gmail.com> date: Wed Nov 30 00:25:06 2011 +0200 summary: Issue #6715: Add module for compression using the LZMA algorithm. files: Doc/library/archiving.rst | 4 +- Doc/library/bz2.rst | 2 +- Doc/library/gzip.rst | 2 +- Doc/library/lzma.rst | 344 ++++ Doc/library/zipfile.rst | 2 +- Doc/library/zlib.rst | 4 +- Lib/lzma.py | 398 ++++ Lib/test/test_lzma.py | 1335 ++++++++++++++++ Misc/NEWS | 2 + Modules/_lzmamodule.c | 1106 +++++++++++++ PCbuild/_lzma.vcproj | 537 ++++++ PCbuild/pcbuild.sln | 21 + PCbuild/pyproject.vsprops | 4 + PCbuild/readme.txt | 6 + Tools/buildbot/external-common.bat | 5 + setup.py | 7 + 16 files changed, 3773 insertions(+), 6 deletions(-) diff --git a/Doc/library/archiving.rst b/Doc/library/archiving.rst --- a/Doc/library/archiving.rst +++ b/Doc/library/archiving.rst @@ -5,7 +5,8 @@ ****************************** The modules described in this chapter support data compression with the zlib, -gzip, and bzip2 algorithms, and the creation of ZIP- and tar-format archives. +gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format +archives. .. toctree:: @@ -13,5 +14,6 @@ zlib.rst gzip.rst bz2.rst + lzma.rst zipfile.rst tarfile.rst diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -12,7 +12,7 @@ This module provides a comprehensive interface for compressing and decompressing data using the bzip2 compression algorithm. -For related file formats, see the :mod:`gzip`, :mod:`zipfile`, and +For related file formats, see the :mod:`gzip`, :mod:`lzma`, :mod:`zipfile`, and :mod:`tarfile` modules. The :mod:`bz2` module contains: diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -21,7 +21,7 @@ :program:`gzip` and :program:`gunzip` programs, such as those produced by :program:`compress` and :program:`pack`, are not supported by this module. -For other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and +For related file formats, see the :mod:`bz2`, :mod:`lzma`, :mod:`zipfile`, and :mod:`tarfile` modules. The module defines the following items: diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst new file mode 100644 --- /dev/null +++ b/Doc/library/lzma.rst @@ -0,0 +1,344 @@ +:mod:`lzma` --- Compression using the LZMA algorithm +==================================================== + +.. module:: lzma + :synopsis: A Python wrapper for the liblzma compression library. +.. moduleauthor:: Nadeem Vawda <nadeem.vawda(a)gmail.com> +.. sectionauthor:: Nadeem Vawda <nadeem.vawda(a)gmail.com> + +.. versionadded:: 3.3 + + +This module provides classes and convenience functions for compressing and +decompressing data using the LZMA compression algorithm. Also included is a file +interface supporting the ``.xz`` and legacy ``.lzma`` file formats used by the +:program:`xz` utility, as well as raw compressed streams. + +For related file formats, see the :mod:`bz2`, :mod:`gzip`, :mod:`zipfile`, and +:mod:`tarfile` modules. + +The interface provided by this module is very similar to that of the :mod:`bz2` +module. However, note that :class:`LZMAFile` is *not* thread-safe, unlike +:class:`bz2.BZ2File`, so if you need to use a single :class:`LZMAFile` instance +from multiple threads, it is necessary to protect it with a lock. + + +.. exception:: LZMAError + + This exception is raised when an error occurs during compression or + decompression, or while initializing the compressor/decompressor state. + + +Reading and writing compressed files +------------------------------------ + +.. class:: LZMAFile(filename=None, mode="r", fileobj=None, format=None, check=-1, preset=None, filters=None) + + Open an LZMA-compressed file. + + An :class:`LZMAFile` can wrap an existing :term:`file object` (given by + *fileobj*), or operate directly on a named file (named by *filename*). + Exactly one of these two parameters should be provided. If *fileobj* is + provided, it is not closed when the :class:`LZMAFile` is closed. + + The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for + overwriting, or ``"a"`` for appending. If *fileobj* is provided, a mode of + ``"w"`` does not truncate the file, and is instead equivalent to ``"a"``. + + When opening a file for reading, the input file may be the concatenation of + multiple separate compressed streams. These are transparently decoded as a + single logical stream. + + When opening a file for reading, the *format* and *filters* arguments have + the same meanings as for :class:`LZMADecompressor`. In this case, the *check* + and *preset* arguments should not be used. + + When opening a file for writing, the *format*, *check*, *preset* and + *filters* arguments have the same meanings as for :class:`LZMACompressor`. + + :class:`LZMAFile` supports all the members specified by + :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. + Iteration and the :keyword:`with` statement are supported. + + The following method is also provided: + + .. method:: peek(size=-1) + + Return buffered data without advancing the file position. At least one + byte of data will be returned, unless EOF has been reached. The exact + number of bytes returned is unspecified (the *size* argument is ignored). + + +Compressing and decompressing data in memory +-------------------------------------------- + +.. class:: LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None) + + Create a compressor object, which can be used to compress data incrementally. + + For a more convenient way of compressing a single chunk of data, see + :func:`compress`. + + The *format* argument specifies what container format should be used. + Possible values are: + + * :const:`FORMAT_XZ`: The ``.xz`` container format. + This is the default format. + + * :const:`FORMAT_ALONE`: The legacy ``.lzma`` container format. + This format is more limited than ``.xz`` -- it does not support integrity + checks or multiple filters. + + * :const:`FORMAT_RAW`: A raw data stream, not using any container format. + This format specifier does not support integrity checks, and requires that + you always specify a custom filter chain (for both compression and + decompression). Additionally, data compressed in this manner cannot be + decompressed using :const:`FORMAT_AUTO` (see :class:`LZMADecompressor`). + + The *check* argument specifies the type of integrity check to include in the + compressed data. This check is used when decompressing, to ensure that the + data has not been corrupted. Possible values are: + + * :const:`CHECK_NONE`: No integrity check. + This is the default (and the only acceptable value) for + :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`. + + * :const:`CHECK_CRC32`: 32-bit Cyclic Redundancy Check. + + * :const:`CHECK_CRC64`: 64-bit Cyclic Redundancy Check. + This is the default for :const:`FORMAT_XZ`. + + * :const:`CHECK_SHA256`: 256-bit Secure Hash Algorithm. + + If the specified check is not supported, an :class:`LZMAError` is raised. + + The compression settings can be specified either as a preset compression + level (with the *preset* argument), or in detail as a custom filter chain + (with the *filters* argument). + + The *preset* argument (if provided) should be an integer between ``0`` and + ``9`` (inclusive), optionally OR-ed with the constant + :const:`PRESET_EXTREME`. If neither *preset* nor *filters* are given, the + default behavior is to use :const:`PRESET_DEFAULT` (preset level ``6``). + Higher presets produce smaller output, but make compression more CPU- and + memory-intensive, and also increase the memory required for decompression. + + The *filters* argument (if provided) should be a filter chain specifier. + See :ref:`filter-chain-specs` for details. + + .. method:: compress(data) + + Compress *data* (a :class:`bytes` object), returning a :class:`bytes` + object containing compressed data for at least part of the input. Some of + *data* may be buffered internally, for use in later calls to + :meth:`compress` and :meth:`flush`. The returned data should be + concatenated with the output of any previous calls to :meth:`compress`. + + .. method:: flush() + + Finish the compression process, returning a :class:`bytes` object + containing any data stored in the compressor's internal buffers. + + The compressor cannot be used after this method has been called. + + +.. class:: LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None) + + Create a decompressor object, which can be used to decompress data + incrementally. + + For a more convenient way of decompressing an entire compressed stream at + once, see :func:`decompress`. + + The *format* argument specifies the container format that should be used. The + default is :const:`FORMAT_AUTO`, which can decompress both ``.xz`` and + ``.lzma`` files. Other possible values are :const:`FORMAT_XZ`, + :const:`FORMAT_ALONE`, and :const:`FORMAT_RAW`. + + The *memlimit* argument specifies a limit (in bytes) on the amount of memory + that the decompressor can use. When this argument is used, decompression will + fail with an :class:`LZMAError` if it is not possible to decompress the input + within the given memory limit. + + The *filters* argument specifies the filter chain that was used to create + the stream being decompressed. This argument is required if *format* is + :const:`FORMAT_RAW`, but should not be used for other formats. + See :ref:`filter-chain-specs` for more information about filter chains. + + .. note:: + This class does not transparently handle inputs containing multiple + compressed streams, unlike :func:`decompress` and :class:`LZMAFile`. To + decompress a multi-stream input with :class:`LZMADecompressor`, you must + create a new decompressor for each stream. + + .. method:: decompress(data) + + Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` + object containing the decompressed data for at least part of the input. + Some of *data* may be buffered internally, for use in later calls to + :meth:`decompress`. The returned data should be concatenated with the + output of any previous calls to :meth:`decompress`. + + .. attribute:: check + + The ID of the integrity check used by the input stream. This may be + :const:`CHECK_UNKNOWN` until enough of the input has been decoded to + determine what integrity check it uses. + + .. attribute:: eof + + True if the end-of-stream marker has been reached. + + .. attribute:: unused_data + + Data found after the end of the compressed stream. + + Before the end of the stream is reached, this will be ``b""``. + + +.. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) + + Compress *data* (a :class:`bytes` object), returning the compressed data as a + :class:`bytes` object. + + See :class:`LZMACompressor` above for a description of the *format*, *check*, + *preset* and *filters* arguments. + + +.. function:: decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None) + + Decompress *data* (a :class:`bytes` object), returning the uncompressed data + as a :class:`bytes` object. + + If *data* is the concatenation of multiple distinct compressed streams, + decompress all of these streams, and return the concatenation of the results. + + See :class:`LZMADecompressor` above for a description of the *format*, + *memlimit* and *filters* arguments. + + +Miscellaneous +------------- + +.. function:: check_is_supported(check) + + Returns true if the given integrity check is supported on this system. + + :const:`CHECK_NONE` and :const:`CHECK_CRC32` are always supported. + :const:`CHECK_CRC64` and :const:`CHECK_SHA256` may be unavailable if you are + using a version of :program:`liblzma` that was compiled with a limited + feature set. + + +.. _filter-chain-specs: + +Specifying custom filter chains +------------------------------- + +A filter chain specifier is a sequence of dictionaries, where each dictionary +contains the ID and options for a single filter. Each dictionary must contain +the key ``"id"``, and may contain additional keys to specify filter-dependent +options. Valid filter IDs are as follows: + +* Compression filters: + * :const:`FILTER_LZMA1` (for use with :const:`FORMAT_ALONE`) + * :const:`FILTER_LZMA2` (for use with :const:`FORMAT_XZ` and :const:`FORMAT_RAW`) + +* Delta filter: + * :const:`FILTER_DELTA` + +* Branch-Call-Jump (BCJ) filters: + * :const:`FILTER_X86` + * :const:`FILTER_IA64` + * :const:`FILTER_ARM` + * :const:`FILTER_ARMTHUMB` + * :const:`FILTER_POWERPC` + * :const:`FILTER_SPARC` + +A filter chain can consist of up to 4 filters, and cannot be empty. The last +filter in the chain must be a compression filter, and any other filters must be +delta or BCJ filters. + +Compression filters support the following options (specified as additional +entries in the dictionary representing the filter): + + * ``preset``: A compression preset to use as a source of default values for + options that are not specified explicitly. + * ``dict_size``: Dictionary size in bytes. This should be between 4KiB and + 1.5GiB (inclusive). + * ``lc``: Number of literal context bits. + * ``lp``: Number of literal position bits. The sum ``lc + lp`` must be at + most 4. + * ``pb``: Number of position bits; must be at most 4. + * ``mode``: :const:`MODE_FAST` or :const:`MODE_NORMAL`. + * ``nice_len``: What should be considered a "nice length" for a match. + This should be 273 or less. + * ``mf``: What match finder to use -- :const:`MF_HC3`, :const:`MF_HC4`, + :const:`MF_BT2`, :const:`MF_BT3`, or :const:`MF_BT4`. + * ``depth``: Maximum search depth used by match finder. 0 (default) means to + select automatically based on other filter options. + +The delta filter stores the differences between bytes, producing more repetitive +input for the compressor in certain circumstances. It only supports a single +The delta filter supports only one option, ``dist``. This indicates the distance +between bytes to be subtracted. The default is 1, i.e. take the differences +between adjacent bytes. + +The BCJ filters are intended to be applied to machine code. They convert +relative branches, calls and jumps in the code to use absolute addressing, with +the aim of increasing the redundancy that can be exploited by the compressor. +These filters support one option, ``start_offset``. This specifies the address +that should be mapped to the beginning of the input data. The default is 0. + + +Examples +-------- + +Reading in a compressed file:: + + import lzma + with lzma.LZMAFile("file.xz") as f: + file_content = f.read() + +Creating a compressed file:: + + import lzma + data = b"Insert Data Here" + with lzma.LZMAFile("file.xz", "w") as f: + f.write(data) + +Compressing data in memory:: + + import lzma + data_in = b"Insert Data Here" + data_out = lzma.compress(data_in) + +Incremental compression:: + + import lzma + lzc = lzma.LZMACompressor() + out1 = lzc.compress(b"Some data\n") + out2 = lzc.compress(b"Another piece of data\n") + out3 = lzc.compress(b"Even more data\n") + out4 = lzc.flush() + # Concatenate all the partial results: + result = b"".join([out1, out2, out3, out4]) + +Writing compressed data to an already-open file:: + + import lzma + with open("file.xz", "wb") as f: + f.write(b"This data will not be compressed\n") + with lzma.LZMAFile(fileobj=f, mode="w") as lzf: + lzf.write(b"This *will* be compressed\n") + f.write(b"Not compressed\n") + +Creating a compressed file using a custom filter chain:: + + import lzma + my_filters = [ + {"id": lzma.FILTER_DELTA, "dist": 5}, + {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME}, + ] + with lzma.LZMAFile("file.xz", "w", filters=my_filters) as f: + f.write(b"blah blah blah") diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -23,7 +23,7 @@ create an encrypted file. Decryption is extremely slow as it is implemented in native Python rather than C. -For other archive formats, see the :mod:`bz2`, :mod:`gzip`, and +For related file formats, see the :mod:`bz2`, :mod:`gzip`, :mod:`lzma`, and :mod:`tarfile` modules. The module defines the following items: diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -18,8 +18,8 @@ consult the zlib manual at http://www.zlib.net/manual.html for authoritative information. -For reading and writing ``.gz`` files see the :mod:`gzip` module. For -other related file formats, see the :mod:`bz2`, :mod:`zipfile`, and +For reading and writing ``.gz`` files see the :mod:`gzip` module. For other +related file formats, see the :mod:`bz2`, :mod:`lzma`, :mod:`zipfile`, and :mod:`tarfile` modules. The available exception and functions in this module are: diff --git a/Lib/lzma.py b/Lib/lzma.py new file mode 100644 --- /dev/null +++ b/Lib/lzma.py @@ -0,0 +1,398 @@ +"""Interface to the liblzma compression library. + +This module provides a class for reading and writing compressed files, +classes for incremental (de)compression, and convenience functions for +one-shot (de)compression. + +These classes and functions support both the XZ and legacy LZMA +container formats, as well as raw compressed data streams. +""" + +__all__ = [ + "CHECK_NONE", "CHECK_CRC32", "CHECK_CRC64", "CHECK_SHA256", + "CHECK_ID_MAX", "CHECK_UNKNOWN", + "FILTER_LZMA1", "FILTER_LZMA2", "FILTER_DELTA", "FILTER_X86", "FILTER_IA64", + "FILTER_ARM", "FILTER_ARMTHUMB", "FILTER_POWERPC", "FILTER_SPARC", + "FORMAT_AUTO", "FORMAT_XZ", "FORMAT_ALONE", "FORMAT_RAW", + "MF_HC3", "MF_HC4", "MF_BT2", "MF_BT3", "MF_BT4", + "MODE_FAST", "MODE_NORMAL", "PRESET_DEFAULT", "PRESET_EXTREME", + + "LZMACompressor", "LZMADecompressor", "LZMAFile", "LZMAError", + "compress", "decompress", "check_is_supported", +] + +import io +from _lzma import * + + +_MODE_CLOSED = 0 +_MODE_READ = 1 +_MODE_READ_EOF = 2 +_MODE_WRITE = 3 + +_BUFFER_SIZE = 8192 + + +class LZMAFile(io.BufferedIOBase): + + """A file object providing transparent LZMA (de)compression. + + An LZMAFile can act as a wrapper for an existing file object, or + refer directly to a named file on disk. + + Note that LZMAFile provides a *binary* file interface - data read + is returned as bytes, and data to be written must be given as bytes. + """ + + def __init__(self, filename=None, mode="r", *, + fileobj=None, format=None, check=-1, + preset=None, filters=None): + """Open an LZMA-compressed file. + + If filename is given, open the named file. Otherwise, operate on + the file object given by fileobj. Exactly one of these two + parameters should be provided. + + mode can be "r" for reading (default), "w" for (over)writing, or + "a" for appending. + + format specifies the container format to use for the file. + If mode is "r", this defaults to FORMAT_AUTO. Otherwise, the + default is FORMAT_XZ. + + check specifies the integrity check to use. This argument can + only be used when opening a file for writing. For FORMAT_XZ, + the default is CHECK_CRC64. FORMAT_ALONE and FORMAT_RAW do not + support integrity checks - for these formats, check must be + omitted, or be CHECK_NONE. + + When opening a file for reading, the *preset* argument is not + meaningful, and should be omitted. The *filters* argument should + also be omitted, except when format is FORMAT_RAW (in which case + it is required). + + When opening a file for writing, the settings used by the + compressor can be specified either as a preset compression + level (with the *preset* argument), or in detail as a custom + filter chain (with the *filters* argument). For FORMAT_XZ and + FORMAT_ALONE, the default is to use the PRESET_DEFAULT preset + level. For FORMAT_RAW, the caller must always specify a filter + chain; the raw compressor does not support preset compression + levels. + + preset (if provided) should be an integer in the range 0-9, + optionally OR-ed with the constant PRESET_EXTREME. + + filters (if provided) should be a sequence of dicts. Each dict + should have an entry for "id" indicating ID of the filter, plus + additional entries for options to the filter. + """ + self._fp = None + self._closefp = False + self._mode = _MODE_CLOSED + self._pos = 0 + self._size = -1 + + if mode == "r": + if check != -1: + raise ValueError("Cannot specify an integrity check " + "when opening a file for reading") + if preset is not None: + raise ValueError("Cannot specify a preset compression " + "level when opening a file for reading") + if format is None: + format = FORMAT_AUTO + mode_code = _MODE_READ + # Save the args to pass to the LZMADecompressor initializer. + # If the file contains multiple compressed streams, each + # stream will need a separate decompressor object. + self._init_args = {"format":format, "filters":filters} + self._decompressor = LZMADecompressor(**self._init_args) + self._buffer = None + elif mode in ("w", "a"): + if format is None: + format = FORMAT_XZ + mode_code = _MODE_WRITE + self._compressor = LZMACompressor(format=format, check=check, + preset=preset, filters=filters) + else: + raise ValueError("Invalid mode: {!r}".format(mode)) + + if filename is not None and fileobj is None: + mode += "b" + self._fp = open(filename, mode) + self._closefp = True + self._mode = mode_code + elif fileobj is not None and filename is None: + self._fp = fileobj + self._mode = mode_code + else: + raise ValueError("Must give exactly one of filename and fileobj") + + def close(self): + """Flush and close the file. + + May be called more than once without error. Once the file is + closed, any other operation on it will raise a ValueError. + """ + if self._mode == _MODE_CLOSED: + return + try: + if self._mode in (_MODE_READ, _MODE_READ_EOF): + self._decompressor = None + self._buffer = None + elif self._mode == _MODE_WRITE: + self._fp.write(self._compressor.flush()) + self._compressor = None + finally: + try: + if self._closefp: + self._fp.close() + finally: + self._fp = None + self._closefp = False + self._mode = _MODE_CLOSED + + @property + def closed(self): + """True if this file is closed.""" + return self._mode == _MODE_CLOSED + + def fileno(self): + """Return the file descriptor for the underlying file.""" + self._check_not_closed() + return self._fp.fileno() + + def seekable(self): + """Return whether the file supports seeking.""" + return self.readable() + + def readable(self): + """Return whether the file was opened for reading.""" + self._check_not_closed() + return self._mode in (_MODE_READ, _MODE_READ_EOF) + + def writable(self): + """Return whether the file was opened for writing.""" + self._check_not_closed() + return self._mode == _MODE_WRITE + + # Mode-checking helper functions. + + def _check_not_closed(self): + if self.closed: + raise ValueError("I/O operation on closed file") + + def _check_can_read(self): + if not self.readable(): + raise io.UnsupportedOperation("File not open for reading") + + def _check_can_write(self): + if not self.writable(): + raise io.UnsupportedOperation("File not open for writing") + + def _check_can_seek(self): + if not self.seekable(): + raise io.UnsupportedOperation("Seeking is only supported " + "on files open for reading") + + # Fill the readahead buffer if it is empty. Returns False on EOF. + def _fill_buffer(self): + if self._buffer: + return True + + if self._decompressor.unused_data: + rawblock = self._decompressor.unused_data + else: + rawblock = self._fp.read(_BUFFER_SIZE) + + if not rawblock: + if self._decompressor.eof: + self._mode = _MODE_READ_EOF + self._size = self._pos + return False + else: + raise EOFError("Compressed file ended before the " + "end-of-stream marker was reached") + + # Continue to next stream. + if self._decompressor.eof: + self._decompressor = LZMADecompressor(**self._init_args) + + self._buffer = self._decompressor.decompress(rawblock) + return True + + # Read data until EOF. + # If return_data is false, consume the data without returning it. + def _read_all(self, return_data=True): + blocks = [] + while self._fill_buffer(): + if return_data: + blocks.append(self._buffer) + self._pos += len(self._buffer) + self._buffer = None + if return_data: + return b"".join(blocks) + + # Read a block of up to n bytes. + # If return_data is false, consume the data without returning it. + def _read_block(self, n, return_data=True): + blocks = [] + while n > 0 and self._fill_buffer(): + if n < len(self._buffer): + data = self._buffer[:n] + self._buffer = self._buffer[n:] + else: + data = self._buffer + self._buffer = None + if return_data: + blocks.append(data) + self._pos += len(data) + n -= len(data) + if return_data: + return b"".join(blocks) + + def peek(self, size=-1): + """Return buffered data without advancing the file position. + + Always returns at least one byte of data, unless at EOF. + The exact number of bytes returned is unspecified. + """ + self._check_can_read() + if self._mode == _MODE_READ_EOF or not self._fill_buffer(): + return b"" + return self._buffer + + def read(self, size=-1): + """Read up to size uncompressed bytes from the file. + + If size is negative or omitted, read until EOF is reached. + Returns b"" if the file is already at EOF. + """ + self._check_can_read() + if self._mode == _MODE_READ_EOF or size == 0: + return b"" + elif size < 0: + return self._read_all() + else: + return self._read_block(size) + + def read1(self, size=-1): + """Read up to size uncompressed bytes with at most one read + from the underlying stream. + + Returns b"" if the file is at EOF. + """ + self._check_can_read() + if (size == 0 or self._mode == _MODE_READ_EOF or + not self._fill_buffer()): + return b"" + if 0 < size < len(self._buffer): + data = self._buffer[:size] + self._buffer = self._buffer[size:] + else: + data = self._buffer + self._buffer = None + self._pos += len(data) + return data + + def write(self, data): + """Write a bytes object to the file. + + Returns the number of uncompressed bytes written, which is + always len(data). Note that due to buffering, the file on disk + may not reflect the data written until close() is called. + """ + self._check_can_write() + compressed = self._compressor.compress(data) + self._fp.write(compressed) + self._pos += len(data) + return len(data) + + # Rewind the file to the beginning of the data stream. + def _rewind(self): + self._fp.seek(0, 0) + self._mode = _MODE_READ + self._pos = 0 + self._decompressor = LZMADecompressor(**self._init_args) + self._buffer = None + + def seek(self, offset, whence=0): + """Change the file position. + + The new position is specified by offset, relative to the + position indicated by whence. Possible values for whence are: + + 0: start of stream (default): offset must not be negative + 1: current stream position + 2: end of stream; offset must not be positive + + Returns the new file position. + + Note that seeking is emulated, sp depending on the parameters, + this operation may be extremely slow. + """ + self._check_can_seek() + + # Recalculate offset as an absolute file position. + if whence == 0: + pass + elif whence == 1: + offset = self._pos + offset + elif whence == 2: + # Seeking relative to EOF - we need to know the file's size. + if self._size < 0: + self._read_all(return_data=False) + offset = self._size + offset + else: + raise ValueError("Invalid value for whence: {}".format(whence)) + + # Make it so that offset is the number of bytes to skip forward. + if offset < self._pos: + self._rewind() + else: + offset -= self._pos + + # Read and discard data until we reach the desired position. + if self._mode != _MODE_READ_EOF: + self._read_block(offset, return_data=False) + + return self._pos + + def tell(self): + """Return the current file position.""" + self._check_not_closed() + return self._pos + + +def compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None): + """Compress a block of data. + + Refer to LZMACompressor's docstring for a description of the + optional arguments *format*, *check*, *preset* and *filters*. + + For incremental compression, use an LZMACompressor object instead. + """ + comp = LZMACompressor(format, check, preset, filters) + return comp.compress(data) + comp.flush() + + +def decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None): + """Decompress a block of data. + + Refer to LZMADecompressor's docstring for a description of the + optional arguments *format*, *check* and *filters*. + + For incremental decompression, use a LZMADecompressor object instead. + """ + results = [] + while True: + decomp = LZMADecompressor(format, memlimit, filters) + results.append(decomp.decompress(data)) + if not decomp.eof: + raise LZMAError("Compressed data ended before the " + "end-of-stream marker was reached") + if not decomp.unused_data: + return b"".join(results) + # There is unused data left over. Proceed to next stream. + data = decomp.unused_data diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_lzma.py @@ -0,0 +1,1335 @@ +from io import BytesIO, UnsupportedOperation +import os +import random +import unittest + +from test.support import ( + _4G, TESTFN, import_module, bigmemtest, run_unittest, unlink +) + +lzma = import_module("lzma") +from lzma import LZMACompressor, LZMADecompressor, LZMAError, LZMAFile + + +class CompressorDecompressorTestCase(unittest.TestCase): + + # Test error cases. + + def test_simple_bad_args(self): + self.assertRaises(TypeError, LZMACompressor, []) + self.assertRaises(TypeError, LZMACompressor, format=3.45) + self.assertRaises(TypeError, LZMACompressor, check="") + self.assertRaises(TypeError, LZMACompressor, preset="asdf") + self.assertRaises(TypeError, LZMACompressor, filters=3) + # Can't specify FORMAT_AUTO when compressing. + self.assertRaises(ValueError, LZMACompressor, format=lzma.FORMAT_AUTO) + # Can't specify a preset and a custom filter chain at the same time. + with self.assertRaises(ValueError): + LZMACompressor(preset=7, filters=[{"id": lzma.FILTER_LZMA2}]) + + self.assertRaises(TypeError, LZMADecompressor, ()) + self.assertRaises(TypeError, LZMADecompressor, memlimit=b"qw") + with self.assertRaises(TypeError): + LZMADecompressor(lzma.FORMAT_RAW, filters="zzz") + # Cannot specify a memory limit with FILTER_RAW. + with self.assertRaises(ValueError): + LZMADecompressor(lzma.FORMAT_RAW, memlimit=0x1000000) + # Can only specify a custom filter chain with FILTER_RAW. + self.assertRaises(ValueError, LZMADecompressor, filters=FILTERS_RAW_1) + with self.assertRaises(ValueError): + LZMADecompressor(format=lzma.FORMAT_XZ, filters=FILTERS_RAW_1) + with self.assertRaises(ValueError): + LZMADecompressor(format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1) + + lzc = LZMACompressor() + self.assertRaises(TypeError, lzc.compress) + self.assertRaises(TypeError, lzc.compress, b"foo", b"bar") + self.assertRaises(TypeError, lzc.flush, b"blah") + empty = lzc.flush() + self.assertRaises(ValueError, lzc.compress, b"quux") + self.assertRaises(ValueError, lzc.flush) + + lzd = LZMADecompressor() + self.assertRaises(TypeError, lzd.decompress) + self.assertRaises(TypeError, lzd.decompress, b"foo", b"bar") + lzd.decompress(empty) + self.assertRaises(EOFError, lzd.decompress, b"quux") + + def test_bad_filter_spec(self): + self.assertRaises(TypeError, LZMACompressor, filters=[b"wobsite"]) + self.assertRaises(ValueError, LZMACompressor, filters=[{"xyzzy": 3}]) + self.assertRaises(ValueError, LZMACompressor, filters=[{"id": 98765}]) + with self.assertRaises(ValueError): + LZMACompressor(filters=[{"id": lzma.FILTER_LZMA2, "foo": 0}]) + with self.assertRaises(ValueError): + LZMACompressor(filters=[{"id": lzma.FILTER_DELTA, "foo": 0}]) + with self.assertRaises(ValueError): + LZMACompressor(filters=[{"id": lzma.FILTER_X86, "foo": 0}]) + + def test_decompressor_after_eof(self): + lzd = LZMADecompressor() + lzd.decompress(COMPRESSED_XZ) + self.assertRaises(EOFError, lzd.decompress, b"nyan") + + def test_decompressor_memlimit(self): + lzd = LZMADecompressor(memlimit=1024) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) + + lzd = LZMADecompressor(lzma.FORMAT_XZ, memlimit=1024) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) + + lzd = LZMADecompressor(lzma.FORMAT_ALONE, memlimit=1024) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_ALONE) + + # Test LZMADecompressor on known-good input data. + + def _test_decompressor(self, lzd, data, check, unused_data=b""): + self.assertFalse(lzd.eof) + out = lzd.decompress(data) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, check) + self.assertTrue(lzd.eof) + self.assertEqual(lzd.unused_data, unused_data) + + def test_decompressor_auto(self): + lzd = LZMADecompressor() + self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64) + + lzd = LZMADecompressor() + self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE) + + def test_decompressor_xz(self): + lzd = LZMADecompressor(lzma.FORMAT_XZ) + self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64) + + def test_decompressor_alone(self): + lzd = LZMADecompressor(lzma.FORMAT_ALONE) + self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE) + + def test_decompressor_raw_1(self): + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1) + self._test_decompressor(lzd, COMPRESSED_RAW_1, lzma.CHECK_NONE) + + def test_decompressor_raw_2(self): + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_2) + self._test_decompressor(lzd, COMPRESSED_RAW_2, lzma.CHECK_NONE) + + def test_decompressor_raw_3(self): + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_3) + self._test_decompressor(lzd, COMPRESSED_RAW_3, lzma.CHECK_NONE) + + def test_decompressor_raw_4(self): + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self._test_decompressor(lzd, COMPRESSED_RAW_4, lzma.CHECK_NONE) + + def test_decompressor_chunks(self): + lzd = LZMADecompressor() + out = [] + for i in range(0, len(COMPRESSED_XZ), 10): + self.assertFalse(lzd.eof) + out.append(lzd.decompress(COMPRESSED_XZ[i:i+10])) + out = b"".join(out) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, lzma.CHECK_CRC64) + self.assertTrue(lzd.eof) + self.assertEqual(lzd.unused_data, b"") + + def test_decompressor_unused_data(self): + lzd = LZMADecompressor() + extra = b"fooblibar" + self._test_decompressor(lzd, COMPRESSED_XZ + extra, lzma.CHECK_CRC64, + unused_data=extra) + + def test_decompressor_bad_input(self): + lzd = LZMADecompressor() + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_RAW_1) + + lzd = LZMADecompressor(lzma.FORMAT_XZ) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_ALONE) + + lzd = LZMADecompressor(lzma.FORMAT_ALONE) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) + + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1) + self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ) + + # Test that LZMACompressor->LZMADecompressor preserves the input data. + + def test_roundtrip_xz(self): + lzc = LZMACompressor() + cdata = lzc.compress(INPUT) + lzc.flush() + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + + def test_roundtrip_alone(self): + lzc = LZMACompressor(lzma.FORMAT_ALONE) + cdata = lzc.compress(INPUT) + lzc.flush() + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + + def test_roundtrip_raw(self): + lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + cdata = lzc.compress(INPUT) + lzc.flush() + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + + def test_roundtrip_chunks(self): + lzc = LZMACompressor() + cdata = [] + for i in range(0, len(INPUT), 10): + cdata.append(lzc.compress(INPUT[i:i+10])) + cdata.append(lzc.flush()) + cdata = b"".join(cdata) + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + + # LZMADecompressor intentionally does not handle concatenated streams. + + def test_decompressor_multistream(self): + lzd = LZMADecompressor() + self._test_decompressor(lzd, COMPRESSED_XZ + COMPRESSED_ALONE, + lzma.CHECK_CRC64, unused_data=COMPRESSED_ALONE) + + # Test with inputs larger than 4GiB. + + @bigmemtest(size=_4G + 100, memuse=2) + def test_compressor_bigmem(self, size): + lzc = LZMACompressor() + cdata = lzc.compress(b"x" * size) + lzc.flush() + ddata = lzma.decompress(cdata) + try: + self.assertEqual(len(ddata), size) + self.assertEqual(len(ddata.strip(b"x")), 0) + finally: + ddata = None + + @bigmemtest(size=_4G + 100, memuse=3) + def test_decompressor_bigmem(self, size): + lzd = LZMADecompressor() + blocksize = 10 * 1024 * 1024 + block = random.getrandbits(blocksize * 8).to_bytes(blocksize, "little") + try: + input = block * (size // blocksize + 1) + cdata = lzma.compress(input) + ddata = lzd.decompress(cdata) + self.assertEqual(ddata, input) + finally: + input = cdata = ddata = None + + +class CompressDecompressFunctionTestCase(unittest.TestCase): + + # Test error cases: + + def test_bad_args(self): + self.assertRaises(TypeError, lzma.compress) + self.assertRaises(TypeError, lzma.compress, []) + self.assertRaises(TypeError, lzma.compress, b"", format="xz") + self.assertRaises(TypeError, lzma.compress, b"", check="none") + self.assertRaises(TypeError, lzma.compress, b"", preset="blah") + self.assertRaises(TypeError, lzma.compress, b"", filters=1024) + # Can't specify a preset and a custom filter chain at the same time. + with self.assertRaises(ValueError): + lzma.compress(b"", preset=3, filters=[{"id": lzma.FILTER_LZMA2}]) + + self.assertRaises(TypeError, lzma.decompress) + self.assertRaises(TypeError, lzma.decompress, []) + self.assertRaises(TypeError, lzma.decompress, b"", format="lzma") + self.assertRaises(TypeError, lzma.decompress, b"", memlimit=7.3e9) + with self.assertRaises(TypeError): + lzma.decompress(b"", format=lzma.FORMAT_RAW, filters={}) + # Cannot specify a memory limit with FILTER_RAW. + with self.assertRaises(ValueError): + lzma.decompress(b"", format=lzma.FORMAT_RAW, memlimit=0x1000000) + # Can only specify a custom filter chain with FILTER_RAW. + with self.assertRaises(ValueError): + lzma.decompress(b"", filters=FILTERS_RAW_1) + with self.assertRaises(ValueError): + lzma.decompress(b"", format=lzma.FORMAT_XZ, filters=FILTERS_RAW_1) + with self.assertRaises(ValueError): + lzma.decompress( + b"", format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1) + + def test_decompress_memlimit(self): + with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_XZ, memlimit=1024) + with self.assertRaises(LZMAError): + lzma.decompress( + COMPRESSED_XZ, format=lzma.FORMAT_XZ, memlimit=1024) + with self.assertRaises(LZMAError): + lzma.decompress( + COMPRESSED_ALONE, format=lzma.FORMAT_ALONE, memlimit=1024) + + # Test LZMADecompressor on known-good input data. + + def test_decompress_good_input(self): + ddata = lzma.decompress(COMPRESSED_XZ) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress(COMPRESSED_ALONE) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress(COMPRESSED_XZ, lzma.FORMAT_XZ) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress(COMPRESSED_ALONE, lzma.FORMAT_ALONE) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress( + COMPRESSED_RAW_1, lzma.FORMAT_RAW, filters=FILTERS_RAW_1) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress( + COMPRESSED_RAW_2, lzma.FORMAT_RAW, filters=FILTERS_RAW_2) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress( + COMPRESSED_RAW_3, lzma.FORMAT_RAW, filters=FILTERS_RAW_3) + self.assertEqual(ddata, INPUT) + + ddata = lzma.decompress( + COMPRESSED_RAW_4, lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self.assertEqual(ddata, INPUT) + + def test_decompress_incomplete_input(self): + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_XZ[:128]) + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_ALONE[:128]) + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_1[:128], + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_1) + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_2[:128], + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_3[:128], + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) + self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_4[:128], + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + + def test_decompress_bad_input(self): + with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_RAW_1) + with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_ALONE, format=lzma.FORMAT_XZ) + with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_XZ, format=lzma.FORMAT_ALONE) + with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_XZ, format=lzma.FORMAT_RAW, + filters=FILTERS_RAW_1) + + # Test that compress()->decompress() preserves the input data. + + def test_roundtrip(self): + cdata = lzma.compress(INPUT) + ddata = lzma.decompress(cdata) + self.assertEqual(ddata, INPUT) + + cdata = lzma.compress(INPUT, lzma.FORMAT_XZ) + ddata = lzma.decompress(cdata) + self.assertEqual(ddata, INPUT) + + cdata = lzma.compress(INPUT, lzma.FORMAT_ALONE) + ddata = lzma.decompress(cdata) + self.assertEqual(ddata, INPUT) + + cdata = lzma.compress(INPUT, lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + ddata = lzma.decompress(cdata, lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self.assertEqual(ddata, INPUT) + + # Unlike LZMADecompressor, decompress() *does* handle concatenated streams. + + def test_decompress_multistream(self): + ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_ALONE) + self.assertEqual(ddata, INPUT * 2) + + +class TempFile: + """Context manager - creates a file, and deletes it on __exit__.""" + + def __init__(self, filename, data=b""): + self.filename = filename + self.data = data + + def __enter__(self): + with open(self.filename, "wb") as f: + f.write(self.data) + + def __exit__(self, *args): + unlink(self.filename) + + +class FileTestCase(unittest.TestCase): + + def test_init(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + pass + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + pass + with LZMAFile(fileobj=BytesIO(), mode="a") as f: + pass + + def test_init_with_filename(self): + with TempFile(TESTFN, COMPRESSED_XZ): + with LZMAFile(TESTFN) as f: + pass + with LZMAFile(TESTFN, "w") as f: + pass + with LZMAFile(TESTFN, "a") as f: + pass + + def test_init_bad_mode(self): + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode=(3, "x")) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="x") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="rb") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="r+") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="wb") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="w+") + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="rw") + + def test_init_bad_check(self): + with self.assertRaises(TypeError): + LZMAFile(fileobj=BytesIO(), mode="w", check=b"asd") + # CHECK_UNKNOWN and anything above CHECK_ID_MAX should be invalid. + with self.assertRaises(LZMAError): + LZMAFile(fileobj=BytesIO(), mode="w", check=lzma.CHECK_UNKNOWN) + with self.assertRaises(LZMAError): + LZMAFile(fileobj=BytesIO(), mode="w", check=lzma.CHECK_ID_MAX + 3) + # Cannot specify a check with mode="r". + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_NONE) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_CRC32) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_CRC64) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_SHA256) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_UNKNOWN) + + def test_init_bad_preset(self): + with self.assertRaises(TypeError): + LZMAFile(fileobj=BytesIO(), mode="w", preset=4.39) + with self.assertRaises(LZMAError): + LZMAFile(fileobj=BytesIO(), mode="w", preset=10) + with self.assertRaises(LZMAError): + LZMAFile(fileobj=BytesIO(), mode="w", preset=23) + with self.assertRaises(OverflowError): + LZMAFile(fileobj=BytesIO(), mode="w", preset=-1) + with self.assertRaises(OverflowError): + LZMAFile(fileobj=BytesIO(), mode="w", preset=-7) + with self.assertRaises(TypeError): + LZMAFile(fileobj=BytesIO(), mode="w", preset="foo") + # Cannot specify a preset with mode="r". + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), preset=3) + + def test_init_bad_filter_spec(self): + with self.assertRaises(TypeError): + LZMAFile(fileobj=BytesIO(), mode="w", filters=[b"wobsite"]) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", filters=[{"xyzzy": 3}]) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", filters=[{"id": 98765}]) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", + filters=[{"id": lzma.FILTER_LZMA2, "foo": 0}]) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", + filters=[{"id": lzma.FILTER_DELTA, "foo": 0}]) + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", + filters=[{"id": lzma.FILTER_X86, "foo": 0}]) + + def test_init_with_preset_and_filters(self): + with self.assertRaises(ValueError): + LZMAFile(fileobj=BytesIO(), mode="w", format=lzma.FORMAT_RAW, + preset=6, filters=FILTERS_RAW_1) + + def test_init_with_filename_and_fileobj(self): + with self.assertRaises(ValueError): + LZMAFile("/dev/null", fileobj=BytesIO()) + + def test_close(self): + with BytesIO(COMPRESSED_XZ) as src: + f = LZMAFile(fileobj=src) + f.close() + # LZMAFile.close() should not close the underlying file object. + self.assertFalse(src.closed) + # Try closing an already-closed LZMAFile. + f.close() + self.assertFalse(src.closed) + + # Test with a real file on disk, opened directly by LZMAFile. + with TempFile(TESTFN, COMPRESSED_XZ): + f = LZMAFile(TESTFN) + fp = f._fp + f.close() + # Here, LZMAFile.close() *should* close the underlying file object. + self.assertTrue(fp.closed) + # Try closing an already-closed LZMAFile. + f.close() + + def test_closed(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + try: + self.assertFalse(f.closed) + f.read() + self.assertFalse(f.closed) + finally: + f.close() + self.assertTrue(f.closed) + + f = LZMAFile(fileobj=BytesIO(), mode="w") + try: + self.assertFalse(f.closed) + finally: + f.close() + self.assertTrue(f.closed) + + def test_fileno(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + try: + self.assertRaises(UnsupportedOperation, f.fileno) + finally: + f.close() + self.assertRaises(ValueError, f.fileno) + with TempFile(TESTFN, COMPRESSED_XZ): + f = LZMAFile(TESTFN) + try: + self.assertEqual(f.fileno(), f._fp.fileno()) + self.assertIsInstance(f.fileno(), int) + finally: + f.close() + self.assertRaises(ValueError, f.fileno) + + def test_seekable(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + try: + self.assertTrue(f.seekable()) + f.read() + self.assertTrue(f.seekable()) + finally: + f.close() + self.assertRaises(ValueError, f.seekable) + + f = LZMAFile(fileobj=BytesIO(), mode="w") + try: + self.assertFalse(f.seekable()) + finally: + f.close() + self.assertRaises(ValueError, f.seekable) + + def test_readable(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + try: + self.assertTrue(f.readable()) + f.read() + self.assertTrue(f.readable()) + finally: + f.close() + self.assertRaises(ValueError, f.readable) + + f = LZMAFile(fileobj=BytesIO(), mode="w") + try: + self.assertFalse(f.readable()) + finally: + f.close() + self.assertRaises(ValueError, f.readable) + + def test_writable(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + try: + self.assertFalse(f.writable()) + f.read() + self.assertFalse(f.writable()) + finally: + f.close() + self.assertRaises(ValueError, f.writable) + + f = LZMAFile(fileobj=BytesIO(), mode="w") + try: + self.assertTrue(f.writable()) + finally: + f.close() + self.assertRaises(ValueError, f.writable) + + def test_read(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f: + self.assertEqual(f.read(), INPUT) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), + format=lzma.FORMAT_XZ) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE), + format=lzma.FORMAT_ALONE) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_1), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_1) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_2), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_3), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_4), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_4) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + + def test_read_0(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertEqual(f.read(0), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f: + self.assertEqual(f.read(0), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), + format=lzma.FORMAT_XZ) as f: + self.assertEqual(f.read(0), b"") + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE), + format=lzma.FORMAT_ALONE) as f: + self.assertEqual(f.read(0), b"") + + def test_read_10(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + chunks = [] + while True: + result = f.read(10) + if not result: + break + self.assertLessEqual(len(result), 10) + chunks.append(result) + self.assertEqual(b"".join(chunks), INPUT) + + def test_read_multistream(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 5)) as f: + self.assertEqual(f.read(), INPUT * 5) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ + COMPRESSED_ALONE)) as f: + self.assertEqual(f.read(), INPUT * 2) + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_3 * 4), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) as f: + self.assertEqual(f.read(), INPUT * 4) + + def test_read_multistream_buffer_size_aligned(self): + # Test the case where a stream boundary coincides with the end + # of the raw read buffer. + saved_buffer_size = lzma._BUFFER_SIZE + lzma._BUFFER_SIZE = len(COMPRESSED_XZ) + try: + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 5)) as f: + self.assertEqual(f.read(), INPUT * 5) + finally: + lzma._BUFFER_SIZE = saved_buffer_size + + def test_read_from_file(self): + with TempFile(TESTFN, COMPRESSED_XZ): + with LZMAFile(TESTFN) as f: + self.assertEqual(f.read(), INPUT) + self.assertEqual(f.read(), b"") + + def test_read_incomplete(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ[:128])) as f: + self.assertRaises(EOFError, f.read) + + def test_read_bad_args(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + f.close() + self.assertRaises(ValueError, f.read) + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + self.assertRaises(ValueError, f.read) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertRaises(TypeError, f.read, None) + + def test_read1(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + blocks = [] + while True: + result = f.read1() + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), INPUT) + self.assertEqual(f.read1(), b"") + + def test_read1_0(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertEqual(f.read1(0), b"") + + def test_read1_10(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + blocks = [] + while True: + result = f.read1(10) + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), INPUT) + self.assertEqual(f.read1(), b"") + + def test_read1_multistream(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 5)) as f: + blocks = [] + while True: + result = f.read1() + if not result: + break + blocks.append(result) + self.assertEqual(b"".join(blocks), INPUT * 5) + self.assertEqual(f.read1(), b"") + + def test_read1_bad_args(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + f.close() + self.assertRaises(ValueError, f.read1) + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + self.assertRaises(ValueError, f.read1) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertRaises(TypeError, f.read1, None) + + def test_peek(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + result = f.peek() + self.assertGreater(len(result), 0) + self.assertTrue(INPUT.startswith(result)) + self.assertEqual(f.read(), INPUT) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + result = f.peek(10) + self.assertGreater(len(result), 0) + self.assertTrue(INPUT.startswith(result)) + self.assertEqual(f.read(), INPUT) + + def test_peek_bad_args(self): + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + self.assertRaises(ValueError, f.peek) + + def test_iterator(self): + with BytesIO(INPUT) as f: + lines = f.readlines() + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertListEqual(list(iter(f)), lines) + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f: + self.assertListEqual(list(iter(f)), lines) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), + format=lzma.FORMAT_XZ) as f: + self.assertListEqual(list(iter(f)), lines) + with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE), + format=lzma.FORMAT_ALONE) as f: + self.assertListEqual(list(iter(f)), lines) + with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_2), + format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f: + self.assertListEqual(list(iter(f)), lines) + + def test_readline(self): + with BytesIO(INPUT) as f: + lines = f.readlines() + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + for line in lines: + self.assertEqual(f.readline(), line) + + def test_readlines(self): + with BytesIO(INPUT) as f: + lines = f.readlines() + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertListEqual(f.readlines(), lines) + + def test_write(self): + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w") as f: + f.write(INPUT) + expected = lzma.compress(INPUT) + self.assertEqual(dst.getvalue(), expected) + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_XZ) as f: + f.write(INPUT) + expected = lzma.compress(INPUT, format=lzma.FORMAT_XZ) + self.assertEqual(dst.getvalue(), expected) + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_ALONE) as f: + f.write(INPUT) + expected = lzma.compress(INPUT, format=lzma.FORMAT_ALONE) + self.assertEqual(dst.getvalue(), expected) + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_RAW, + filters=FILTERS_RAW_2) as f: + f.write(INPUT) + expected = lzma.compress(INPUT, format=lzma.FORMAT_RAW, + filters=FILTERS_RAW_2) + self.assertEqual(dst.getvalue(), expected) + + def test_write_10(self): + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w") as f: + for start in range(0, len(INPUT), 10): + f.write(INPUT[start:start+10]) + expected = lzma.compress(INPUT) + self.assertEqual(dst.getvalue(), expected) + + def test_write_append(self): + part1 = INPUT[:1024] + part2 = INPUT[1024:1536] + part3 = INPUT[1536:] + expected = b"".join(lzma.compress(x) for x in (part1, part2, part3)) + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w") as f: + f.write(part1) + with LZMAFile(fileobj=dst, mode="a") as f: + f.write(part2) + with LZMAFile(fileobj=dst, mode="a") as f: + f.write(part3) + self.assertEqual(dst.getvalue(), expected) + + def test_write_to_file(self): + try: + with LZMAFile(TESTFN, "w") as f: + f.write(INPUT) + expected = lzma.compress(INPUT) + with open(TESTFN, "rb") as f: + self.assertEqual(f.read(), expected) + finally: + unlink(TESTFN) + + def test_write_append_to_file(self): + part1 = INPUT[:1024] + part2 = INPUT[1024:1536] + part3 = INPUT[1536:] + expected = b"".join(lzma.compress(x) for x in (part1, part2, part3)) + try: + with LZMAFile(TESTFN, "w") as f: + f.write(part1) + with LZMAFile(TESTFN, "a") as f: + f.write(part2) + with LZMAFile(TESTFN, "a") as f: + f.write(part3) + with open(TESTFN, "rb") as f: + self.assertEqual(f.read(), expected) + finally: + unlink(TESTFN) + + def test_write_bad_args(self): + f = LZMAFile(fileobj=BytesIO(), mode="w") + f.close() + self.assertRaises(ValueError, f.write, b"foo") + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="r") as f: + self.assertRaises(ValueError, f.write, b"bar") + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + self.assertRaises(TypeError, f.write, None) + self.assertRaises(TypeError, f.write, "text") + self.assertRaises(TypeError, f.write, 789) + + def test_writelines(self): + with BytesIO(INPUT) as f: + lines = f.readlines() + with BytesIO() as dst: + with LZMAFile(fileobj=dst, mode="w") as f: + f.writelines(lines) + expected = lzma.compress(INPUT) + self.assertEqual(dst.getvalue(), expected) + + def test_seek_forward(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.seek(555) + self.assertEqual(f.read(), INPUT[555:]) + + def test_seek_forward_across_streams(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 2)) as f: + f.seek(len(INPUT) + 123) + self.assertEqual(f.read(), INPUT[123:]) + + def test_seek_forward_relative_to_current(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.read(100) + f.seek(1236, 1) + self.assertEqual(f.read(), INPUT[1336:]) + + def test_seek_forward_relative_to_end(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.seek(-555, 2) + self.assertEqual(f.read(), INPUT[-555:]) + + def test_seek_backward(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.read(1001) + f.seek(211) + self.assertEqual(f.read(), INPUT[211:]) + + def test_seek_backward_across_streams(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 2)) as f: + f.read(len(INPUT) + 333) + f.seek(737) + self.assertEqual(f.read(), INPUT[737:] + INPUT) + + def test_seek_backward_relative_to_end(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.seek(-150, 2) + self.assertEqual(f.read(), INPUT[-150:]) + + def test_seek_past_end(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.seek(len(INPUT) + 9001) + self.assertEqual(f.tell(), len(INPUT)) + self.assertEqual(f.read(), b"") + + def test_seek_past_start(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + f.seek(-88) + self.assertEqual(f.tell(), 0) + self.assertEqual(f.read(), INPUT) + + def test_seek_bad_args(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + f.close() + self.assertRaises(ValueError, f.seek, 0) + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + self.assertRaises(ValueError, f.seek, 0) + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + self.assertRaises(ValueError, f.seek, 0, 3) + self.assertRaises(ValueError, f.seek, 9, ()) + self.assertRaises(TypeError, f.seek, None) + self.assertRaises(TypeError, f.seek, b"derp") + + def test_tell(self): + with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f: + pos = 0 + while True: + self.assertEqual(f.tell(), pos) + result = f.read(183) + if not result: + break + pos += len(result) + self.assertEqual(f.tell(), len(INPUT)) + with LZMAFile(fileobj=BytesIO(), mode="w") as f: + for pos in range(0, len(INPUT), 144): + self.assertEqual(f.tell(), pos) + f.write(INPUT[pos:pos+144]) + self.assertEqual(f.tell(), len(INPUT)) + + def test_tell_bad_args(self): + f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) + f.close() + self.assertRaises(ValueError, f.tell) + + +class MiscellaneousTestCase(unittest.TestCase): + + def test_is_check_supported(self): + # CHECK_NONE and CHECK_CRC32 should always be supported, + # regardless of the options liblzma was compiled with. + self.assertTrue(lzma.check_is_supported(lzma.CHECK_NONE)) + self.assertTrue(lzma.check_is_supported(lzma.CHECK_CRC32)) + + # The .xz format spec cannot store check IDs above this value. + self.assertFalse(lzma.check_is_supported(lzma.CHECK_ID_MAX + 1)) + + # This value should not be a valid check ID. + self.assertFalse(lzma.check_is_supported(lzma.CHECK_UNKNOWN)) + + +# Test data: + +INPUT = b""" +LAERTES + + O, fear me not. + I stay too long: but here my father comes. + + Enter POLONIUS + + A double blessing is a double grace, + Occasion smiles upon a second leave. + +LORD POLONIUS + + Yet here, Laertes! aboard, aboard, for shame! + The wind sits in the shoulder of your sail, + And you are stay'd for. There; my blessing with thee! + And these few precepts in thy memory + See thou character. Give thy thoughts no tongue, + Nor any unproportioned thought his act. + Be thou familiar, but by no means vulgar. + Those friends thou hast, and their adoption tried, + Grapple them to thy soul with hoops of steel; + But do not dull thy palm with entertainment + Of each new-hatch'd, unfledged comrade. Beware + Of entrance to a quarrel, but being in, + Bear't that the opposed may beware of thee. + Give every man thy ear, but few thy voice; + Take each man's censure, but reserve thy judgment. + Costly thy habit as thy purse can buy, + But not express'd in fancy; rich, not gaudy; + For the apparel oft proclaims the man, + And they in France of the best rank and station + Are of a most select and generous chief in that. + Neither a borrower nor a lender be; + For loan oft loses both itself and friend, + And borrowing dulls the edge of husbandry. + This above all: to thine ownself be true, + And it must follow, as the night the day, + Thou canst not then be false to any man. + Farewell: my blessing season this in thee! + +LAERTES + + Most humbly do I take my leave, my lord. + +LORD POLONIUS + + The time invites you; go; your servants tend. + +LAERTES + + Farewell, Ophelia; and remember well + What I have said to you. + +OPHELIA + + 'Tis in my memory lock'd, + And you yourself shall keep the key of it. + +LAERTES + + Farewell. +""" + +COMPRESSED_XZ = ( + b"\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3" + b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d" + b"o\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8\xe2\xfc\xe7\xd9\xfe6\xb8(" + b"\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02\x17/\xa6=\xf0\xa2\xdf/M\x89" + b"\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ\x15\x80\x8c\xf8\x8do\xfa\x12" + b"\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t\xca6 BF$\xe5Q\xa4\x98\xee\xde" + b"l\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81\xe4N\xc8\x86\x153\xf5x2\xa2O" + b"\x13@Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z\xc4\xcdS\xb6t<\x16\xf2\x9cI#" + b"\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0\xaa\x96-Pe\xade:\x04\t\x1b\xf7" + b"\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b" + b"\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa" + b"\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8\x84b\xf6\xc3\xd4c-H\x93oJl\xd0iQ\xe4k" + b"\x84\x0b\xc1\xb7\xbc\xb1\x17\x88\xb1\xca?@\xf6\x07\xea\xe6x\xf1H12P\x0f" + b"\x8a\xc9\xeauw\xe3\xbe\xaai\xa9W\xd0\x80\xcd#cb5\x99\xd8]\xa9d\x0c\xbd" + b"\xa2\xdcWl\xedUG\xbf\x89yF\xf77\x81v\xbd5\x98\xbeh8\x18W\x08\xf0\x1b\x99" + b"5:\x1a?rD\x96\xa1\x04\x0f\xae\xba\x85\xeb\x9d5@\xf5\x83\xd37\x83\x8ac" + b"\x06\xd4\x97i\xcdt\x16S\x82k\xf6K\x01vy\x88\x91\x9b6T\xdae\r\xfd]:k\xbal" + b"\xa9\xbba\xc34\xf9r\xeb}r\xdb\xc7\xdb*\x8f\x03z\xdc8h\xcc\xc9\xd3\xbcl" + b"\xa5-\xcb\xeaK\xa2\xc5\x15\xc0\xe3\xc1\x86Z\xfb\xebL\xe13\xcf\x9c\xe3" + b"\x1d\xc9\xed\xc2\x06\xcc\xce!\x92\xe5\xfe\x9c^\xa59w \x9bP\xa3PK\x08d" + b"\xf9\xe2Z}\xa7\xbf\xed\xeb%$\x0c\x82\xb8/\xb0\x01\xa9&,\xf7qh{Q\x96)\xf2" + b"q\x96\xc3\x80\xb4\x12\xb0\xba\xe6o\xf4!\xb4[\xd4\x8aw\x10\xf7t\x0c\xb3" + b"\xd9\xd5\xc3`^\x81\x11??\\\xa4\x99\x85R\xd4\x8e\x83\xc9\x1eX\xbfa\xf1" + b"\xac\xb0\xea\xea\xd7\xd0\xab\x18\xe2\xf2\xed\xe1\xb7\xc9\x18\xcbS\xe4>" + b"\xc9\x95H\xe8\xcb\t\r%\xeb\xc7$.o\xf1\xf3R\x17\x1db\xbb\xd8U\xa5^\xccS" + b"\x16\x01\x87\xf3/\x93\xd1\xf0v\xc0r\xd7\xcc\xa2Gkz\xca\x80\x0e\xfd\xd0" + b"\x8b\xbb\xd2Ix\xb3\x1ey\xca-0\xe3z^\xd6\xd6\x8f_\xf1\x9dP\x9fi\xa7\xd1" + b"\xe8\x90\x84\xdc\xbf\xcdky\x8e\xdc\x81\x7f\xa3\xb2+\xbf\x04\xef\xd8\\" + b"\xc4\xdf\xe1\xb0\x01\xe9\x93\xe3Y\xf1\x1dY\xe8h\x81\xcf\xf1w\xcc\xb4\xef" + b" \x8b|\x04\xea\x83ej\xbe\x1f\xd4z\x9c`\xd3\x1a\x92A\x06\xe5\x8f\xa9\x13" + b"\t\x9e=\xfa\x1c\xe5_\x9f%v\x1bo\x11ZO\xd8\xf4\t\xddM\x16-\x04\xfc\x18<\"" + b"CM\xddg~b\xf6\xef\x8e\x0c\xd0\xde|\xa0'\x8a\x0c\xd6x\xae!J\xa6F\x88\x15u" + b"\x008\x17\xbc7y\xb3\xd8u\xac_\x85\x8d\xe7\xc1@\x9c\xecqc\xa3#\xad\xf1" + b"\x935\xb5)_\r\xec3]\x0fo]5\xd0my\x07\x9b\xee\x81\xb5\x0f\xcfK+\x00\xc0" + b"\xe4b\x10\xe4\x0c\x1a \x9b\xe0\x97t\xf6\xa1\x9e\x850\xba\x0c\x9a\x8d\xc8" + b"\x8f\x07\xd7\xae\xc8\xf9+i\xdc\xb9k\xb0>f\x19\xb8\r\xa8\xf8\x1f$\xa5{p" + b"\xc6\x880\xce\xdb\xcf\xca_\x86\xac\x88h6\x8bZ%'\xd0\n\xbf\x0f\x9c\"\xba" + b"\xe5\x86\x9f\x0f7X=mNX[\xcc\x19FU\xc9\x860\xbc\x90a+* \xae_$\x03\x1e\xd3" + b"\xcd_\xa0\x9c\xde\xaf46q\xa5\xc9\x92\xd7\xca\xe3`\x9d\x85}\xb4\xff\xb3" + b"\x83\xfb\xb6\xca\xae`\x0bw\x7f\xfc\xd8\xacVe\x19\xc8\x17\x0bZ\xad\x88" + b"\xeb#\x97\x03\x13\xb1d\x0f{\x0c\x04w\x07\r\x97\xbd\xd6\xc1\xc3B:\x95\x08" + b"^\x10V\xaeaH\x02\xd9\xe3\n\\\x01X\xf6\x9c\x8a\x06u#%\xbe*\xa1\x18v\x85" + b"\xec!\t4\x00\x00\x00\x00Vj?uLU\xf3\xa6\x00\x01\xfb\x07\x81\x0f\x00\x00tw" + b"\x99P\xb1\xc4g\xfb\x02\x00\x00\x00\x00\x04YZ" +) + +COMPRESSED_ALONE = ( + b"]\x00\x00\x80\x00\xff\xff\xff\xff\xff\xff\xff\xff\x00\x05\x14\x07bX\x19" + b"\xcd\xddn\x98\x15\xe4\xb4\x9do\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8" + b"\xe2\xfc\xe7\xd9\xfe6\xb8(\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02" + b"\x17/\xa6=\xf0\xa2\xdf/M\x89\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ" + b"\x15\x80\x8c\xf8\x8do\xfa\x12\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t" + b"\xca6 BF$\xe5Q\xa4\x98\xee\xdel\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81" + b"\xe4N\xc8\x86\x153\xf5x2\xa2O\x13@Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z" + b"\xc4\xcdS\xb6t<\x16\xf2\x9cI#\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0" + b"\xaa\x96-Pe\xade:\x04\t\x1b\xf7\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9" + b"\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7" + b"\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8" + b"\x84b\xf8\x1epl\xeajr\xd1=\t\x03\xdd\x13\x1b3!E\xf9vV\xdaF\xf3\xd7\xb4" + b"\x0c\xa9P~\xec\xdeE\xe37\xf6\x1d\xc6\xbb\xddc%\xb6\x0fI\x07\xf0;\xaf\xe7" + b"\xa0\x8b\xa7Z\x99(\xe9\xe2\xf0o\x18>`\xe1\xaa\xa8\xd9\xa1\xb2}\xe7\x8d" + b"\x834T\xb6\xef\xc1\xde\xe3\x98\xbcD\x03MA@\xd8\xed\xdc\xc8\x93\x03\x1a" + b"\x93\x0b\x7f\x94\x12\x0b\x02Sa\x18\xc9\xc5\x9bTJE}\xf6\xc8g\x17#ZV\x01" + b"\xc9\x9dc\x83\x0e>0\x16\x90S\xb8/\x03y_\x18\xfa(\xd7\x0br\xa2\xb0\xba?" + b"\x8c\xe6\x83@\x84\xdf\x02:\xc5z\x9e\xa6\x84\xc9\xf5BeyX\x83\x1a\xf1 :\t" + b"\xf7\x19\xfexD\\&G\xf3\x85Y\xa2J\xf9\x0bv{\x89\xf6\xe7)A\xaf\x04o\x00" + b"\x075\xd3\xe0\x7f\x97\x98F\x0f?v\x93\xedVtTf\xb5\x97\x83\xed\x19\xd7\x1a" + b"'k\xd7\xd9\xc5\\Y\xd1\xdc\x07\x15|w\xbc\xacd\x87\x08d\xec\xa7\xf6\x82" + b"\xfc\xb3\x93\xeb\xb9 \x8d\xbc ,\xb3X\xb0\xd2s\xd7\xd1\xffv\x05\xdf}\xa2" + b"\x96\xfb%\n\xdf\xa2\x7f\x08.\xa16\n\xe0\x19\x93\x7fh\n\x1c\x8c\x0f \x11" + b"\xc6Bl\x95\x19U}\xe4s\xb5\x10H\xea\x86pB\xe88\x95\xbe\x8cZ\xdb\xe4\x94A" + b"\x92\xb9;z\xaa\xa7{\x1c5!\xc0\xaf\xc1A\xf9\xda\xf0$\xb0\x02qg\xc8\xc7/|" + b"\xafr\x99^\x91\x88\xbf\x03\xd9=\xd7n\xda6{>8\n\xc7:\xa9'\xba.\x0b\xe2" + b"\xb5\x1d\x0e\n\x9a\x8e\x06\x8f:\xdd\x82'[\xc3\"wD$\xa7w\xecq\x8c,1\x93" + b"\xd0,\xae2w\x93\x12$Jd\x19mg\x02\x93\x9cA\x95\x9d&\xca8i\x9c\xb0;\xe7NQ" + b"\x1frh\x8beL;\xb0m\xee\x07Q\x9b\xc6\xd8\x03\xb5\xdeN\xd4\xfe\x98\xd0\xdc" + b"\x1a[\x04\xde\x1a\xf6\x91j\xf8EOli\x8eB^\x1d\x82\x07\xb2\xb5R]\xb7\xd7" + b"\xe9\xa6\xc3.\xfb\xf0-\xb4e\x9b\xde\x03\x88\xc6\xc1iN\x0e\x84wbQ\xdf~" + b"\xe9\xa4\x884\x96kM\xbc)T\xf3\x89\x97\x0f\x143\xe7)\xa0\xb3B\x00\xa8\xaf" + b"\x82^\xcb\xc7..\xdb\xc7\t\x9dH\xee5\xe9#\xe6NV\x94\xcb$Kk\xe3\x7f\r\xe3t" + b"\x12\xcf'\xefR\x8b\xf42\xcf-LH\xac\xe5\x1f0~?SO\xeb\xc1E\x1a\x1c]\xf2" + b"\xc4<\x11\x02\x10Z0a*?\xe4r\xff\xfb\xff\xf6\x14nG\xead^\xd6\xef8\xb6uEI" + b"\x99\nV\xe2\xb3\x95\x8e\x83\xf6i!\xb5&1F\xb1DP\xf4 SO3D!w\x99_G\x7f+\x90" + b".\xab\xbb]\x91>\xc9#h;\x0f5J\x91K\xf4^-[\x9e\x8a\\\x94\xca\xaf\xf6\x19" + b"\xd4\xa1\x9b\xc4\xb8p\xa1\xae\x15\xe9r\x84\xe0\xcar.l []\x8b\xaf+0\xf2g" + b"\x01aKY\xdfI\xcf,\n\xe8\xf0\xe7V\x80_#\xb2\xf2\xa9\x06\x8c>w\xe2W,\xf4" + b"\x8c\r\xf963\xf5J\xcc2\x05=kT\xeaUti\xe5_\xce\x1b\xfa\x8dl\x02h\xef\xa8" + b"\xfbf\x7f\xff\xf0\x19\xeax" +) + +FILTERS_RAW_1 = [{"id": lzma.FILTER_LZMA2, "preset": 3}] +COMPRESSED_RAW_1 = ( + b"\xe0\x07\x80\x03\xfd]\x00\x05\x14\x07bX\x19\xcd\xddn\x96cyq\xa1\xdd\xee" + b"\xf8\xfam\xe3'\x88\xd3\xff\xe4\x9e \xceQ\x91\xa4\x14I\xf6\xb9\x9dVL8\x15" + b"_\x0e\x12\xc3\xeb\xbc\xa5\xcd\nW\x1d$=R;\x1d\xf8k8\t\xb1{\xd4\xc5+\x9d" + b"\x87c\xe5\xef\x98\xb4\xd7S3\xcd\xcc\xd2\xed\xa4\x0em\xe5\xf4\xdd\xd0b" + b"\xbe4*\xaa\x0b\xc5\x08\x10\x85+\x81.\x17\xaf9\xc9b\xeaZrA\xe20\x7fs\"r" + b"\xdaG\x81\xde\x90cu\xa5\xdb\xa9.A\x08l\xb0<\xf6\x03\xddOi\xd0\xc5\xb4" + b"\xec\xecg4t6\"\xa6\xb8o\xb5?\x18^}\xb6}\x03[:\xeb\x03\xa9\n[\x89l\x19g" + b"\x16\xc82\xed\x0b\xfb\x86n\xa2\x857@\x93\xcd6T\xc3u\xb0\t\xf9\x1b\x918" + b"\xfc[\x1b\x1e4\xb3\x14\x06PCV\xa8\"\xf5\x81x~\xe9\xb5N\x9cK\x9f\xc6\xc3%" + b"\xc8k:{6\xe7\xf7\xbd\x05\x02\xb4\xc4\xc3\xd3\xfd\xc3\xa8\\\xfc@\xb1F_" + b"\xc8\x90\xd9sU\x98\xad8\x05\x07\xde7J\x8bM\xd0\xb3;X\xec\x87\xef\xae\xb3" + b"eO,\xb1z,d\x11y\xeejlB\x02\x1d\xf28\x1f#\x896\xce\x0b\xf0\xf5\xa9PK\x0f" + b"\xb3\x13P\xd8\x88\xd2\xa1\x08\x04C?\xdb\x94_\x9a\"\xe9\xe3e\x1d\xde\x9b" + b"\xa1\xe8>H\x98\x10;\xc5\x03#\xb5\x9d4\x01\xe7\xc5\xba%v\xa49\x97A\xe0\"" + b"\x8c\xc22\xe3i\xc1\x9d\xab3\xdf\xbe\xfdDm7\x1b\x9d\xab\xb5\x15o:J\x92" + b"\xdb\x816\x17\xc2O\x99\x1b\x0e\x8d\xf3\tQ\xed\x8e\x95S/\x16M\xb2S\x04" + b"\x0f\xc3J\xc6\xc7\xe4\xcb\xc5\xf4\xe7d\x14\xe4=^B\xfb\xd3E\xd3\x1e\xcd" + b"\x91\xa5\xd0G\x8f.\xf6\xf9\x0bb&\xd9\x9f\xc2\xfdj\xa2\x9e\xc4\\\x0e\x1dC" + b"v\xe8\xd2\x8a?^H\xec\xae\xeb>\xfe\xb8\xab\xd4IqY\x8c\xd4K7\x11\xf4D\xd0W" + b"\xa5\xbe\xeaO\xbf\xd0\x04\xfdl\x10\xae5\xd4U\x19\x06\xf9{\xaa\xe0\x81" + b"\x0f\xcf\xa3k{\x95\xbd\x19\xa2\xf8\xe4\xa3\x08O*\xf1\xf1B-\xc7(\x0eR\xfd" + b"@E\x9f\xd3\x1e:\xfdV\xb7\x04Y\x94\xeb]\x83\xc4\xa5\xd7\xc0gX\x98\xcf\x0f" + b"\xcd3\x00]n\x17\xec\xbd\xa3Y\x86\xc5\xf3u\xf6*\xbdT\xedA$A\xd9A\xe7\x98" + b"\xef\x14\x02\x9a\xfdiw\xec\xa0\x87\x11\xd9%\xc5\xeb\x8a=\xae\xc0\xc4\xc6" + b"D\x80\x8f\xa8\xd1\xbbq\xb2\xc0\xa0\xf5Cqp\xeeL\xe3\xe5\xdc \x84\"\xe9" + b"\x80t\x83\x05\xba\xf1\xc5~\x93\xc9\xf0\x01c\xceix\x9d\xed\xc5)l\x16)\xd1" + b"\x03@l\x04\x7f\x87\xa5yn\x1b\x01D\xaa:\xd2\x96\xb4\xb3?\xb0\xf9\xce\x07" + b"\xeb\x81\x00\xe4\xc3\xf5%_\xae\xd4\xf9\xeb\xe2\rh\xb2#\xd67Q\x16D\x82hn" + b"\xd1\xa3_?q\xf0\xe2\xac\xf317\x9e\xd0_\x83|\xf1\xca\xb7\x95S\xabW\x12" + b"\xff\xddt\xf69L\x01\xf2|\xdaW\xda\xees\x98L\x18\xb8_\xe8$\x82\xea\xd6" + b"\xd1F\xd4\x0b\xcdk\x01vf\x88h\xc3\xae\xb91\xc7Q\x9f\xa5G\xd9\xcc\x1f\xe3" + b"5\xb1\xdcy\x7fI\x8bcw\x8e\x10rIp\x02:\x19p_\xc8v\xcea\"\xc1\xd9\x91\x03" + b"\xbfe\xbe\xa6\xb3\xa8\x14\x18\xc3\xabH*m}\xc2\xc1\x9a}>l%\xce\x84\x99" + b"\xb3d\xaf\xd3\x82\x15\xdf\xc1\xfc5fOg\x9b\xfc\x8e^&\t@\xce\x9f\x06J\xb8" + b"\xb5\x86\x1d\xda{\x9f\xae\xb0\xff\x02\x81r\x92z\x8cM\xb7ho\xc9^\x9c\xb6" + b"\x9c\xae\xd1\xc9\xf4\xdfU7\xd6\\!\xea\x0b\x94k\xb9Ud~\x98\xe7\x86\x8az" + b"\x10;\xe3\x1d\xe5PG\xf8\xa4\x12\x05w\x98^\xc4\xb1\xbb\xfb\xcf\xe0\x7f" + b"\x033Sf\x0c \xb1\xf6@\x94\xe5\xa3\xb2\xa7\x10\x9a\xc0\x14\xc3s\xb5xRD" + b"\xf4`W\xd9\xe5\xd3\xcf\x91\rTZ-X\xbe\xbf\xb5\xe2\xee|\x1a\xbf\xfb\x08" + b"\x91\xe1\xfc\x9a\x18\xa3\x8b\xd6^\x89\xf5[\xef\x87\xd1\x06\x1c7\xd6\xa2" + b"\t\tQ5/@S\xc05\xd2VhAK\x03VC\r\x9b\x93\xd6M\xf1xO\xaaO\xed\xb9<\x0c\xdae" + b"*\xd0\x07Hk6\x9fG+\xa1)\xcd\x9cl\x87\xdb\xe1\xe7\xefK}\x875\xab\xa0\x19u" + b"\xf6*F\xb32\x00\x00\x00" +) + +FILTERS_RAW_2 = [{"id": lzma.FILTER_DELTA, "dist": 2}, + {"id": lzma.FILTER_LZMA2, + "preset": lzma.PRESET_DEFAULT | lzma.PRESET_EXTREME}] +COMPRESSED_RAW_2 = ( + b"\xe0\x07\x80\x05\x91]\x00\x05\x14\x06-\xd4\xa8d?\xef\xbe\xafH\xee\x042" + b"\xcb.\xb5g\x8f\xfb\x14\xab\xa5\x9f\x025z\xa4\xdd\xd8\t[}W\xf8\x0c\x1dmH" + b"\xfa\x05\xfcg\xba\xe5\x01Q\x0b\x83R\xb6A\x885\xc0\xba\xee\n\x1cv~\xde:o" + b"\x06:J\xa7\x11Cc\xea\xf7\xe5*o\xf7\x83\\l\xbdE\x19\x1f\r\xa8\x10\xb42" + b"\x0caU{\xd7\xb8w\xdc\xbe\x1b\xfc8\xb4\xcc\xd38\\\xf6\x13\xf6\xe7\x98\xfa" + b"\xc7[\x17_9\x86%\xa8\xf8\xaa\xb8\x8dfs#\x1e=\xed<\x92\x10\\t\xff\x86\xfb" + b"=\x9e7\x18\x1dft\\\xb5\x01\x95Q\xc5\x19\xb38\xe0\xd4\xaa\x07\xc3\x7f\xd8" + b"\xa2\x00>-\xd3\x8e\xa1#\xfa\x83ArAm\xdbJ~\x93\xa3B\x82\xe0\xc7\xcc(\x08`" + b"WK\xad\x1b\x94kaj\x04 \xde\xfc\xe1\xed\xb0\x82\x91\xefS\x84%\x86\xfbi" + b"\x99X\xf1B\xe7\x90;E\xfde\x98\xda\xca\xd6T\xb4bg\xa4\n\x9aj\xd1\x83\x9e]" + b"\"\x7fM\xb5\x0fr\xd2\\\xa5j~P\x10GH\xbfN*Z\x10.\x81\tpE\x8a\x08\xbe1\xbd" + b"\xcd\xa9\xe1\x8d\x1f\x04\xf9\x0eH\xb9\xae\xd6\xc3\xc1\xa5\xa9\x95P\xdc~" + b"\xff\x01\x930\xa9\x04\xf6\x03\xfe\xb5JK\xc3]\xdd9\xb1\xd3\xd7F\xf5\xd1" + b"\x1e\xa0\x1c_\xed[\x0c\xae\xd4\x8b\x946\xeb\xbf\xbb\xe3$kS{\xb5\x80,f:Sj" + b"\x0f\x08z\x1c\xf5\xe8\xe6\xae\x98\xb0Q~r\x0f\xb0\x05?\xb6\x90\x19\x02&" + b"\xcb\x80\t\xc4\xea\x9c|x\xce\x10\x9c\xc5|\xcbdhh+\x0c'\xc5\x81\xc33\xb5" + b"\x14q\xd6\xc5\xe3`Z#\xdc\x8a\xab\xdd\xea\x08\xc2I\xe7\x02l{\xec\x196\x06" + b"\x91\x8d\xdc\xd5\xb3x\xe1hz%\xd1\xf8\xa5\xdd\x98!\x8c\x1c\xc1\x17RUa\xbb" + b"\x95\x0f\xe4X\xea1\x0c\xf1=R\xbe\xc60\xe3\xa4\x9a\x90bd\x97$]B\x01\xdd" + b"\x1f\xe3h2c\x1e\xa0L`4\xc6x\xa3Z\x8a\r\x14]T^\xd8\x89\x1b\x92\r;\xedY" + b"\x0c\xef\x8d9z\xf3o\xb6)f\xa9]$n\rp\x93\xd0\x10\xa4\x08\xb8\xb2\x8b\xb6" + b"\x8f\x80\xae;\xdcQ\xf1\xfa\x9a\x06\x8e\xa5\x0e\x8cK\x9c @\xaa:UcX\n!\xc6" + b"\x02\x12\xcb\x1b\"=\x16.\x1f\x176\xf2g=\xe1Wn\xe9\xe1\xd4\xf1O\xad\x15" + b"\x86\xe9\xa3T\xaf\xa9\xd7D\xb5\xd1W3pnt\x11\xc7VOj\xb7M\xc4i\xa1\xf1$3" + b"\xbb\xdc\x8af\xb0\xc5Y\r\xd1\xfb\xf2\xe7K\xe6\xc5hwO\xfe\x8c2^&\x07\xd5" + b"\x1fV\x19\xfd\r\x14\xd2i=yZ\xe6o\xaf\xc6\xb6\x92\x9d\xc4\r\xb3\xafw\xac%" + b"\xcfc\x1a\xf1`]\xf2\x1a\x9e\x808\xedm\xedQ\xb2\xfe\xe4h`[q\xae\xe0\x0f" + b"\xba0g\xb6\"N\xc3\xfb\xcfR\x11\xc5\x18)(\xc40\\\xa3\x02\xd9G!\xce\x1b" + b"\xc1\x96x\xb5\xc8z\x1f\x01\xb4\xaf\xde\xc2\xcd\x07\xe7H\xb3y\xa8M\n\\A\t" + b"ar\xddM\x8b\x9a\xea\x84\x9b!\xf1\x8d\xb1\xf1~\x1e\r\xa5H\xba\xf1\x84o" + b"\xda\x87\x01h\xe9\xa2\xbe\xbeqN\x9d\x84\x0b!WG\xda\xa1\xa5A\xb7\xc7`j" + b"\x15\xf2\xe9\xdd?\x015B\xd2~E\x06\x11\xe0\x91!\x05^\x80\xdd\xa8y\x15}" + b"\xa1)\xb1)\x81\x18\xf4\xf4\xf8\xc0\xefD\xe3\xdb2f\x1e\x12\xabu\xc9\x97" + b"\xcd\x1e\xa7\x0c\x02x4_6\x03\xc4$t\xf39\x94\x1d=\xcb\xbfv\\\xf5\xa3\x1d" + b"\x9d8jk\x95\x13)ff\xf9n\xc4\xa9\xe3\x01\xb8\xda\xfb\xab\xdfM\x99\xfb\x05" + b"\xe0\xe9\xb0I\xf4E\xab\xe2\x15\xa3\x035\xe7\xdeT\xee\x82p\xb4\x88\xd3" + b"\x893\x9c/\xc0\xd6\x8fou;\xf6\x95PR\xa9\xb2\xc1\xefFj\xe2\xa7$\xf7h\xf1" + b"\xdfK(\xc9c\xba7\xe8\xe3)\xdd\xb2,\x83\xfb\x84\x18.y\x18Qi\x88\xf8`h-" + b"\xef\xd5\xed\x8c\t\xd8\xc3^\x0f\x00\xb7\xd0[!\xafM\x9b\xd7.\x07\xd8\xfb" + b"\xd9\xe2-S+\xaa8,\xa0\x03\x1b \xea\xa8\x00\xc3\xab~\xd0$e\xa5\x7f\xf7" + b"\x95P]\x12\x19i\xd9\x7fo\x0c\xd8g^\rE\xa5\x80\x18\xc5\x01\x80\xaek`\xff~" + b"\xb6y\xe7+\xe5\x11^D\xa7\x85\x18\"!\xd6\xd2\xa7\xf4\x1eT\xdb\x02\xe15" + b"\x02Y\xbc\x174Z\xe7\x9cH\x1c\xbf\x0f\xc6\xe9f]\xcf\x8cx\xbc\xe5\x15\x94" + b"\xfc3\xbc\xa7TUH\xf1\x84\x1b\xf7\xa9y\xc07\x84\xf8X\xd8\xef\xfc \x1c\xd8" + b"( /\xf2\xb7\xec\xc1\\\x8c\xf6\x95\xa1\x03J\x83vP8\xe1\xe3\xbb~\xc24kA" + b"\x98y\xa1\xf2P\xe9\x9d\xc9J\xf8N\x99\xb4\xceaO\xde\x16\x1e\xc2\x19\xa7" + b"\x03\xd2\xe0\x8f:\x15\xf3\x84\x9e\xee\xe6e\xb8\x02q\xc7AC\x1emw\xfd\t" + b"\x9a\x1eu\xc1\xa9\xcaCwUP\x00\xa5\xf78L4w!\x91L2 \x87\xd0\xf2\x06\x81j" + b"\x80;\x03V\x06\x87\x92\xcb\x90lv@E\x8d\x8d\xa5\xa6\xe7Z[\xdf\xd6E\x03`>" + b"\x8f\xde\xa1bZ\x84\xd0\xa9`\x05\x0e{\x80;\xe3\xbef\x8d\x1d\xebk1.\xe3" + b"\xe9N\x15\xf7\xd4(\xfa\xbb\x15\xbdu\xf7\x7f\x86\xae!\x03L\x1d\xb5\xc1" + b"\xb9\x11\xdb\xd0\x93\xe4\x02\xe1\xd2\xcbBjc_\xe8}d\xdb\xc3\xa0Y\xbe\xc9/" + b"\x95\x01\xa3,\xe6bl@\x01\xdbp\xc2\xce\x14\x168\xc2q\xe3uH\x89X\xa4\xa9" + b"\x19\x1d\xc1}\x7fOX\x19\x9f\xdd\xbe\x85\x83\xff\x96\x1ee\x82O`CF=K\xeb$I" + b"\x17_\xefX\x8bJ'v\xde\x1f+\xd9.v\xf8Tv\x17\xf2\x9f5\x19\xe1\xb9\x91\xa8S" + b"\x86\xbd\x1a\"(\xa5x\x8dC\x03X\x81\x91\xa8\x11\xc4pS\x13\xbc\xf2'J\xae!" + b"\xef\xef\x84G\t\x8d\xc4\x10\x132\x00oS\x9e\xe0\xe4d\x8f\xb8y\xac\xa6\x9f" + b",\xb8f\x87\r\xdf\x9eE\x0f\xe1\xd0\\L\x00\xb2\xe1h\x84\xef}\x98\xa8\x11" + b"\xccW#\\\x83\x7fo\xbbz\x8f\x00" +) + +FILTERS_RAW_3 = [{"id": lzma.FILTER_IA64, "start_offset": 0x100}, + {"id": lzma.FILTER_LZMA2}] +COMPRESSED_RAW_3 = ( + b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d" + b"o\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8\xe2\xfc\xe7\xd9\xfe6\xb8(" + b"\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02\x17/\xa6=\xf0\xa2\xdf/M\x89" + b"\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ\x15\x80\x8c\xf8\x8do\xfa\x12" + b"\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t\xca6 BF$\xe5Q\xa4\x98\xee\xde" + b"l\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81\xe4N\xc8\x86\x153\xf5x2\xa2O" + b"\x13@Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z\xc4\xcdS\xb6t<\x16\xf2\x9cI#" + b"\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0\xaa\x96-Pe\xade:\x04\t\x1b\xf7" + b"\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b" + b"\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa" + b"\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8\x84b\xf6\xc3\xd4c-H\x93oJl\xd0iQ\xe4k" + b"\x84\x0b\xc1\xb7\xbc\xb1\x17\x88\xb1\xca?@\xf6\x07\xea\xe6x\xf1H12P\x0f" + b"\x8a\xc9\xeauw\xe3\xbe\xaai\xa9W\xd0\x80\xcd#cb5\x99\xd8]\xa9d\x0c\xbd" + b"\xa2\xdcWl\xedUG\xbf\x89yF\xf77\x81v\xbd5\x98\xbeh8\x18W\x08\xf0\x1b\x99" + b"5:\x1a?rD\x96\xa1\x04\x0f\xae\xba\x85\xeb\x9d5@\xf5\x83\xd37\x83\x8ac" + b"\x06\xd4\x97i\xcdt\x16S\x82k\xf6K\x01vy\x88\x91\x9b6T\xdae\r\xfd]:k\xbal" + b"\xa9\xbba\xc34\xf9r\xeb}r\xdb\xc7\xdb*\x8f\x03z\xdc8h\xcc\xc9\xd3\xbcl" + b"\xa5-\xcb\xeaK\xa2\xc5\x15\xc0\xe3\xc1\x86Z\xfb\xebL\xe13\xcf\x9c\xe3" + b"\x1d\xc9\xed\xc2\x06\xcc\xce!\x92\xe5\xfe\x9c^\xa59w \x9bP\xa3PK\x08d" + b"\xf9\xe2Z}\xa7\xbf\xed\xeb%$\x0c\x82\xb8/\xb0\x01\xa9&,\xf7qh{Q\x96)\xf2" + b"q\x96\xc3\x80\xb4\x12\xb0\xba\xe6o\xf4!\xb4[\xd4\x8aw\x10\xf7t\x0c\xb3" + b"\xd9\xd5\xc3`^\x81\x11??\\\xa4\x99\x85R\xd4\x8e\x83\xc9\x1eX\xbfa\xf1" + b"\xac\xb0\xea\xea\xd7\xd0\xab\x18\xe2\xf2\xed\xe1\xb7\xc9\x18\xcbS\xe4>" + b"\xc9\x95H\xe8\xcb\t\r%\xeb\xc7$.o\xf1\xf3R\x17\x1db\xbb\xd8U\xa5^\xccS" + b"\x16\x01\x87\xf3/\x93\xd1\xf0v\xc0r\xd7\xcc\xa2Gkz\xca\x80\x0e\xfd\xd0" + b"\x8b\xbb\xd2Ix\xb3\x1ey\xca-0\xe3z^\xd6\xd6\x8f_\xf1\x9dP\x9fi\xa7\xd1" + b"\xe8\x90\x84\xdc\xbf\xcdky\x8e\xdc\x81\x7f\xa3\xb2+\xbf\x04\xef\xd8\\" + b"\xc4\xdf\xe1\xb0\x01\xe9\x93\xe3Y\xf1\x1dY\xe8h\x81\xcf\xf1w\xcc\xb4\xef" + b" \x8b|\x04\xea\x83ej\xbe\x1f\xd4z\x9c`\xd3\x1a\x92A\x06\xe5\x8f\xa9\x13" + b"\t\x9e=\xfa\x1c\xe5_\x9f%v\x1bo\x11ZO\xd8\xf4\t\xddM\x16-\x04\xfc\x18<\"" + b"CM\xddg~b\xf6\xef\x8e\x0c\xd0\xde|\xa0'\x8a\x0c\xd6x\xae!J\xa6F\x88\x15u" + b"\x008\x17\xbc7y\xb3\xd8u\xac_\x85\x8d\xe7\xc1@\x9c\xecqc\xa3#\xad\xf1" + b"\x935\xb5)_\r\xec3]\x0fo]5\xd0my\x07\x9b\xee\x81\xb5\x0f\xcfK+\x00\xc0" + b"\xe4b\x10\xe4\x0c\x1a \x9b\xe0\x97t\xf6\xa1\x9e\x850\xba\x0c\x9a\x8d\xc8" + b"\x8f\x07\xd7\xae\xc8\xf9+i\xdc\xb9k\xb0>f\x19\xb8\r\xa8\xf8\x1f$\xa5{p" + b"\xc6\x880\xce\xdb\xcf\xca_\x86\xac\x88h6\x8bZ%'\xd0\n\xbf\x0f\x9c\"\xba" + b"\xe5\x86\x9f\x0f7X=mNX[\xcc\x19FU\xc9\x860\xbc\x90a+* \xae_$\x03\x1e\xd3" + b"\xcd_\xa0\x9c\xde\xaf46q\xa5\xc9\x92\xd7\xca\xe3`\x9d\x85}\xb4\xff\xb3" + b"\x83\xfb\xb6\xca\xae`\x0bw\x7f\xfc\xd8\xacVe\x19\xc8\x17\x0bZ\xad\x88" + b"\xeb#\x97\x03\x13\xb1d\x0f{\x0c\x04w\x07\r\x97\xbd\xd6\xc1\xc3B:\x95\x08" + b"^\x10V\xaeaH\x02\xd9\xe3\n\\\x01X\xf6\x9c\x8a\x06u#%\xbe*\xa1\x18v\x85" + b"\xec!\t4\x00\x00\x00" +) + +FILTERS_RAW_4 = [{"id": lzma.FILTER_DELTA, "dist": 4}, + {"id": lzma.FILTER_X86, "start_offset": 0x40}, + {"id": lzma.FILTER_LZMA2, "preset": 4, "lc": 2}] +COMPRESSED_RAW_4 = ( + b"\xe0\x07\x80\x06\x0e\\\x00\x05\x14\x07bW\xaah\xdd\x10\xdc'\xd6\x90,\xc6v" + b"Jq \x14l\xb7\x83xB\x0b\x97f=&fx\xba\n>Tn\xbf\x8f\xfb\x1dF\xca\xc3v_\xca?" + b"\xfbV<\x92#\xd4w\xa6\x8a\xeb\xf6\x03\xc0\x01\x94\xd8\x9e\x13\x12\x98\xd1" + b"*\xfa]c\xe8\x1e~\xaf\xb5]Eg\xfb\x9e\x01\"8\xb2\x90\x06=~\xe9\x91W\xcd" + b"\xecD\x12\xc7\xfa\xe1\x91\x06\xc7\x99\xb9\xe3\x901\x87\x19u\x0f\x869\xff" + b"\xc1\xb0hw|\xb0\xdcl\xcck\xb16o7\x85\xee{Y_b\xbf\xbc$\xf3=\x8d\x8bw\xe5Z" + b"\x08@\xc4kmE\xad\xfb\xf6*\xd8\xad\xa1\xfb\xc5{\xdej,)\x1emB\x1f<\xaeca" + b"\x80(\xee\x07 \xdf\xe9\xf8\xeb\x0e-\x97\x86\x90c\xf9\xea'B\xf7`\xd7\xb0" + b"\x92\xbd\xa0\x82]\xbd\x0e\x0eB\x19\xdc\x96\xc6\x19\xd86D\xf0\xd5\x831" + b"\x03\xb7\x1c\xf7&5\x1a\x8f PZ&j\xf8\x98\x1bo\xcc\x86\x9bS\xd3\xa5\xcdu" + b"\xf9$\xcc\x97o\xe5V~\xfb\x97\xb5\x0b\x17\x9c\xfdxW\x10\xfep4\x80\xdaHDY" + b"\xfa)\xfet\xb5\"\xd4\xd3F\x81\xf4\x13\x1f\xec\xdf\xa5\x13\xfc\"\x91x\xb7" + b"\x99\xce\xc8\x92\n\xeb[\x10l*Y\xd8\xb1@\x06\xc8o\x8d7r\xebu\xfd5\x0e\x7f" + b"\xf1$U{\t}\x1fQ\xcfxN\x9d\x9fXX\xe9`\x83\xc1\x06\xf4\x87v-f\x11\xdb/\\" + b"\x06\xff\xd7)B\xf3g\x06\x88#2\x1eB244\x7f4q\t\xc893?mPX\x95\xa6a\xfb)d" + b"\x9b\xfc\x98\x9aj\x04\xae\x9b\x9d\x19w\xba\xf92\xfaA\x11\\\x17\x97C3\xa4" + b"\xbc!\x88\xcdo[\xec:\x030\x91.\x85\xe0@\\4\x16\x12\x9d\xcaJv\x97\xb04" + b"\xack\xcbkf\xa3ss\xfc\x16^\x8ce\x85a\xa5=&\xecr\xb3p\xd1E\xd5\x80y\xc7" + b"\xda\xf6\xfek\xbcT\xbfH\xee\x15o\xc5\x8c\x830\xec\x1d\x01\xae\x0c-e\\" + b"\x91\x90\x94\xb2\xf8\x88\x91\xe8\x0b\xae\xa7>\x98\xf6\x9ck\xd2\xc6\x08" + b"\xe6\xab\t\x98\xf2!\xa0\x8c^\xacqA\x99<\x1cEG\x97\xc8\xf1\xb6\xb9\x82" + b"\x8d\xf7\x08s\x98a\xff\xe3\xcc\x92\x0e\xd2\xb6U\xd7\xd9\x86\x7fa\xe5\x1c" + b"\x8dTG@\t\x1e\x0e7*\xfc\xde\xbc]6N\xf7\xf1\x84\x9e\x9f\xcf\xe9\x1e\xb5'" + b"\xf4<\xdf\x99sq\xd0\x9d\xbd\x99\x0b\xb4%p4\xbf{\xbb\x8a\xd2\x0b\xbc=M" + b"\x94H:\xf5\xa8\xd6\xa4\xc90\xc2D\xb9\xd3\xa8\xb0S\x87 `\xa2\xeb\xf3W\xce" + b" 7\xf9N#\r\xe6\xbe\t\x9d\xe7\x811\xf9\x10\xc1\xc2\x14\xf6\xfc\xcba\xb7" + b"\xb1\x7f\x95l\xe4\tjA\xec:\x10\xe5\xfe\xc2\\=D\xe2\x0c\x0b3]\xf7\xc1\xf7" + b"\xbceZ\xb1A\xea\x16\xe5\xfddgFQ\xed\xaf\x04\xa3\xd3\xf8\xa2q\x19B\xd4r" + b"\xc5\x0c\x9a\x14\x94\xea\x91\xc4o\xe4\xbb\xb4\x99\xf4@\xd1\xe6\x0c\xe3" + b"\xc6d\xa0Q\n\xf2/\xd8\xb8S5\x8a\x18:\xb5g\xac\x95D\xce\x17\x07\xd4z\xda" + b"\x90\xe65\x07\x19H!\t\xfdu\x16\x8e\x0eR\x19\xf4\x8cl\x0c\xf9Q\xf1\x80" + b"\xe3\xbf\xd7O\xf8\x8c\x18\x0b\x9c\xf1\x1fb\xe1\tR\xb2\xf1\xe1A\xea \xcf-" + b"IGE\xf1\x14\x98$\x83\x15\xc9\xd8j\xbf\x19\x0f\xd5\xd1\xaa\xb3\xf3\xa5I2s" + b"\x8d\x145\xca\xd5\xd93\x9c\xb8D0\xe6\xaa%\xd0\xc0P}JO^h\x8e\x08\xadlV." + b"\x18\x88\x13\x05o\xb0\x07\xeaw\xe0\xb6\xa4\xd5*\xe4r\xef\x07G+\xc1\xbei[" + b"w\xe8\xab(a)_\xef\x15y\xe5\x12\xc9W\x1b.\xad\x85-\xc2\xf7\xe3mU6g\x8eSA" + b"\x01(\xd3\xdb\x16\x13=\xde\x92\xf9,D\xb8\x8a\xb2\xb4\xc9\xc3\xefnE\xe8\\" + b"\xa6\xe2Y\xd2\xcf\xcb\x8c\xb6\xd5\xe9\x1d\x1e\x9a\x8b~\xe2\xa6\rE\x84uV" + b"\xed\xc6\x99\xddm<\x10[\x0fu\x1f\xc1\x1d1\n\xcfw\xb2%!\xf0[\xce\x87\x83B" + b"\x08\xaa,\x08%d\xcef\x94\"\xd9g.\xc83\xcbXY+4\xec\x85qA\n\x1d=9\xf0*\xb1" + b"\x1f/\xf3s\xd61b\x7f@\xfb\x9d\xe3FQ\\\xbd\x82\x1e\x00\xf3\xce\xd3\xe1" + b"\xca,E\xfd7[\xab\xb6\xb7\xac!mA}\xbd\x9d3R5\x9cF\xabH\xeb\x92)cc\x13\xd0" + b"\xbd\xee\xe9n{\x1dIJB\xa5\xeb\x11\xe8`w&`\x8b}@Oxe\t\x8a\x07\x02\x95\xf2" + b"\xed\xda|\xb1e\xbe\xaa\xbbg\x19@\xe1Y\x878\x84\x0f\x8c\xe3\xc98\xf2\x9e" + b"\xd5N\xb5J\xef\xab!\xe2\x8dq\xe1\xe5q\xc5\xee\x11W\xb7\xe4k*\x027\xa0" + b"\xa3J\xf4\xd8m\xd0q\x94\xcb\x07\n:\xb6`.\xe4\x9c\x15+\xc0)\xde\x80X\xd4" + b"\xcfQm\x01\xc2cP\x1cA\x85'\xc9\xac\x8b\xe6\xb2)\xe6\x84t\x1c\x92\xe4Z" + b"\x1cR\xb0\x9e\x96\xd1\xfb\x1c\xa6\x8b\xcb`\x10\x12]\xf2gR\x9bFT\xe0\xc8H" + b"S\xfb\xac<\x04\xc7\xc1\xe8\xedP\xf4\x16\xdb\xc0\xd7e\xc2\x17J^\x1f\xab" + b"\xff[\x08\x19\xb4\xf5\xfb\x19\xb4\x04\xe5c~']\xcb\xc2A\xec\x90\xd0\xed" + b"\x06,\xc5K{\x86\x03\xb1\xcdMx\xdeQ\x8c3\xf9\x8a\xea=\x89\xaba\xd2\xc89a" + b"\xd72\xf0\xc3\x19\x8a\xdfs\xd4\xfd\xbb\x81b\xeaE\"\xd8\xf4d\x0cD\xf7IJ!" + b"\xe5d\xbbG\xe9\xcam\xaa\x0f_r\x95\x91NBq\xcaP\xce\xa7\xa9\xb5\x10\x94eP!" + b"|\x856\xcd\xbfIir\xb8e\x9bjP\x97q\xabwS7\x1a\x0ehM\xe7\xca\x86?\xdeP}y~" + b"\x0f\x95I\xfc\x13\xe1<Q\x1b\x868\x1d\x11\xdf\x94\xf4\x82>r\xa9k\x88\xcb" + b"\xfd\xc3v\xe2\xb9\x8a\x02\x8eq\x92I\xf8\xf6\xf1\x03s\x9b\xb8\xe3\"\xe3" + b"\xa9\xa5>D\xb8\x96;\xe7\x92\xd133\xe8\xdd'e\xc9.\xdc;\x17\x1f\xf5H\x13q" + b"\xa4W\x0c\xdb~\x98\x01\xeb\xdf\xe32\x13\x0f\xddx\n6\xa0\t\x10\xb6\xbb" + b"\xb0\xc3\x18\xb6;\x9fj[\xd9\xd5\xc9\x06\x8a\x87\xcd\xe5\xee\xfc\x9c-%@" + b"\xee\xe0\xeb\xd2\xe3\xe8\xfb\xc0\x122\\\xc7\xaf\xc2\xa1Oth\xb3\x8f\x82" + b"\xb3\x18\xa8\x07\xd5\xee_\xbe\xe0\x1cA\x1e_\r\x9a\xb0\x17W&\xa2D\x91\x94" + b"\x1a\xb2\xef\xf2\xdc\x85;X\xb0,\xeb>-7S\xe5\xca\x07)\x1fp\x7f\xcaQBL\xca" + b"\xf3\xb9d\xfc\xb5su\xb0\xc8\x95\x90\xeb*)\xa0v\xe4\x9a{FW\xf4l\xde\xcdj" + b"\x00" +) + + +def test_main(): + run_unittest( + CompressorDecompressorTestCase, + CompressDecompressFunctionTestCase, + FileTestCase, + MiscellaneousTestCase, + ) + +if __name__ == "__main__": + test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -399,6 +399,8 @@ Library ------- +- Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. + - Issue #13487: Make inspect.getmodule robust against changes done to sys.modules while it is iterating over it. diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c new file mode 100644 --- /dev/null +++ b/Modules/_lzmamodule.c @@ -0,0 +1,1106 @@ +/* _lzma - Low-level Python interface to liblzma. */ + +#define PY_SSIZE_T_CLEAN + +#include "Python.h" +#include "structmember.h" +#ifdef WITH_THREAD +#include "pythread.h" +#endif + +#include <stdarg.h> +#include <string.h> + +#include <lzma.h> + + +#ifndef PY_LONG_LONG +#error "This module requires PY_LONG_LONG to be defined" +#endif + + +#ifdef WITH_THREAD +#define ACQUIRE_LOCK(obj) do { \ + if (!PyThread_acquire_lock((obj)->lock, 0)) { \ + Py_BEGIN_ALLOW_THREADS \ + PyThread_acquire_lock((obj)->lock, 1); \ + Py_END_ALLOW_THREADS \ + } } while (0) +#define RELEASE_LOCK(obj) PyThread_release_lock((obj)->lock) +#else +#define ACQUIRE_LOCK(obj) +#define RELEASE_LOCK(obj) +#endif + + +/* Container formats: */ +enum { + FORMAT_AUTO, + FORMAT_XZ, + FORMAT_ALONE, + FORMAT_RAW, +}; + +#define LZMA_CHECK_UNKNOWN (LZMA_CHECK_ID_MAX + 1) + + +typedef struct { + PyObject_HEAD + lzma_stream lzs; + int flushed; +#ifdef WITH_THREAD + PyThread_type_lock lock; +#endif +} Compressor; + +typedef struct { + PyObject_HEAD + lzma_stream lzs; + int check; + char eof; + PyObject *unused_data; +#ifdef WITH_THREAD + PyThread_type_lock lock; +#endif +} Decompressor; + +/* LZMAError class object. */ +static PyObject *Error; + +/* An empty tuple, used by the filter specifier parsing code. */ +static PyObject *empty_tuple; + + +/* Helper functions. */ + +static int +catch_lzma_error(lzma_ret lzret) +{ + switch (lzret) { + case LZMA_OK: + case LZMA_GET_CHECK: + case LZMA_NO_CHECK: + case LZMA_STREAM_END: + return 0; + case LZMA_UNSUPPORTED_CHECK: + PyErr_SetString(Error, "Unsupported integrity check"); + return 1; + case LZMA_MEM_ERROR: + PyErr_NoMemory(); + return 1; + case LZMA_MEMLIMIT_ERROR: + PyErr_SetString(Error, "Memory usage limit exceeded"); + return 1; + case LZMA_FORMAT_ERROR: + PyErr_SetString(Error, "Input format not supported by decoder"); + return 1; + case LZMA_OPTIONS_ERROR: + PyErr_SetString(Error, "Invalid or unsupported options"); + return 1; + case LZMA_DATA_ERROR: + PyErr_SetString(Error, "Corrupt input data"); + return 1; + case LZMA_BUF_ERROR: + PyErr_SetString(Error, "Insufficient buffer space"); + return 1; + case LZMA_PROG_ERROR: + PyErr_SetString(Error, "Internal error"); + return 1; + default: + PyErr_Format(Error, "Unrecognized error from liblzma: %d", lzret); + return 1; + } +} + +#if BUFSIZ < 8192 +#define INITIAL_BUFFER_SIZE 8192 +#else +#define INITIAL_BUFFER_SIZE BUFSIZ +#endif + +static int +grow_buffer(PyObject **buf) +{ + size_t size = PyBytes_GET_SIZE(*buf); + return _PyBytes_Resize(buf, size + (size >> 3) + 6); +} + + +/* Some custom type conversions for PyArg_ParseTupleAndKeywords(), + since the predefined conversion specifiers do not suit our needs: + + uint32_t - the "I" (unsigned int) specifier is the right size, but + silently ignores overflows on conversion. + + lzma_mode and lzma_match_finder - these are enumeration types, and + so the size of each is implementation-defined. Worse, different + enum types can be of different sizes within the same program, so + to be strictly correct, we need to define two separate converters. + */ + +#define INT_TYPE_CONVERTER_FUNC(TYPE, FUNCNAME) \ + static int \ + FUNCNAME(PyObject *obj, void *ptr) \ + { \ + unsigned long val; \ + \ + val = PyLong_AsUnsignedLong(obj); \ + if (PyErr_Occurred()) \ + return 0; \ + if ((unsigned long)(TYPE)val != val) { \ + PyErr_SetString(PyExc_OverflowError, \ + "Value too large for " #TYPE " type"); \ + return 0; \ + } \ + *(TYPE *)ptr = val; \ + return 1; \ + } + +INT_TYPE_CONVERTER_FUNC(uint32_t, uint32_converter) +INT_TYPE_CONVERTER_FUNC(lzma_mode, lzma_mode_converter) +INT_TYPE_CONVERTER_FUNC(lzma_match_finder, lzma_mf_converter) + +#undef INT_TYPE_CONVERTER_FUNC + + +/* Filter specifier parsing functions. */ + +static void * +parse_filter_spec_lzma(PyObject *spec) +{ + static char *optnames[] = {"id", "preset", "dict_size", "lc", "lp", + "pb", "mode", "nice_len", "mf", "depth", NULL}; + PyObject *id; + PyObject *preset_obj; + uint32_t preset = LZMA_PRESET_DEFAULT; + lzma_options_lzma *options; + + /* First, fill in default values for all the options using a preset. + Then, override the defaults with any values given by the caller. */ + + preset_obj = PyMapping_GetItemString(spec, "preset"); + if (preset_obj == NULL) { + if (PyErr_ExceptionMatches(PyExc_KeyError)) + PyErr_Clear(); + else + return NULL; + } else { + int ok = uint32_converter(preset_obj, &preset); + Py_DECREF(preset_obj); + if (!ok) + return NULL; + } + + options = (lzma_options_lzma *)PyMem_Malloc(sizeof *options); + if (options == NULL) + return PyErr_NoMemory(); + memset(options, 0, sizeof *options); + + if (lzma_lzma_preset(options, preset)) { + PyMem_Free(options); + PyErr_Format(Error, "lzma_lzma_preset() failed for preset %#x", preset); + return NULL; + } + + if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec, + "|OOO&O&O&O&O&O&O&O&", optnames, + &id, &preset_obj, + uint32_converter, &options->dict_size, + uint32_converter, &options->lc, + uint32_converter, &options->lp, + uint32_converter, &options->pb, + lzma_mode_converter, &options->mode, + uint32_converter, &options->nice_len, + lzma_mf_converter, &options->mf, + uint32_converter, &options->depth)) { + PyErr_SetString(PyExc_ValueError, + "Invalid filter specifier for LZMA filter"); + PyMem_Free(options); + options = NULL; + } + return options; +} + +static void * +parse_filter_spec_delta(PyObject *spec) +{ + static char *optnames[] = {"id", "dist", NULL}; + PyObject *id; + uint32_t dist = 1; + lzma_options_delta *options; + + if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec, "|OO&", optnames, + &id, uint32_converter, &dist)) { + PyErr_SetString(PyExc_ValueError, + "Invalid filter specifier for delta filter"); + return NULL; + } + + options = (lzma_options_delta *)PyMem_Malloc(sizeof *options); + if (options == NULL) + return PyErr_NoMemory(); + memset(options, 0, sizeof *options); + options->type = LZMA_DELTA_TYPE_BYTE; + options->dist = dist; + return options; +} + +static void * +parse_filter_spec_bcj(PyObject *spec) +{ + static char *optnames[] = {"id", "start_offset", NULL}; + PyObject *id; + uint32_t start_offset = 0; + lzma_options_bcj *options; + + if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec, "|OO&", optnames, + &id, uint32_converter, &start_offset)) { + PyErr_SetString(PyExc_ValueError, + "Invalid filter specifier for BCJ filter"); + return NULL; + } + + options = (lzma_options_bcj *)PyMem_Malloc(sizeof *options); + if (options == NULL) + return PyErr_NoMemory(); + memset(options, 0, sizeof *options); + options->start_offset = start_offset; + return options; +} + +static void * +parse_filter_spec(lzma_filter *f, PyObject *spec) +{ + PyObject *id_obj; + + if (!PyMapping_Check(spec)) { + PyErr_SetString(PyExc_TypeError, + "Filter specifier must be a dict or dict-like object"); + return NULL; + } + id_obj = PyMapping_GetItemString(spec, "id"); + if (id_obj == NULL) { + if (PyErr_ExceptionMatches(PyExc_KeyError)) + PyErr_SetString(PyExc_ValueError, + "Filter specifier must have an \"id\" entry"); + return NULL; + } + f->id = PyLong_AsUnsignedLongLong(id_obj); + Py_DECREF(id_obj); + if (PyErr_Occurred()) + return NULL; + + switch (f->id) { + case LZMA_FILTER_LZMA1: + case LZMA_FILTER_LZMA2: + f->options = parse_filter_spec_lzma(spec); + return f->options; + case LZMA_FILTER_DELTA: + f->options = parse_filter_spec_delta(spec); + return f->options; + case LZMA_FILTER_X86: + case LZMA_FILTER_POWERPC: + case LZMA_FILTER_IA64: + case LZMA_FILTER_ARM: + case LZMA_FILTER_ARMTHUMB: + case LZMA_FILTER_SPARC: + f->options = parse_filter_spec_bcj(spec); + return f->options; + default: + PyErr_Format(PyExc_ValueError, "Invalid filter ID: %llu", f->id); + return NULL; + } +} + +static void +free_filter_chain(lzma_filter filters[]) +{ + int i; + + for (i = 0; filters[i].id != LZMA_VLI_UNKNOWN; i++) + PyMem_Free(filters[i].options); +} + +static int +parse_filter_chain_spec(lzma_filter filters[], PyObject *filterspecs) +{ + Py_ssize_t i, num_filters; + + num_filters = PySequence_Length(filterspecs); + if (num_filters == -1) + return -1; + if (num_filters > LZMA_FILTERS_MAX) { + PyErr_Format(PyExc_ValueError, + "Too many filters - liblzma supports a maximum of %d", + LZMA_FILTERS_MAX); + return -1; + } + + for (i = 0; i < num_filters; i++) { + int ok = 1; + PyObject *spec = PySequence_GetItem(filterspecs, i); + if (spec == NULL || parse_filter_spec(&filters[i], spec) == NULL) + ok = 0; + Py_XDECREF(spec); + if (!ok) { + filters[i].id = LZMA_VLI_UNKNOWN; + free_filter_chain(filters); + return -1; + } + } + filters[num_filters].id = LZMA_VLI_UNKNOWN; + return 0; +} + + +/* LZMACompressor class. */ + +static PyObject * +compress(Compressor *c, uint8_t *data, size_t len, lzma_action action) +{ + size_t data_size = 0; + PyObject *result; + + result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); + if (result == NULL) + return NULL; + c->lzs.next_in = data; + c->lzs.avail_in = len; + c->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result); + c->lzs.avail_out = PyBytes_GET_SIZE(result); + for (;;) { + lzma_ret lzret; + + Py_BEGIN_ALLOW_THREADS + lzret = lzma_code(&c->lzs, action); + data_size = (char *)c->lzs.next_out - PyBytes_AS_STRING(result); + Py_END_ALLOW_THREADS + if (catch_lzma_error(lzret)) + goto error; + if ((action == LZMA_RUN && c->lzs.avail_in == 0) || + (action == LZMA_FINISH && lzret == LZMA_STREAM_END)) { + break; + } else if (c->lzs.avail_out == 0) { + if (grow_buffer(&result) == -1) + goto error; + c->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size; + c->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size; + } + } + if (data_size != PyBytes_GET_SIZE(result)) + if (_PyBytes_Resize(&result, data_size) == -1) + goto error; + return result; + +error: + Py_XDECREF(result); + return NULL; +} + +PyDoc_STRVAR(Compressor_compress_doc, +"compress(data) -> bytes\n" +"\n" +"Provide data to the compressor object. Returns a chunk of\n" +"compressed data if possible, or b\"\" otherwise.\n" +"\n" +"When you have finished providing data to the compressor, call the\n" +"flush() method to finish the conversion process.\n"); + +static PyObject * +Compressor_compress(Compressor *self, PyObject *args) +{ + Py_buffer buffer; + PyObject *result = NULL; + + if (!PyArg_ParseTuple(args, "y*:compress", &buffer)) + return NULL; + + ACQUIRE_LOCK(self); + if (self->flushed) + PyErr_SetString(PyExc_ValueError, "Compressor has been flushed"); + else + result = compress(self, buffer.buf, buffer.len, LZMA_RUN); + RELEASE_LOCK(self); + PyBuffer_Release(&buffer); + return result; +} + +PyDoc_STRVAR(Compressor_flush_doc, +"flush() -> bytes\n" +"\n" +"Finish the compression process. Returns the compressed data left\n" +"in internal buffers.\n" +"\n" +"The compressor object cannot be used after this method is called.\n"); + +static PyObject * +Compressor_flush(Compressor *self, PyObject *noargs) +{ + PyObject *result = NULL; + + ACQUIRE_LOCK(self); + if (self->flushed) { + PyErr_SetString(PyExc_ValueError, "Repeated call to flush()"); + } else { + self->flushed = 1; + result = compress(self, NULL, 0, LZMA_FINISH); + } + RELEASE_LOCK(self); + return result; +} + +static int +Compressor_init_xz(lzma_stream *lzs, int check, uint32_t preset, + PyObject *filterspecs) +{ + lzma_ret lzret; + + if (filterspecs == Py_None) { + lzret = lzma_easy_encoder(lzs, preset, check); + } else { + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + if (parse_filter_chain_spec(filters, filterspecs) == -1) + return -1; + lzret = lzma_stream_encoder(lzs, filters, check); + free_filter_chain(filters); + } + if (catch_lzma_error(lzret)) + return -1; + else + return 0; +} + +static int +Compressor_init_alone(lzma_stream *lzs, uint32_t preset, PyObject *filterspecs) +{ + lzma_ret lzret; + + if (filterspecs == Py_None) { + lzma_options_lzma options; + + if (lzma_lzma_preset(&options, preset)) { + PyErr_Format(Error, "Invalid compression preset: %#x", preset); + return -1; + } + lzret = lzma_alone_encoder(lzs, &options); + } else { + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + if (parse_filter_chain_spec(filters, filterspecs) == -1) + return -1; + if (filters[0].id == LZMA_FILTER_LZMA1 && + filters[1].id == LZMA_VLI_UNKNOWN) { + lzret = lzma_alone_encoder(lzs, filters[0].options); + } else { + PyErr_SetString(PyExc_ValueError, + "Invalid filter chain for FORMAT_ALONE - " + "must be a single LZMA1 filter"); + lzret = LZMA_PROG_ERROR; + } + free_filter_chain(filters); + } + if (PyErr_Occurred() || catch_lzma_error(lzret)) + return -1; + else + return 0; +} + +static int +Compressor_init_raw(lzma_stream *lzs, PyObject *filterspecs) +{ + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + lzma_ret lzret; + + if (filterspecs == Py_None) { + PyErr_SetString(PyExc_ValueError, + "Must specify filters for FORMAT_RAW"); + return -1; + } + if (parse_filter_chain_spec(filters, filterspecs) == -1) + return -1; + lzret = lzma_raw_encoder(lzs, filters); + free_filter_chain(filters); + if (catch_lzma_error(lzret)) + return -1; + else + return 0; +} + +static int +Compressor_init(Compressor *self, PyObject *args, PyObject *kwargs) +{ + static char *arg_names[] = {"format", "check", "preset", "filters", NULL}; + int format = FORMAT_XZ; + int check = -1; + uint32_t preset = LZMA_PRESET_DEFAULT; + PyObject *preset_obj = Py_None; + PyObject *filterspecs = Py_None; + + if (!PyArg_ParseTupleAndKeywords(args, kwargs, + "|iiOO:LZMACompressor", arg_names, + &format, &check, &preset_obj, + &filterspecs)) + return -1; + + if (format != FORMAT_XZ && check != -1 && check != LZMA_CHECK_NONE) { + PyErr_SetString(PyExc_ValueError, + "Integrity checks are only supported by FORMAT_XZ"); + return -1; + } + + if (preset_obj != Py_None && filterspecs != Py_None) { + PyErr_SetString(PyExc_ValueError, + "Cannot specify both preset and filter chain"); + return -1; + } + + if (preset_obj != Py_None) + if (!uint32_converter(preset_obj, &preset)) + return -1; + +#ifdef WITH_THREAD + self->lock = PyThread_allocate_lock(); + if (self->lock == NULL) { + PyErr_SetString(PyExc_MemoryError, "Unable to allocate lock"); + return -1; + } +#endif + + self->flushed = 0; + switch (format) { + case FORMAT_XZ: + if (check == -1) + check = LZMA_CHECK_CRC64; + if (Compressor_init_xz(&self->lzs, check, preset, filterspecs) != 0) + break; + return 0; + + case FORMAT_ALONE: + if (Compressor_init_alone(&self->lzs, preset, filterspecs) != 0) + break; + return 0; + + case FORMAT_RAW: + if (Compressor_init_raw(&self->lzs, filterspecs) != 0) + break; + return 0; + + default: + PyErr_Format(PyExc_ValueError, + "Invalid container format: %d", format); + break; + } + +#ifdef WITH_THREAD + PyThread_free_lock(self->lock); + self->lock = NULL; +#endif + return -1; +} + +static void +Compressor_dealloc(Compressor *self) +{ + lzma_end(&self->lzs); +#ifdef WITH_THREAD + if (self->lock != NULL) + PyThread_free_lock(self->lock); +#endif + Py_TYPE(self)->tp_free((PyObject *)self); +} + +static PyMethodDef Compressor_methods[] = { + {"compress", (PyCFunction)Compressor_compress, METH_VARARGS, + Compressor_compress_doc}, + {"flush", (PyCFunction)Compressor_flush, METH_NOARGS, + Compressor_flush_doc}, + {NULL} +}; + +PyDoc_STRVAR(Compressor_doc, +"LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)\n" +"\n" +"Create a compressor object for compressing data incrementally.\n" +"\n" +"format specifies the container format to use for the output. This can\n" +"be FORMAT_XZ (default), FORMAT_ALONE, or FORMAT_RAW.\n" +"\n" +"check specifies the integrity check to use. For FORMAT_XZ, the default\n" +"is CHECK_CRC64. FORMAT_ALONE and FORMAT_RAW do not suport integrity\n" +"checks; for these formats, check must be omitted, or be CHECK_NONE.\n" +"\n" +"The settings used by the compressor can be specified either as a\n" +"preset compression level (with the 'preset' argument), or in detail\n" +"as a custom filter chain (with the 'filters' argument). For FORMAT_XZ\n" +"and FORMAT_ALONE, the default is to use the PRESET_DEFAULT preset\n" +"level. For FORMAT_RAW, the caller must always specify a filter chain;\n" +"the raw compressor does not support preset compression levels.\n" +"\n" +"preset (if provided) should be an integer in the range 0-9, optionally\n" +"OR-ed with the constant PRESET_EXTREME.\n" +"\n" +"filters (if provided) should be a sequence of dicts. Each dict should\n" +"have an entry for \"id\" indicating the ID of the filter, plus\n" +"additional entries for options to the filter.\n" +"\n" +"For one-shot compression, use the compress() function instead.\n"); + +static PyTypeObject Compressor_type = { + PyVarObject_HEAD_INIT(NULL, 0) + "_lzma.LZMACompressor", /* tp_name */ + sizeof(Compressor), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor)Compressor_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + Compressor_doc, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + Compressor_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc)Compressor_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + + +/* LZMADecompressor class. */ + +static PyObject * +decompress(Decompressor *d, uint8_t *data, size_t len) +{ + size_t data_size = 0; + PyObject *result; + + result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); + if (result == NULL) + return NULL; + d->lzs.next_in = data; + d->lzs.avail_in = len; + d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result); + d->lzs.avail_out = PyBytes_GET_SIZE(result); + for (;;) { + lzma_ret lzret; + + Py_BEGIN_ALLOW_THREADS + lzret = lzma_code(&d->lzs, LZMA_RUN); + data_size = (char *)d->lzs.next_out - PyBytes_AS_STRING(result); + Py_END_ALLOW_THREADS + if (catch_lzma_error(lzret)) + goto error; + if (lzret == LZMA_GET_CHECK || lzret == LZMA_NO_CHECK) + d->check = lzma_get_check(&d->lzs); + if (lzret == LZMA_STREAM_END) { + d->eof = 1; + if (d->lzs.avail_in > 0) { + Py_CLEAR(d->unused_data); + d->unused_data = PyBytes_FromStringAndSize( + (char *)d->lzs.next_in, d->lzs.avail_in); + if (d->unused_data == NULL) + goto error; + } + break; + } else if (d->lzs.avail_in == 0) { + break; + } else if (d->lzs.avail_out == 0) { + if (grow_buffer(&result) == -1) + goto error; + d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size; + d->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size; + } + } + if (data_size != PyBytes_GET_SIZE(result)) + if (_PyBytes_Resize(&result, data_size) == -1) + goto error; + return result; + +error: + Py_XDECREF(result); + return NULL; +} + +PyDoc_STRVAR(Decompressor_decompress_doc, +"decompress(data) -> bytes\n" +"\n" +"Provide data to the decompressor object. Returns a chunk of\n" +"decompressed data if possible, or b\"\" otherwise.\n" +"\n" +"Attempting to decompress data after the end of the stream is\n" +"reached raises an EOFError. Any data found after the end of the\n" +"stream is ignored, and saved in the unused_data attribute.\n"); + +static PyObject * +Decompressor_decompress(Decompressor *self, PyObject *args) +{ + Py_buffer buffer; + PyObject *result = NULL; + + if (!PyArg_ParseTuple(args, "y*:decompress", &buffer)) + return NULL; + + ACQUIRE_LOCK(self); + if (self->eof) + PyErr_SetString(PyExc_EOFError, "Already at end of stream"); + else + result = decompress(self, buffer.buf, buffer.len); + RELEASE_LOCK(self); + PyBuffer_Release(&buffer); + return result; +} + +static int +Decompressor_init_raw(lzma_stream *lzs, PyObject *filterspecs) +{ + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + lzma_ret lzret; + + if (parse_filter_chain_spec(filters, filterspecs) == -1) + return -1; + lzret = lzma_raw_decoder(lzs, filters); + free_filter_chain(filters); + if (catch_lzma_error(lzret)) + return -1; + else + return 0; +} + +static int +Decompressor_init(Decompressor *self, PyObject *args, PyObject *kwargs) +{ + static char *arg_names[] = {"format", "memlimit", "filters", NULL}; + const uint32_t decoder_flags = LZMA_TELL_ANY_CHECK | LZMA_TELL_NO_CHECK; + int format = FORMAT_AUTO; + uint64_t memlimit = UINT64_MAX; + PyObject *memlimit_obj = Py_None; + PyObject *filterspecs = Py_None; + lzma_ret lzret; + + if (!PyArg_ParseTupleAndKeywords(args, kwargs, + "|iOO:LZMADecompressor", arg_names, + &format, &memlimit_obj, &filterspecs)) + return -1; + + if (memlimit_obj != Py_None) { + if (format == FORMAT_RAW) { + PyErr_SetString(PyExc_ValueError, + "Cannot specify memory limit with FORMAT_RAW"); + return -1; + } + memlimit = PyLong_AsUnsignedLongLong(memlimit_obj); + if (PyErr_Occurred()) + return -1; + } + + if (format == FORMAT_RAW && filterspecs == Py_None) { + PyErr_SetString(PyExc_ValueError, + "Must specify filters for FORMAT_RAW"); + return -1; + } else if (format != FORMAT_RAW && filterspecs != Py_None) { + PyErr_SetString(PyExc_ValueError, + "Cannot specify filters except with FORMAT_RAW"); + return -1; + } + +#ifdef WITH_THREAD + self->lock = PyThread_allocate_lock(); + if (self->lock == NULL) { + PyErr_SetString(PyExc_MemoryError, "Unable to allocate lock"); + return -1; + } +#endif + + self->check = LZMA_CHECK_UNKNOWN; + self->unused_data = PyBytes_FromStringAndSize(NULL, 0); + if (self->unused_data == NULL) + goto error; + + switch (format) { + case FORMAT_AUTO: + lzret = lzma_auto_decoder(&self->lzs, memlimit, decoder_flags); + if (catch_lzma_error(lzret)) + break; + return 0; + + case FORMAT_XZ: + lzret = lzma_stream_decoder(&self->lzs, memlimit, decoder_flags); + if (catch_lzma_error(lzret)) + break; + return 0; + + case FORMAT_ALONE: + self->check = LZMA_CHECK_NONE; + lzret = lzma_alone_decoder(&self->lzs, memlimit); + if (catch_lzma_error(lzret)) + break; + return 0; + + case FORMAT_RAW: + self->check = LZMA_CHECK_NONE; + if (Decompressor_init_raw(&self->lzs, filterspecs) == -1) + break; + return 0; + + default: + PyErr_Format(PyExc_ValueError, + "Invalid container format: %d", format); + break; + } + +error: + Py_CLEAR(self->unused_data); +#ifdef WITH_THREAD + PyThread_free_lock(self->lock); + self->lock = NULL; +#endif + return -1; +} + +static void +Decompressor_dealloc(Decompressor *self) +{ + lzma_end(&self->lzs); + Py_CLEAR(self->unused_data); +#ifdef WITH_THREAD + if (self->lock != NULL) + PyThread_free_lock(self->lock); +#endif + Py_TYPE(self)->tp_free((PyObject *)self); +} + +static PyMethodDef Decompressor_methods[] = { + {"decompress", (PyCFunction)Decompressor_decompress, METH_VARARGS, + Decompressor_decompress_doc}, + {NULL} +}; + +PyDoc_STRVAR(Decompressor_check_doc, +"ID of the integrity check used by the input stream."); + +PyDoc_STRVAR(Decompressor_eof_doc, +"True if the end-of-stream marker has been reached."); + +PyDoc_STRVAR(Decompressor_unused_data_doc, +"Data found after the end of the compressed stream."); + +static PyMemberDef Decompressor_members[] = { + {"check", T_INT, offsetof(Decompressor, check), READONLY, + Decompressor_check_doc}, + {"eof", T_BOOL, offsetof(Decompressor, eof), READONLY, + Decompressor_eof_doc}, + {"unused_data", T_OBJECT_EX, offsetof(Decompressor, unused_data), READONLY, + Decompressor_unused_data_doc}, + {NULL} +}; + +PyDoc_STRVAR(Decompressor_doc, +"LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)\n" +"\n" +"Create a decompressor object for decompressing data incrementally.\n" +"\n" +"format specifies the container format of the input stream. If this is\n" +"FORMAT_AUTO (the default), the decompressor will automatically detect\n" +"whether the input is FORMAT_XZ or FORMAT_ALONE. Streams created with\n" +"FORMAT_RAW cannot be autodetected.\n" +"\n" +"memlimit can be specified to limit the amount of memory used by the\n" +"decompressor. This will cause decompression to fail if the input\n" +"cannot be decompressed within the given limit.\n" +"\n" +"filters specifies a custom filter chain. This argument is required for\n" +"FORMAT_RAW, and not accepted with any other format. When provided,\n" +"this should be a sequence of dicts, each indicating the ID and options\n" +"for a single filter.\n" +"\n" +"For one-shot decompression, use the decompress() function instead.\n"); + +static PyTypeObject Decompressor_type = { + PyVarObject_HEAD_INIT(NULL, 0) + "_lzma.LZMADecompressor", /* tp_name */ + sizeof(Decompressor), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor)Decompressor_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + Decompressor_doc, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + Decompressor_methods, /* tp_methods */ + Decompressor_members, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc)Decompressor_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + + +/* Module-level functions. */ + +PyDoc_STRVAR(check_is_supported_doc, +"check_is_supported(check_id) -> bool\n" +"\n" +"Test whether the given integrity check is supported.\n" +"\n" +"Always returns True for CHECK_NONE and CHECK_CRC32.\n"); + +static PyObject * +check_is_supported(PyObject *self, PyObject *args) +{ + int check_id; + + if (!PyArg_ParseTuple(args, "i:check_is_supported", &check_id)) + return NULL; + + return PyBool_FromLong(lzma_check_is_supported(check_id)); +} + + +/* Module initialization. */ + +static PyMethodDef module_methods[] = { + {"check_is_supported", (PyCFunction)check_is_supported, + METH_VARARGS, check_is_supported_doc}, + {NULL} +}; + +static PyModuleDef _lzmamodule = { + PyModuleDef_HEAD_INIT, + "_lzma", + NULL, + -1, + module_methods, + NULL, + NULL, + NULL, + NULL, +}; + +/* Some of our constants are more than 32 bits wide, so PyModule_AddIntConstant + would not work correctly on platforms with 32-bit longs. */ +static int +module_add_int_constant(PyObject *m, const char *name, PY_LONG_LONG value) +{ + PyObject *o = PyLong_FromLongLong(value); + if (o == NULL) + return -1; + if (PyModule_AddObject(m, name, o) == 0) + return 0; + Py_DECREF(o); + return -1; +} + +#define ADD_INT_PREFIX_MACRO(m, macro) \ + module_add_int_constant(m, #macro, LZMA_ ## macro) + +PyMODINIT_FUNC +PyInit__lzma(void) +{ + PyObject *m; + + empty_tuple = PyTuple_New(0); + if (empty_tuple == NULL) + return NULL; + + m = PyModule_Create(&_lzmamodule); + if (m == NULL) + return NULL; + + if (PyModule_AddIntMacro(m, FORMAT_AUTO) == -1 || + PyModule_AddIntMacro(m, FORMAT_XZ) == -1 || + PyModule_AddIntMacro(m, FORMAT_ALONE) == -1 || + PyModule_AddIntMacro(m, FORMAT_RAW) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_NONE) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_CRC32) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_CRC64) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_SHA256) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_ID_MAX) == -1 || + ADD_INT_PREFIX_MACRO(m, CHECK_UNKNOWN) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_LZMA1) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_LZMA2) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_DELTA) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_X86) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_IA64) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_ARM) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_ARMTHUMB) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_SPARC) == -1 || + ADD_INT_PREFIX_MACRO(m, FILTER_POWERPC) == -1 || + ADD_INT_PREFIX_MACRO(m, MF_HC3) == -1 || + ADD_INT_PREFIX_MACRO(m, MF_HC4) == -1 || + ADD_INT_PREFIX_MACRO(m, MF_BT2) == -1 || + ADD_INT_PREFIX_MACRO(m, MF_BT3) == -1 || + ADD_INT_PREFIX_MACRO(m, MF_BT4) == -1 || + ADD_INT_PREFIX_MACRO(m, MODE_FAST) == -1 || + ADD_INT_PREFIX_MACRO(m, MODE_NORMAL) == -1 || + ADD_INT_PREFIX_MACRO(m, PRESET_DEFAULT) == -1 || + ADD_INT_PREFIX_MACRO(m, PRESET_EXTREME) == -1) + return NULL; + + Error = PyErr_NewExceptionWithDoc( + "_lzma.LZMAError", "Call to liblzma failed.", NULL, NULL); + if (Error == NULL) + return NULL; + Py_INCREF(Error); + if (PyModule_AddObject(m, "LZMAError", Error) == -1) + return NULL; + + if (PyType_Ready(&Compressor_type) == -1) + return NULL; + Py_INCREF(&Compressor_type); + if (PyModule_AddObject(m, "LZMACompressor", + (PyObject *)&Compressor_type) == -1) + return NULL; + + if (PyType_Ready(&Decompressor_type) == -1) + return NULL; + Py_INCREF(&Decompressor_type); + if (PyModule_AddObject(m, "LZMADecompressor", + (PyObject *)&Decompressor_type) == -1) + return NULL; + + return m; +} diff --git a/PCbuild/_lzma.vcproj b/PCbuild/_lzma.vcproj new file mode 100644 --- /dev/null +++ b/PCbuild/_lzma.vcproj @@ -0,0 +1,537 @@ +<?xml version="1.0" encoding="Windows-1252"?> +<VisualStudioProject + ProjectType="Visual C++" + Version="9,00" + Name="_lzma" + ProjectGUID="{F9D71780-F393-11E0-BE50-0800200C9A66}" + RootNamespace="lzma" + Keyword="Win32Proj" + TargetFrameworkVersion="196613" + > + <Platforms> + <Platform + Name="Win32" + /> + <Platform + Name="x64" + /> + </Platforms> + <ToolFiles> + </ToolFiles> + <Configurations> + <Configuration + Name="Debug|Win32" + ConfigurationType="2" + InheritedPropertySheets=".\pyd_d.vsprops" + CharacterSet="0" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="Debug|x64" + ConfigurationType="2" + InheritedPropertySheets=".\pyd_d.vsprops;.\x64.vsprops" + CharacterSet="0" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + TargetEnvironment="3" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="Release|Win32" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="Release|x64" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + TargetEnvironment="3" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="PGInstrument|Win32" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops;.\pginstrument.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="PGInstrument|x64" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pginstrument.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + TargetEnvironment="3" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a" + TargetMachine="17" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="PGUpdate|Win32" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops;.\pgupdate.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="PGUpdate|x64" + ConfigurationType="2" + InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pgupdate.vsprops" + CharacterSet="0" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + TargetEnvironment="3" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="$(lzmaDir)\include" + PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a" + TargetMachine="17" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + </Configurations> + <References> + </References> + <Files> + <Filter + Name="Source Files" + > + <File + RelativePath="..\Modules\_lzmamodule.c" + > + </File> + </Filter> + </Files> + <Globals> + </Globals> +</VisualStudioProject> diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -92,6 +92,11 @@ {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_lzma", "_lzma.vcproj", "{F9D71780-F393-11E0-BE50-0800200C9A66}" + ProjectSection(ProjectDependencies) = postProject + {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} + EndProjectSection +EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "select", "select.vcproj", "{18CAE28C-B454-46C1-87A0-493D91D97F03}" ProjectSection(ProjectDependencies) = postProject {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} @@ -421,6 +426,22 @@ {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|Win32.Build.0 = Release|Win32 {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.ActiveCfg = Release|x64 {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.Build.0 = Release|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.ActiveCfg = Debug|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.Build.0 = Debug|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|x64.ActiveCfg = Debug|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|x64.Build.0 = Debug|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|Win32.ActiveCfg = Release|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|Win32.Build.0 = Release|Win32 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.ActiveCfg = Release|x64 + {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.Build.0 = Release|x64 {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.ActiveCfg = Debug|Win32 {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.Build.0 = Debug|Win32 {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.ActiveCfg = Debug|x64 diff --git a/PCbuild/pyproject.vsprops b/PCbuild/pyproject.vsprops --- a/PCbuild/pyproject.vsprops +++ b/PCbuild/pyproject.vsprops @@ -57,6 +57,10 @@ Value="$(externalsDir)\bzip2-1.0.5" /> <UserMacro + Name="lzmaDir" + Value="$(externalsDir)\xz-5.0.3" + /> + <UserMacro Name="opensslDir" Value="$(externalsDir)\openssl-1.0.0a" /> diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -133,6 +133,12 @@ All of this managed to build libbz2.lib in bzip2-1.0.5\$platform-$configuration\, which the Python project links in. +_lzma + Python wrapper for the liblzma compression library. + + Download the pre-built Windows binaries from http://tukaani.org/xz/, and + extract to ..\xz-5.0.3. If you are using a more recent version of liblzma, + it will be necessary to rename the directory from xz-<VERSION> to xz-5.0.3. _ssl Python wrapper for the secure sockets library. diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -41,3 +41,8 @@ rd /s/q sqlite-source-3.6.21 svn export http://svn.python.org/projects/external/sqlite-3.7.4 ) + +@rem lzma +if not exist xz-5.0.3 ( + svn export http://svn.python.org/projects/external/xz-5.0.3 +) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1279,6 +1279,13 @@ else: missing.append('_bz2') + # LZMA compression support. + if self.compiler.find_library_file(lib_dirs, 'lzma'): + exts.append( Extension('_lzma', ['_lzmamodule.c'], + libraries = ['lzma']) ) + else: + missing.append('_lzma') + # Interface to the Expat XML parser # # Expat was written by James Clark and is now maintained by a group of -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Py_UNICODE_HIGH_SURROGATE() and Py_UNICODE_LOW_SURROGATE() macros
by victor.stinner Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/351cdf87bed0 changeset: 73793:351cdf87bed0 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Tue Nov 29 22:58:13 2011 +0100 summary: Py_UNICODE_HIGH_SURROGATE() and Py_UNICODE_LOW_SURROGATE() macros And use surrogates macros everywhere in unicodeobject.c files: Include/unicodeobject.h | 4 ++ Objects/unicodeobject.c | 45 ++++++++++++---------------- 2 files changed, 23 insertions(+), 26 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -187,6 +187,10 @@ #define Py_UNICODE_JOIN_SURROGATES(high, low) \ (((((Py_UCS4)(high) & 0x03FF) << 10) | \ ((Py_UCS4)(low) & 0x03FF)) + 0x10000) +/* high surrogate = top 10 bits added to D800 */ +#define Py_UNICODE_HIGH_SURROGATE(ch) (0xD800 | (((ch) - 0x10000) >> 10)) +/* low surrogate = bottom 10 bits added to DC00 */ +#define Py_UNICODE_LOW_SURROGATE(ch) (0xDC00 | (((ch) - 0x10000) & 0x3FF)) /* Check if substring matches at given offset. The offset must be valid, and the substring must not be empty. */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1077,10 +1077,11 @@ for (iter = begin; iter < end; ) { assert(ucs4_out < (PyUnicode_4BYTE_DATA(unicode) + _PyUnicode_GET_LENGTH(unicode))); - if (*iter >= 0xD800 && *iter <= 0xDBFF - && (iter+1) < end && iter[1] >= 0xDC00 && iter[1] <= 0xDFFF) + if (Py_UNICODE_IS_HIGH_SURROGATE(iter[0]) + && (iter+1) < end + && Py_UNICODE_IS_LOW_SURROGATE(iter[1])) { - *ucs4_out++ = (((iter[0] & 0x3FF)<<10) | (iter[1] & 0x3FF)) + 0x10000; + *ucs4_out++ = Py_UNICODE_JOIN_SURROGATES(iter[0], iter[1]); iter += 2; } else { @@ -3473,8 +3474,8 @@ if (*four_bytes > 0xFFFF) { assert(*four_bytes <= 0x10FFFF); /* encode surrogate pair in this case */ - *w++ = 0xD800 | ((*four_bytes - 0x10000) >> 10); - *w = 0xDC00 | ((*four_bytes - 0x10000) & 0x3FF); + *w++ = Py_UNICODE_HIGH_SURROGATE(*four_bytes); + *w = Py_UNICODE_LOW_SURROGATE(*four_bytes); } else *w = *four_bytes; @@ -3906,9 +3907,8 @@ base64buffer &= (1 << base64bits) - 1; /* clear high bits */ if (surrogate) { /* expecting a second surrogate */ - if (outCh >= 0xDC00 && outCh <= 0xDFFF) { - Py_UCS4 ch2 = (((surrogate & 0x3FF)<<10) - | (outCh & 0x3FF)) + 0x10000; + if (Py_UNICODE_IS_LOW_SURROGATE(outCh)) { + Py_UCS4 ch2 = Py_UNICODE_JOIN_SURROGATES(surrogate, outCh); if (unicode_putchar(&unicode, &outpos, ch2) < 0) goto onError; surrogate = 0; @@ -3920,7 +3920,7 @@ surrogate = 0; } } - if (outCh >= 0xD800 && outCh <= 0xDBFF) { + if (Py_UNICODE_IS_HIGH_SURROGATE(outCh)) { /* first surrogate */ surrogate = outCh; } @@ -4128,7 +4128,7 @@ base64bits -= 6; } /* prepare second surrogate */ - ch = 0xDC00 | ((ch-0x10000) & 0x3FF); + ch = Py_UNICODE_LOW_SURROGATE(ch); } base64bits += 16; base64buffer = (base64buffer << 16) | ch; @@ -4720,15 +4720,8 @@ *p++ = (wchar_t)ch; #else /* compute and append the two surrogates: */ - - /* translate from 10000..10FFFF to 0..FFFF */ - ch -= 0x10000; - - /* high surrogate = top 10 bits added to D800 */ - *p++ = (wchar_t)(0xD800 + (ch >> 10)); - - /* low surrogate = bottom 10 bits added to DC00 */ - *p++ = (wchar_t)(0xDC00 + (ch & 0x03FF)); + *p++ = (wchar_t)Py_UNICODE_HIGH_SURROGATE(ch); + *p++ = (wchar_t)Py_UNICODE_LOW_SURROGATE(ch); #endif break; } @@ -4819,7 +4812,7 @@ /* Encode Latin-1 */ *p++ = (char)(0xc0 | (ch >> 6)); *p++ = (char)(0x80 | (ch & 0x3f)); - } else if (0xD800 <= ch && ch <= 0xDFFF) { + } else if (Py_UNICODE_IS_SURROGATE(ch)) { Py_ssize_t newpos; Py_ssize_t repsize, k, startpos; startpos = i-1; @@ -5367,7 +5360,7 @@ q += 2; - if (ch < 0xD800 || ch > 0xDFFF) { + if (!Py_UNICODE_IS_SURROGATE(ch)) { if (unicode_putchar(&unicode, &outpos, ch) < 0) goto onError; continue; @@ -5538,8 +5531,8 @@ Py_UCS4 ch = PyUnicode_READ(kind, data, i); Py_UCS4 ch2 = 0; if (ch >= 0x10000) { - ch2 = 0xDC00 | ((ch-0x10000) & 0x3FF); - ch = 0xD800 | ((ch-0x10000) >> 10); + ch2 = Py_UNICODE_LOW_SURROGATE(ch); + ch = Py_UNICODE_HIGH_SURROGATE(ch); } STORECHAR(ch); if (ch2) @@ -6300,14 +6293,14 @@ s += Py_UNICODE_SIZE; #ifndef Py_UNICODE_WIDE - if (ch >= 0xD800 && ch <= 0xDBFF && s < end) + if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && s < end) { Py_UNICODE uch2; ((char *) &uch2)[0] = s[0]; ((char *) &uch2)[1] = s[1]; - if (uch2 >= 0xDC00 && uch2 <= 0xDFFF) + if (Py_UNICODE_IS_LOW_SURROGATE(uch2)) { - ch = (((uch & 0x3FF)<<10) | (uch2 & 0x3FF)) + 0x10000; + ch = Py_UNICODE_JOIN_SURROGATES(uch, uch2); s += Py_UNICODE_SIZE; } } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Mention hashable memoryviews, and reformat.
by antoine.pitrou Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/1828e21832b3 changeset: 73792:1828e21832b3 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Tue Nov 29 22:47:11 2011 +0100 summary: Mention hashable memoryviews, and reformat. files: Doc/whatsnew/3.3.rst | 26 +++++++++++++------------- 1 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -259,26 +259,26 @@ Some smaller changes made to the core Python language are: -* Stub +* Added support for Unicode name aliases and named sequences. + Both :func:`unicodedata.lookup()` and ``'\N{...}'`` now resolve name aliases, + and :func:`unicodedata.lookup()` resolves named sequences too. -Added support for Unicode name aliases and named sequences. -Both :func:`unicodedata.lookup()` and ``'\N{...}'`` now resolve name aliases, -and :func:`unicodedata.lookup()` resolves named sequences too. + (Contributed by Ezio Melotti in :issue:`12753`) -(Contributed by Ezio Melotti in :issue:`12753`) +* Equality comparisons on :func:`range` objects now return a result reflecting + the equality of the underlying sequences generated by those range objects. + (:issue:`13021`) -Equality comparisons on :func:`range` objects now return a result reflecting -the equality of the underlying sequences generated by those range objects. +* The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()`` + methods of :class:`bytes` and :class:`bytearray` objects now accept an + integer between 0 and 255 as their first argument. -(:issue:`13021`) + (:issue:`12170`) +* Memoryview objects are now hashable when the underlying object is hashable. -The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()`` -methods of :class:`bytes` and :class:`bytearray` objects now accept an -integer between 0 and 255 as their first argument. - -(:issue:`12170`) + (Contributed by Antoine Pitrou in :issue:`13411`) New and Improved Modules -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Add subheaders to make PEP 393 description clearer
by antoine.pitrou Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/3e295f2a47b9 changeset: 73791:3e295f2a47b9 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Tue Nov 29 22:45:07 2011 +0100 summary: Add subheaders to make PEP 393 description clearer files: Doc/whatsnew/3.3.rst | 50 +++++++++++++++++-------------- 1 files changed, 28 insertions(+), 22 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -69,6 +69,9 @@ a bit more memory, because Python may have to maintain two versions of each string (in the legacy format and in the new efficient storage). +Functionality +------------- + Changes introduced by :pep:`393` are the following: * Python now always supports the full range of Unicode codepoints, including @@ -76,28 +79,6 @@ narrow and wide builds no longer exists and Python now behaves like a wide build, even under Windows. -* The storage of Unicode strings now depends on the highest codepoint in the string: - - * pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint; - - * BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint; - - * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint. - - The net effect is that for most applications, memory usage of string storage - should decrease significantly - especially compared to former wide unicode - builds - as, in many cases, strings will be pure ASCII even in international - contexts (because many strings store non-human language data, such as XML - fragments, HTTP headers, JSON-encoded data, etc.). We also hope that it - will, for the same reasons, increase CPU cache efficiency on non-trivial - applications. - - .. The memory usage of Python 3.3 is two to three times smaller than Python 3.2, - and a little bit better than Python 2.7, on a `Django benchmark - <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_. - XXX The result should be moved in the PEP and a link to the PEP should - be added here. - * With the death of narrow builds, the problems specific to narrow builds have also been fixed, for example: @@ -120,6 +101,31 @@ * The :file:`./configure` flag ``--with-wide-unicode`` has been removed. +Performance and resource usage +------------------------------ + +The storage of Unicode strings now depends on the highest codepoint in the string: + +* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint; + +* BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint; + +* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint. + +The net effect is that for most applications, memory usage of string storage +should decrease significantly - especially compared to former wide unicode +builds - as, in many cases, strings will be pure ASCII even in international +contexts (because many strings store non-human language data, such as XML +fragments, HTTP headers, JSON-encoded data, etc.). We also hope that it +will, for the same reasons, increase CPU cache efficiency on non-trivial +applications. + +.. The memory usage of Python 3.3 is two to three times smaller than Python 3.2, + and a little bit better than Python 2.7, on a `Django benchmark + <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_. + XXX The result should be moved in the PEP and a link to the PEP should + be added here. + PEP 3151: Reworking the OS and IO exception hierarchy ===================================================== -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge 3.2
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/599763461943 changeset: 73790:599763461943 parent: 73782:5d42f22189be parent: 73789:2ef359d7a2e9 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 17:14:27 2011 +0100 summary: Merge 3.2 files: Doc/library/socket.rst | 16 ++++++++-------- Doc/library/sysconfig.rst | 2 +- Lib/inspect.py | 2 +- Lib/keyword.py | 2 +- Lib/locale.py | 3 +-- Lib/symbol.py | 2 +- Lib/test/regrtest.py | 18 ++++++++++++++++++ Lib/token.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ Misc/python.man | 2 +- 11 files changed, 37 insertions(+), 16 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -64,20 +64,20 @@ tuple, and the fields depend on the address type. The general tuple form is ``(addr_type, v1, v2, v3 [, scope])``, where: - - *addr_type* is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or - TIPC_ADDR_ID. - - *scope* is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and - TIPC_NODE_SCOPE. - - If *addr_type* is TIPC_ADDR_NAME, then *v1* is the server type, *v2* is + - *addr_type* is one of :const:`TIPC_ADDR_NAMESEQ`, :const:`TIPC_ADDR_NAME`, + or :const:`TIPC_ADDR_ID`. + - *scope* is one of :const:`TIPC_ZONE_SCOPE`, :const:`TIPC_CLUSTER_SCOPE`, and + :const:`TIPC_NODE_SCOPE`. + - If *addr_type* is :const:`TIPC_ADDR_NAME`, then *v1* is the server type, *v2* is the port identifier, and *v3* should be 0. - If *addr_type* is TIPC_ADDR_NAMESEQ, then *v1* is the server type, *v2* + If *addr_type* is :const:`TIPC_ADDR_NAMESEQ`, then *v1* is the server type, *v2* is the lower port number, and *v3* is the upper port number. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - A tuple ``(interface, )`` is used for the :const:`AF_CAN` address family, diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -130,7 +130,7 @@ one may call this function and get the default value. If *scheme* is provided, it must be a value from the list returned by - :func:`get_path_names`. Otherwise, the default scheme for the current + :func:`get_scheme_names`. Otherwise, the default scheme for the current platform is used. If *vars* is provided, it must be a dictionary of variables that will update diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -482,7 +482,7 @@ return sys.modules.get(modulesbyfile[file]) # Update the filename to module name cache and check yet again # Copy sys.modules in order to cope with changes while iterating - for modname, module in sys.modules.items(): + for modname, module in list(sys.modules.items()): if ismodule(module) and hasattr(module, '__file__'): f = module.__file__ if f == _filesbymodname.get(modname, None): diff --git a/Lib/keyword.py b/Lib/keyword.py --- a/Lib/keyword.py +++ b/Lib/keyword.py @@ -7,7 +7,7 @@ To update the symbols in this file, 'cd' to the top directory of the python source tree after building the interpreter and run: - python Lib/keyword.py + ./python Lib/keyword.py """ __all__ = ["iskeyword", "kwlist"] diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -1598,8 +1598,7 @@ # to include every locale up to Windows Vista. # # NOTE: this mapping is incomplete. If your language is missing, please -# submit a bug report to Python bug manager, which you can find via: -# http://www.python.org/dev/ +# submit a bug report to the Python bug tracker at http://bugs.python.org/ # Make sure you include the missing language identifier and the suggested # locale code. # diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -7,7 +7,7 @@ # To update the symbols in this file, 'cd' to the top directory of # the python source tree after building the interpreter and run: # -# python Lib/symbol.py +# ./python Lib/symbol.py #--start constants-- single_input = 256 diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -177,6 +177,7 @@ import platform import random import re +import shutil import signal import sys import sysconfig @@ -1042,6 +1043,23 @@ asyncore.close_all(ignore_all=True) asyncore.socket_map.update(saved_map) + def get_shutil_archive_formats(self): + # we could call get_archives_formats() but that only returns the + # registry keys; we want to check the values too (the functions that + # are registered) + return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy() + def restore_shutil_archive_formats(self, saved): + shutil._ARCHIVE_FORMATS = saved[0] + shutil._ARCHIVE_FORMATS.clear() + shutil._ARCHIVE_FORMATS.update(saved[1]) + + def get_shutil_unpack_formats(self): + return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy() + def restore_shutil_unpack_formats(self, saved): + shutil._UNPACK_FORMATS = saved[0] + shutil._UNPACK_FORMATS.clear() + shutil._UNPACK_FORMATS.update(saved[1]) + def get_logging__handlers(self): # _handlers is a WeakValueDictionary return id(logging._handlers), logging._handlers, logging._handlers.copy() diff --git a/Lib/token.py b/Lib/token.py --- a/Lib/token.py +++ b/Lib/token.py @@ -7,7 +7,7 @@ # To update the symbols in this file, 'cd' to the top directory of # the python source tree after building the interpreter and run: # -# python Lib/token.py +# ./python Lib/token.py #--start constants-- ENDMARKER = 0 diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -978,6 +978,7 @@ Bennett Todd R Lindsay Todd Eugene Toder +Erik Tollerud Matias Torchinsky Sandro Tosi Richard Townsend diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -399,6 +399,9 @@ Library ------- +- Issue #13487: Make inspect.getmodule robust against changes done to + sys.modules while it is iterating over it. + - Issue #12618: Fix a bug that prevented py_compile from creating byte compiled files in the current directory. Initial patch by Sjoerd de Vries. diff --git a/Misc/python.man b/Misc/python.man --- a/Misc/python.man +++ b/Misc/python.man @@ -409,7 +409,7 @@ .br Documentation: http://docs.python.org/py3k/ .br -Developer resources: http://www.python.org/dev/ +Developer resources: http://docs.python.org/devguide/ .br Downloads: http://python.org/download/ .br -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix inspect.getmodule to use a copy of sys.modules for iteration (#13487).
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/2ef359d7a2e9 changeset: 73789:2ef359d7a2e9 branch: 3.2 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 16:58:53 2011 +0100 summary: Fix inspect.getmodule to use a copy of sys.modules for iteration (#13487). This fixes a regression compared to 2.x, where sys.modules.items() returns a copy, as indicated by a comment in the source. Diagnosis and patch by Erik Tollerud. files: Lib/inspect.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -483,7 +483,7 @@ return sys.modules.get(modulesbyfile[file]) # Update the filename to module name cache and check yet again # Copy sys.modules in order to cope with changes while iterating - for modname, module in sys.modules.items(): + for modname, module in list(sys.modules.items()): if ismodule(module) and hasattr(module, '__file__'): f = module.__file__ if f == _filesbymodname.get(modname, None): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -905,6 +905,7 @@ Frank J. Tobin R Lindsay Todd Bennett Todd +Erik Tollerud Matias Torchinsky Sandro Tosi Richard Townsend diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -87,6 +87,9 @@ Library ------- +- Issue #13487: Make inspect.getmodule robust against changes done to + sys.modules while it is iterating over it. + - Issue #12618: Fix a bug that prevented py_compile from creating byte compiled files in the current directory. Initial patch by Sjoerd de Vries. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Mark up constants in sockets docs as such
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/5641a7c44f2b changeset: 73788:5641a7c44f2b branch: 3.2 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 16:46:38 2011 +0100 summary: Mark up constants in sockets docs as such files: Doc/library/socket.rst | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -64,20 +64,20 @@ tuple, and the fields depend on the address type. The general tuple form is ``(addr_type, v1, v2, v3 [, scope])``, where: - - *addr_type* is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or - TIPC_ADDR_ID. - - *scope* is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and - TIPC_NODE_SCOPE. - - If *addr_type* is TIPC_ADDR_NAME, then *v1* is the server type, *v2* is + - *addr_type* is one of :const:`TIPC_ADDR_NAMESEQ`, :const:`TIPC_ADDR_NAME`, + or :const:`TIPC_ADDR_ID`. + - *scope* is one of :const:`TIPC_ZONE_SCOPE`, :const:`TIPC_CLUSTER_SCOPE`, and + :const:`TIPC_NODE_SCOPE`. + - If *addr_type* is :const:`TIPC_ADDR_NAME`, then *v1* is the server type, *v2* is the port identifier, and *v3* should be 0. - If *addr_type* is TIPC_ADDR_NAMESEQ, then *v1* is the server type, *v2* + If *addr_type* is :const:`TIPC_ADDR_NAMESEQ`, then *v1* is the server type, *v2* is the lower port number, and *v3* is the upper port number. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix instructions on how to rebuild some modules
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/0f7253d6b1cc changeset: 73787:0f7253d6b1cc branch: 3.2 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 16:45:34 2011 +0100 summary: Fix instructions on how to rebuild some modules files: Lib/keyword.py | 2 +- Lib/symbol.py | 2 +- Lib/token.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/keyword.py b/Lib/keyword.py --- a/Lib/keyword.py +++ b/Lib/keyword.py @@ -7,7 +7,7 @@ To update the symbols in this file, 'cd' to the top directory of the python source tree after building the interpreter and run: - python Lib/keyword.py + ./python Lib/keyword.py """ __all__ = ["iskeyword", "kwlist"] diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -7,7 +7,7 @@ # To update the symbols in this file, 'cd' to the top directory of # the python source tree after building the interpreter and run: # -# python Lib/symbol.py +# ./python Lib/symbol.py #--start constants-- single_input = 256 diff --git a/Lib/token.py b/Lib/token.py --- a/Lib/token.py +++ b/Lib/token.py @@ -7,7 +7,7 @@ # To update the symbols in this file, 'cd' to the top directory of # the python source tree after building the interpreter and run: # -# python Lib/token.py +# ./python Lib/token.py #--start constants-- ENDMARKER = 0 -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Add a regrtest cleanup check for shutil registries
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/7292a6c21fe5 changeset: 73786:7292a6c21fe5 branch: 3.2 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 16:45:07 2011 +0100 summary: Add a regrtest cleanup check for shutil registries files: Lib/test/regrtest.py | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -165,6 +165,7 @@ import platform import random import re +import shutil import sys import sysconfig import tempfile @@ -887,6 +888,7 @@ 'os.environ', 'sys.path', 'sys.path_hooks', '__import__', 'warnings.filters', 'asyncore.socket_map', 'logging._handlers', 'logging._handlerList', + 'shutil.archive_formats', 'shutil.unpack_formats', 'sys.warnoptions', 'threading._dangling', 'multiprocessing.process._dangling') @@ -956,6 +958,23 @@ asyncore.close_all(ignore_all=True) asyncore.socket_map.update(saved_map) + def get_shutil_archive_formats(self): + # we could call get_archives_formats() but that only returns the + # registry keys; we want to check the values too (the functions that + # are registered) + return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy() + def restore_shutil_archive_formats(self, saved): + shutil._ARCHIVE_FORMATS = saved[0] + shutil._ARCHIVE_FORMATS.clear() + shutil._ARCHIVE_FORMATS.update(saved[1]) + + def get_shutil_unpack_formats(self): + return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy() + def restore_shutil_unpack_formats(self, saved): + shutil._UNPACK_FORMATS = saved[0] + shutil._UNPACK_FORMATS.clear() + shutil._UNPACK_FORMATS.update(saved[1]) + def get_logging__handlers(self): # _handlers is a WeakValueDictionary return id(logging._handlers), logging._handlers, logging._handlers.copy() -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix last remaining references to ex-devguide
by eric.araujo Nov. 29, 2011

Nov. 29, 2011

http://hg.python.org/cpython/rev/49d69219fa59 changeset: 73785:49d69219fa59 branch: 3.2 user: Éric Araujo <merwok(a)netwok.org> date: Tue Nov 29 16:36:17 2011 +0100 summary: Fix last remaining references to ex-devguide files: Lib/locale.py | 3 +-- Misc/python.man | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -1598,8 +1598,7 @@ # to include every locale up to Windows Vista. # # NOTE: this mapping is incomplete. If your language is missing, please -# submit a bug report to Python bug manager, which you can find via: -# http://www.python.org/dev/ +# submit a bug report to the Python bug tracker at http://bugs.python.org/ # Make sure you include the missing language identifier and the suggested # locale code. # diff --git a/Misc/python.man b/Misc/python.man --- a/Misc/python.man +++ b/Misc/python.man @@ -420,7 +420,7 @@ .br Documentation: http://docs.python.org/py3k/ .br -Developer resources: http://www.python.org/dev/ +Developer resources: http://docs.python.org/devguide/ .br Downloads: http://python.org/download/ .br -- Repository URL: http://hg.python.org/cpython

1 0