Mailman 3 November 2015 - Python-checkins

Daily reference leaks (9fcfdb53e8af): sum=4
by solipsis＠pitrou.net Nov. 27, 2015

Nov. 27, 2015

results for 9fcfdb53e8af on branch "default" -------------------------------------------- test_functools leaked [0, 2, 2] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogGMV3oi', '--timeout', '7200']

1 0

peps: PEP 493: make word order consistent
by nick.coghlan Nov. 27, 2015

Nov. 27, 2015

https://hg.python.org/peps/rev/3cf229a9457a changeset: 6133:3cf229a9457a user: Nick Coghlan <ncoghlan(a)gmail.com> date: Fri Nov 27 17:46:37 2015 +1000 summary: PEP 493: make word order consistent files: pep-0493.txt | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/pep-0493.txt b/pep-0493.txt --- a/pep-0493.txt +++ b/pep-0493.txt @@ -128,7 +128,7 @@ ========================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x -releases. It consists of a new ``ssl._verify_https_certificates()`` to specify +releases. It consists of a new ``ssl._https_verify_certificates()`` to specify the default handling of HTTPS certificates in standard library client libraries. It is not proposed to forward port this change to Python 3, so Python 3 @@ -139,14 +139,14 @@ ----------------- The marker attribute on the ``ssl`` module related to this feature is the -``ssl._verify_https_certificates`` function itself. +``ssl._https_verify_certificates`` function itself. Specification ------------- -The ``ssl._verify_https_certificates`` function will work as follows:: +The ``ssl._https_verify_certificates`` function will work as follows:: - def _verify_https_certificates(enable=True): + def _https_verify_certificates(enable=True): """Verify server HTTPS certificates by default?""" global _create_default_https_context if enable: @@ -166,11 +166,11 @@ The inclusion of this feature will allow security sensitive applications to include the following forward-compatible snippet in their code:: - if hasattr(ssl, "_verify_https_certificates"): - ssl._verify_https_certificates() + if hasattr(ssl, "_https_verify_certificates"): + ssl._https_verify_certificates() Some developers may also choose to opt out of certificate checking using -``ssl._verify_https_certificates(enable=False)``. This doesn't introduce any +``ssl._https_verify_certificates(enable=False)``. This doesn't introduce any major new security concerns, as monkeypatching the affected internal APIs was already possible. @@ -449,7 +449,7 @@ ----------------- There's no specific attribute indicating that this situation applies. Rather, -it is indicated by the ``ssl._verify_https_certificates`` and +it is indicated by the ``ssl._https_verify_certificates`` and ``ssl._https_verify_envvar`` attributes being present in a Python version that is nominally older than Python 2.7.9. -- Repository URL: https://hg.python.org/peps

1 0

cpython (2.7): Issue #23914: Fix test_xpickle with python 2.4 and 2.5
by zach.ware Nov. 27, 2015

Nov. 27, 2015

https://hg.python.org/cpython/rev/322060740b64 changeset: 99378:322060740b64 branch: 2.7 parent: 99374:0eea57ddb75f user: Zachary Ware <zachary.ware(a)gmail.com> date: Fri Nov 27 01:21:51 2015 -0600 summary: Issue #23914: Fix test_xpickle with python 2.4 and 2.5 Remove 'b' prefix from strings, remove unused import. files: Lib/test/pickletester.py | 105 +++++++++++++------------- 1 files changed, 52 insertions(+), 53 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -7,7 +7,7 @@ import pickletools import copy_reg -from test.test_support import TestFailed, verbose, have_unicode, TESTFN, captured_stdout +from test.test_support import TestFailed, verbose, have_unicode, TESTFN try: from test.test_support import _2G, _1M, precisionbigmemtest except ImportError: @@ -636,41 +636,41 @@ def test_bad_stack(self): badpickles = [ - b'0.', # POP - b'1.', # POP_MARK - b'2.', # DUP - # b'(2.', # PyUnpickler doesn't raise - b'R.', # REDUCE - b')R.', - b'a.', # APPEND - b'Na.', - b'b.', # BUILD - b'Nb.', - b'd.', # DICT - b'e.', # APPENDS - # b'(e.', # PyUnpickler raises AttributeError - b'i__builtin__\nlist\n.', # INST - b'l.', # LIST - b'o.', # OBJ - b'(o.', - b'p1\n.', # PUT - b'q\x00.', # BINPUT - b'r\x00\x00\x00\x00.', # LONG_BINPUT - b's.', # SETITEM - b'Ns.', - b'NNs.', - b't.', # TUPLE - b'u.', # SETITEMS - b'(u.', - b'}(Nu.', - b'\x81.', # NEWOBJ - b')\x81.', - b'\x85.', # TUPLE1 - b'\x86.', # TUPLE2 - b'N\x86.', - b'\x87.', # TUPLE3 - b'N\x87.', - b'NN\x87.', + '0.', # POP + '1.', # POP_MARK + '2.', # DUP + # '(2.', # PyUnpickler doesn't raise + 'R.', # REDUCE + ')R.', + 'a.', # APPEND + 'Na.', + 'b.', # BUILD + 'Nb.', + 'd.', # DICT + 'e.', # APPENDS + # '(e.', # PyUnpickler raises AttributeError + 'i__builtin__\nlist\n.', # INST + 'l.', # LIST + 'o.', # OBJ + '(o.', + 'p1\n.', # PUT + 'q\x00.', # BINPUT + 'r\x00\x00\x00\x00.', # LONG_BINPUT + 's.', # SETITEM + 'Ns.', + 'NNs.', + 't.', # TUPLE + 'u.', # SETITEMS + '(u.', + '}(Nu.', + '\x81.', # NEWOBJ + ')\x81.', + '\x85.', # TUPLE1 + '\x86.', # TUPLE2 + 'N\x86.', + '\x87.', # TUPLE3 + 'N\x87.', + 'NN\x87.', ] for p in badpickles: try: @@ -681,25 +681,24 @@ def test_bad_mark(self): badpickles = [ - b'c__builtin__\nlist\n)(R.', # REDUCE - b'c__builtin__\nlist\n()R.', - b']N(a.', # APPEND - b'cexceptions\nValueError\n)R}(b.', # BUILD - b'cexceptions\nValueError\n)R(}b.', - b'(Nd.', # DICT - b'}NN(s.', # SETITEM - b'}N(Ns.', - b'c__builtin__\nlist\n)(\x81.', # NEWOBJ - b'c__builtin__\nlist\n()\x81.', - b'N(\x85.', # TUPLE1 - b'NN(\x86.', # TUPLE2 - b'N(N\x86.', - b'NNN(\x87.', # TUPLE3 - b'NN(N\x87.', - b'N(NN\x87.', + 'c__builtin__\nlist\n)(R.', # REDUCE + 'c__builtin__\nlist\n()R.', + ']N(a.', # APPEND + 'cexceptions\nValueError\n)R}(b.', # BUILD + 'cexceptions\nValueError\n)R(}b.', + '(Nd.', # DICT + '}NN(s.', # SETITEM + '}N(Ns.', + 'c__builtin__\nlist\n)(\x81.', # NEWOBJ + 'c__builtin__\nlist\n()\x81.', + 'N(\x85.', # TUPLE1 + 'NN(\x86.', # TUPLE2 + 'N(N\x86.', + 'NNN(\x87.', # TUPLE3 + 'NN(N\x87.', + 'N(NN\x87.', ] for p in badpickles: - # PyUnpickler prints reduce errors to stdout try: self.loads(p) except (IndexError, AttributeError, TypeError, -- Repository URL: https://hg.python.org/cpython

1 0

peps: PEP 493: First draft as a standards track PEP
by nick.coghlan Nov. 27, 2015

Nov. 27, 2015

https://hg.python.org/peps/rev/f602a47ea795 changeset: 6132:f602a47ea795 user: Nick Coghlan <ncoghlan(a)gmail.com> date: Fri Nov 27 15:51:43 2015 +1000 summary: PEP 493: First draft as a standards track PEP - now has 2 feature sections, and 3 backport recommendations - features are for a new Python API and environment setting - backports are all for targeting versions < 2.7.9 - Python 3 is excluded from scope files: pep-0493.txt | 333 ++++++++++++++++++++++++++++---------- 1 files changed, 240 insertions(+), 93 deletions(-) diff --git a/pep-0493.txt b/pep-0493.txt --- a/pep-0493.txt +++ b/pep-0493.txt @@ -1,78 +1,91 @@ PEP: 493 -Title: HTTPS verification recommendations for Python 2.7 redistributors +Title: HTTPS verification migration tools for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Robert Kuska <rkuska(a)redhat.com>, Marc-André Lemburg <mal(a)lemburg.com> Status: Draft -Type: Informational +Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015 +Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015 Abstract ======== -PEP 476 updated Python's default handling of HTTPS certificates to be -appropriate for communication over the public internet. The Python 2.7 long -term maintenance series was judged to be in scope for this change, with the -new behaviour introduced in the Python 2.7.9 maintenance release. +PEP 476 updated Python's default handling of HTTPS certificates in client +modules to align with certificate handling in web browsers, by validating +that the certificates received belonged to the server the client was attempting +to contact. The Python 2.7 long term maintenance series was judged to be in +scope for this change, with the new behaviour introduced in the Python 2.7.9 +maintenance release. -This PEP provides recommendations to downstream redistributors wishing to -provide a smoother migration experience when helping their users to manage -this change in Python's default behaviour. - -*Note that this PEP is not currently accepted, so it is a *proposed* -recommendation, rather than an active one.* +This has created a non-trivial barrier to adoption for affected Python 2.7 +maintenance releases, so this PEP proposes additional Python 2.7 specific +features that allow system administrators and other users to more easily +decouple the decision to verify server certificates in HTTPS client modules +from the decision to update to newer Python 2.7 maintenance releases. Rationale ========= -PEP 476 changed Python's default behaviour to better match the needs and -expectations of developers operating over the public internet, a category -which appears to include most new Python developers. It is the position of -the authors of this PEP that this was a correct decision. +PEP 476 changed Python's default behaviour to align with expectations +established by web browsers in regards to the semantics of HTTPS URLs: +starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library +validate server certificates by default. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. -The long term answer for such environments is to update their internal -certificate management to at least match the standards set by the public -internet, but in the meantime, it is desirable to offer these administrators -a way to continue receiving maintenance updates to the Python 2.7 series, -without having to gate that on upgrades to their certificate management -infrastructure. +To manage these kinds of situations, web browsers provide users with "click +through" warnings that allow the user to add the server's certificate to the +browser's certificate store. Network client tools like ``curl`` and ``wget`` +offer options to switch off certificate checking entirely (by way of +``curl --insecure`` and ``wget --no-check-certificate``, respectively). -PEP 476 did attempt to address this question, by covering how to revert the -new settings process wide by monkeypatching the ``ssl`` module to restore the +At a different layer of the technology stack, Linux security modules like +`SELinux` and `AppArmor`, while enabled by default by distribution vendors, +offer relatively straightforward mechanisms for turning them off. + +At the moment, no such convenient mechanisms exist to disable Python's +default certificate checking for a whole process. + +PEP 476 did attempt to address this question, by covering how to revert to the +old settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be -insufficient in at least some cases. The specific case of interest to the -authors of this PEP is the one where a Linux distributor aims to provide -their users with a +insufficient in at least some cases. The specific case that led to the +initial creation of this PEP is the one where a Linux distributor aims to +provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques -to bloom, this PEP proposes two alternative approaches that redistributors -may take when addressing these problems. Redistributors may choose to implement -one, both, or neither of these approaches based on their assessment of the -needs of their particular userbase. +to bloom, this PEP proposes an additional feature to be added to Python 2.7.12 +to make it easier to revert a process to the past behaviour of skipping +certificate validation in HTTPS client modules. It also provides additional +recommendations to redistributors backporting these features to versions of +Python prior to Python 2.7.9. -These designs are being proposed as a recommendation for redistributors, rather -than as new upstream features, as they are needed purely to support legacy -environments migrating from older versions of Python 2.7. Neither approach -is being proposed as an upstream Python 2.7 feature, nor as a feature in any -version of Python 3 (whether published directly by the Python Software -Foundation or by a redistributor). +These designs are being proposed purely as tools for helping to manage the +transition to the new default certificate handling behaviour in the context +of Python 2.7. They are not being proposed as new features for Python 3, as +it is expected that the vast majority of client applications affected by this +problem without the ability to update the application itself will be Python 2 +applications. + +It would likely be desirable for a future version of Python 3 to allow default +certificate handling for secure protocols to be configurable on a per-protocol +basis, but that question is beyond the scope of this PEP. Alternatives ------------ @@ -84,20 +97,20 @@ * Continuing to rebase on new Python 2.7.x releases, while providing no additional assistance beyond the mechanisms defined in PEP 476 in migrating from unchecked to checked hostnames in standard library HTTPS clients -* Gating availability of the improved default handling of HTTPS connections on - upgrading from Python 2 to Python 3 -* For Linux distribution vendors, gating availability of the improved default +* Gating availability of the changes in default handling of HTTPS connections + on upgrading from Python 2 to Python 3 +* For Linux distribution vendors, gating availability of the changes in default handling of HTTPS connections on upgrading to a new operating system version -* Implementing one or both of the design suggestions described in this PEP, +* Implementing one or both of the backport suggestions described in this PEP, regardless of the formal status of the PEP Requirements for capability detection ===================================== -As these recommendations are intended to cover backports to earlier Python +As the proposals in this PEP aim to facilitate backports to earlier Python versions, the Python version number cannot be used as a reliable means for -detecting them. Instead, the recommendations are defined to allow the presence +detecting them. Instead, they are designed to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" @@ -108,57 +121,99 @@ The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage -migration process from the legacy HTTPS handling to the new default behaviour. +migration process from the original Python 2.7 HTTPS handling to the new +default behaviour. +Feature: Configuration API +========================== -Recommendation for an environment variable based security downgrade -=================================================================== +This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x +releases. It consists of a new ``ssl._verify_https_certificates()`` to specify +the default handling of HTTPS certificates in standard library client libraries. -Some redistributors may wish to provide a per-application option to disable -certificate verification in selected applications that run on or embed CPython -without needing to modify the application itself. +It is not proposed to forward port this change to Python 3, so Python 3 +applications that need to support skipping certificate verification will still +need to define their own suitable security context. -In these cases, a configuration mechanism is needed that provides: +Feature detection +----------------- -* an opt-out model that allows certificate verification to be selectively - turned off for particular applications after upgrading to a version of - Python that verifies certificates by default -* the ability for all users to configure this setting on a per-application - basis, rather than on a per-system, or per-Python-installation basis +The marker attribute on the ``ssl`` module related to this feature is the +``ssl._verify_https_certificates`` function itself. -This approach may be used for any redistributor provided version of Python 2.7, -including those that advertise themselves as providing Python 2.7.9 or later. +Specification +------------- +The ``ssl._verify_https_certificates`` function will work as follows:: -Required marker attribute -------------------------- + def _verify_https_certificates(enable=True): + """Verify server HTTPS certificates by default?""" + global _create_default_https_context + if enable: + _create_default_https_context = create_default_context + else: + _create_default_https_context = _create_unverified_context -The required marker attribute on the ``ssl`` module when implementing this -recommendation is:: +If called without arguments, or with ``enable`` set to a true value, then +standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification. - _https_verify_envvar = 'PYTHONHTTPSVERIFY' +If called with ``enable`` set to a false value, then standard library client +modules will subsequently skip verifying HTTPS certificates by default. + +Security Considerations +----------------------- + +The inclusion of this feature will allow security sensitive applications to +include the following forward-compatible snippet in their code:: + + if hasattr(ssl, "_verify_https_certificates"): + ssl._verify_https_certificates() + +Some developers may also choose to opt out of certificate checking using +``ssl._verify_https_certificates(enable=False)``. This doesn't introduce any +major new security concerns, as monkeypatching the affected internal APIs was +already possible. + + +Feature: environment based configuration +======================================== + +This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x +releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that +allows the default verification to be disabled without modifying the +application source code (which may not even be available in cases of +bytecode-only application distribution) + +It is not proposed to forward port this change to Python 3, so Python 3 +applications that need to support skipping certificate verification will still +need to define their own suitable security context. + +Feature detection +----------------- + +The marker attribute on the ``ssl`` module related to this feature is: + +* the ``ssl._https_verify_envvar`` attribute, giving the name of environment + variable affecting the default behaviour This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. +Specification +------------- -Recommended modifications to the Python standard library --------------------------------------------------------- +Rather than always defaulting to the use of ``ssl.create_default_context``, +the ``ssl`` module will be modified to: -The recommended approach to providing a per-application configuration setting -for HTTPS certificate verification that doesn't require modifications to the -application itself is to: - -* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment - variable when the module is first imported into a Python process +* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first + imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual - Example implementation ---------------------- @@ -175,14 +230,13 @@ _create_default_https_context = _get_https_context_factory() - Security Considerations ----------------------- -Relative to an unmodified version of CPython 2.7.9 or later, this approach -does introduce a new downgrade attack against the default security settings -that potentially allows a sufficiently determined attacker to revert Python -to the vulnerable configuration used in CPython 2.7.8 and earlier releases. +Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this +approach does introduce a new downgrade attack against the default security +settings that potentially allows a sufficiently determined attacker to revert +Python to the default behaviour used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the @@ -191,13 +245,13 @@ Interaction with Python virtual environments -------------------------------------------- -This setting is read directly from the process environment, and hence works -the same way regardless of whether or not the interpreter is being run inside -an activated Python virtual environment. +The default setting is read directly from the process environment, and hence +works the same way regardless of whether or not the interpreter is being run +inside an activated Python virtual environment. -Recommendation for backporting to earlier Python versions -========================================================= +Backporting PEP 476 to earlier Python versions +============================================== Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on @@ -214,17 +268,21 @@ installations that have been explicitly configured to skip verifying HTTPS certificates by default -This approach should not be used for any Python installation that advertises +As it only affects backports to earlier releases of Python 2.7, this change is +not proposed for inclusion in upstream CPython, but rather is offered as +guidance to redistributors to reduce the likelihood of multiple mutually +incompatible approaches to backporting being adopted. + +This approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. -Required marker attribute -------------------------- +Feature detection +----------------- -The required marker attribute on the ``ssl`` module when implementing this -recommendation is:: +The marker attribute on the ``ssl`` module related to this feature is:: _cert_verification_config = '<path to configuration file>' @@ -320,7 +378,7 @@ Security Considerations ----------------------- -The specific recommendations for the backporting case are designed to work for +The specific recommendations for this backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: @@ -356,19 +414,108 @@ processes using that interpreter, regardless of whether or not the interpreter is being run inside an activated Python virtual environment. +Origins of this recommendation +------------------------------ -Combining the recommendations -============================= +This recommendation is based on the backporting approach adopted for Red Hat +Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP +and described in detail in `this KnowledgeBase article +<https://access.redhat.com/articles/2039753>`__. Red Hat's patches implementing +this backport for Python 2.7.5 can be found in the `CentOS git repository +<https://git.centos.org/commit/rpms!python.git/refs!heads!c7>`__. -If a redistributor chooses to implement both recommendations, then the -environment variable should take precedence over the system-wide configuration -setting. This allows the setting to be changed for a given user, virtual -environment or application, regardless of the system-wide default behaviour. + +Backporting this PEP to earlier Python versions +=============================================== + +The configuration file based backport described above is designed to cover +backporting the PEP 476 changes to default certificate handling without the +additional configuration mechanisms defined in this PEP. + +If this PEP is accepted, then an additional backporting option becomes +available, which is to backport the per-process configuration mechanisms +defined in this PEP, without backporting the ability to change the default behaviour of the overall Python installation. + +Such a backport would differ from the mechanism proposed in this PEP solely in +the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would +continue to default to skipping certificate validation. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. + +Feature detection +----------------- + +There's no specific attribute indicating that this situation applies. Rather, +it is indicated by the ``ssl._verify_https_certificates`` and +``ssl._https_verify_envvar`` attributes being present in a Python version that +is nominally older than Python 2.7.9. + +Specification +------------- + +Implementing this backport involves backporting the changes in PEP 466, 476 and +this PEP, with the following change to the handling of the +``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module: + +* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first + imported into a Python process +* set the ``ssl._create_default_https_context`` function to be an alias for + ``ssl.create_default_context`` if this environment variable is present + and set to any value other than ``'0'`` +* otherwise, set the ``ssl._create_default_https_context`` function to be an + alias for ``ssl._create_unverified_context`` + +Example implementation +---------------------- + +:: + + _https_verify_envvar = 'PYTHONHTTPSVERIFY' + + def _get_https_context_factory(): + if not sys.flags.ignore_environment: + config_setting = os.environ.get(_https_verify_envvar) + if config_setting != '0': + return create_default_context + return _create_unverified_context + + _create_default_https_context = _get_https_context_factory() + + def _disable_https_default_verification(): + """Skip verification of HTTPS certificates by default""" + global _create_default_https_context + _create_default_https_context = _create_unverified_context + +Security Considerations +----------------------- + +This change would be a strict security upgrade for any Python version that +currently defaults to skipping certificate validation in standard library +HTTPS clients. The technical trade-offs to be taken into account relate largely +to the magnitude of the PEP 466 backport also required rather than to anything +security related. + +Interaction with Python virtual environments +-------------------------------------------- + +The default setting is read directly from the process environment, and hence +works the same way regardless of whether or not the interpreter is being run +inside an activated Python virtual environment. + + +Recommendation for combined feature backports +============================================= + +If a redistributor chooses to backport the environment variable based +configuration setting from this PEP to a modified Python version that also +implements the configuration file based PEP 476 , then the environment +variable should take precedence over the system-wide configuration setting. +This allows the setting to be changed for a given user or application, +regardless of the installation-wide default behaviour. + Example implementation ---------------------- -- Repository URL: https://hg.python.org/peps

1 0

cpython (merge 3.5 -> default): Issue #10131: Fixed deep copying of minidom documents. Based on patch
by serhiy.storchaka Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/cpython/rev/9fcfdb53e8af changeset: 99377:9fcfdb53e8af parent: 99372:c6930661599b parent: 99376:5d6b2dc7e3d0 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Thu Nov 26 23:52:34 2015 +0200 summary: Issue #10131: Fixed deep copying of minidom documents. Based on patch by Marian Ganisin. files: Lib/test/test_minidom.py | 94 +++++++++------- Lib/test/test_xml_dom_minicompat.py | 36 ++++++ Lib/xml/dom/minicompat.py | 6 +- Misc/NEWS | 3 + 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -1,5 +1,6 @@ # test for xml.dom.minidom +import copy import pickle from test.support import findfile import unittest @@ -11,6 +12,13 @@ tstfile = findfile("test.xml", subdir="xmltestdata") +sample = ("<?xml version='1.0' encoding='us-ascii'?>\n" + "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" + " 'http://xml.python.org/system' [\n" + " <!ELEMENT e EMPTY>\n" + " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" + "]><doc attr='value'> text\n" + "<?pi sample?>  <e/> </doc>") # The tests of DocumentType importing use these helpers to construct # the documents to work with, since not all DOM builders actually @@ -1481,52 +1489,54 @@ self.confirm(e.isSameNode(doc.getElementById("w")) and a2.isId) + def assert_recursive_equal(self, doc, doc2): + stack = [(doc, doc2)] + while stack: + n1, n2 = stack.pop() + self.assertEqual(n1.nodeType, n2.nodeType) + self.assertEqual(len(n1.childNodes), len(n2.childNodes)) + self.assertEqual(n1.nodeName, n2.nodeName) + self.assertFalse(n1.isSameNode(n2)) + self.assertFalse(n2.isSameNode(n1)) + if n1.nodeType == Node.DOCUMENT_TYPE_NODE: + len(n1.entities) + len(n2.entities) + len(n1.notations) + len(n2.notations) + self.assertEqual(len(n1.entities), len(n2.entities)) + self.assertEqual(len(n1.notations), len(n2.notations)) + for i in range(len(n1.notations)): + # XXX this loop body doesn't seem to be executed? + no1 = n1.notations.item(i) + no2 = n1.notations.item(i) + self.assertEqual(no1.name, no2.name) + self.assertEqual(no1.publicId, no2.publicId) + self.assertEqual(no1.systemId, no2.systemId) + stack.append((no1, no2)) + for i in range(len(n1.entities)): + e1 = n1.entities.item(i) + e2 = n2.entities.item(i) + self.assertEqual(e1.notationName, e2.notationName) + self.assertEqual(e1.publicId, e2.publicId) + self.assertEqual(e1.systemId, e2.systemId) + stack.append((e1, e2)) + if n1.nodeType != Node.DOCUMENT_NODE: + self.assertTrue(n1.ownerDocument.isSameNode(doc)) + self.assertTrue(n2.ownerDocument.isSameNode(doc2)) + for i in range(len(n1.childNodes)): + stack.append((n1.childNodes[i], n2.childNodes[i])) + def testPickledDocument(self): - doc = parseString("<?xml version='1.0' encoding='us-ascii'?>\n" - "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" - " 'http://xml.python.org/system' [\n" - " <!ELEMENT e EMPTY>\n" - " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" - "]><doc attr='value'> text\n" - "<?pi sample?>  <e/> </doc>") + doc = parseString(sample) for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): s = pickle.dumps(doc, proto) doc2 = pickle.loads(s) - stack = [(doc, doc2)] - while stack: - n1, n2 = stack.pop() - self.confirm(n1.nodeType == n2.nodeType - and len(n1.childNodes) == len(n2.childNodes) - and n1.nodeName == n2.nodeName - and not n1.isSameNode(n2) - and not n2.isSameNode(n1)) - if n1.nodeType == Node.DOCUMENT_TYPE_NODE: - len(n1.entities) - len(n2.entities) - len(n1.notations) - len(n2.notations) - self.confirm(len(n1.entities) == len(n2.entities) - and len(n1.notations) == len(n2.notations)) - for i in range(len(n1.notations)): - # XXX this loop body doesn't seem to be executed? - no1 = n1.notations.item(i) - no2 = n1.notations.item(i) - self.confirm(no1.name == no2.name - and no1.publicId == no2.publicId - and no1.systemId == no2.systemId) - stack.append((no1, no2)) - for i in range(len(n1.entities)): - e1 = n1.entities.item(i) - e2 = n2.entities.item(i) - self.confirm(e1.notationName == e2.notationName - and e1.publicId == e2.publicId - and e1.systemId == e2.systemId) - stack.append((e1, e2)) - if n1.nodeType != Node.DOCUMENT_NODE: - self.confirm(n1.ownerDocument.isSameNode(doc) - and n2.ownerDocument.isSameNode(doc2)) - for i in range(len(n1.childNodes)): - stack.append((n1.childNodes[i], n2.childNodes[i])) + self.assert_recursive_equal(doc, doc2) + + def testDeepcopiedDocument(self): + doc = parseString(sample) + doc2 = copy.deepcopy(doc) + self.assert_recursive_equal(doc, doc2) def testSerializeCommentNodeWithDoubleHyphen(self): doc = create_doc_without_doctype() diff --git a/Lib/test/test_xml_dom_minicompat.py b/Lib/test/test_xml_dom_minicompat.py --- a/Lib/test/test_xml_dom_minicompat.py +++ b/Lib/test/test_xml_dom_minicompat.py @@ -1,5 +1,6 @@ # Tests for xml.dom.minicompat +import copy import pickle import unittest @@ -89,6 +90,7 @@ node_list = NodeList() pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) # Non-empty NodeList. @@ -96,7 +98,41 @@ node_list.append(2) pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) + def test_nodelist_copy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIs(x, y) + + def test_nodelist_deepcopy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIsNot(x, y) + self.assertEqual(x, y) + if __name__ == '__main__': unittest.main() diff --git a/Lib/xml/dom/minicompat.py b/Lib/xml/dom/minicompat.py --- a/Lib/xml/dom/minicompat.py +++ b/Lib/xml/dom/minicompat.py @@ -64,10 +64,10 @@ length = property(_get_length, _set_length, doc="The number of nodes in the NodeList.") - def __getstate__(self): - return list(self) - + # For backward compatibility def __setstate__(self, state): + if state is None: + state = [] self[:] = state diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -102,6 +102,9 @@ Library ------- +- Issue #10131: Fixed deep copying of minidom documents. Based on patch + by Marian Ganisin. + - Issue #7990: dir() on ElementTree.Element now lists properties: "tag", "text", "tail" and "attrib". Original patch by Santoso Wijaya. -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.4 -> 3.5): Issue #10131: Fixed deep copying of minidom documents. Based on patch
by serhiy.storchaka Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/cpython/rev/5d6b2dc7e3d0 changeset: 99376:5d6b2dc7e3d0 branch: 3.5 parent: 99371:b55c006b79bc parent: 99375:aa304ad32292 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Thu Nov 26 23:50:54 2015 +0200 summary: Issue #10131: Fixed deep copying of minidom documents. Based on patch by Marian Ganisin. files: Lib/test/test_minidom.py | 94 +++++++++------- Lib/test/test_xml_dom_minicompat.py | 36 ++++++ Lib/xml/dom/minicompat.py | 6 +- Misc/NEWS | 3 + 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -1,5 +1,6 @@ # test for xml.dom.minidom +import copy import pickle from test.support import findfile import unittest @@ -11,6 +12,13 @@ tstfile = findfile("test.xml", subdir="xmltestdata") +sample = ("<?xml version='1.0' encoding='us-ascii'?>\n" + "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" + " 'http://xml.python.org/system' [\n" + " <!ELEMENT e EMPTY>\n" + " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" + "]><doc attr='value'> text\n" + "<?pi sample?>  <e/> </doc>") # The tests of DocumentType importing use these helpers to construct # the documents to work with, since not all DOM builders actually @@ -1481,52 +1489,54 @@ self.confirm(e.isSameNode(doc.getElementById("w")) and a2.isId) + def assert_recursive_equal(self, doc, doc2): + stack = [(doc, doc2)] + while stack: + n1, n2 = stack.pop() + self.assertEqual(n1.nodeType, n2.nodeType) + self.assertEqual(len(n1.childNodes), len(n2.childNodes)) + self.assertEqual(n1.nodeName, n2.nodeName) + self.assertFalse(n1.isSameNode(n2)) + self.assertFalse(n2.isSameNode(n1)) + if n1.nodeType == Node.DOCUMENT_TYPE_NODE: + len(n1.entities) + len(n2.entities) + len(n1.notations) + len(n2.notations) + self.assertEqual(len(n1.entities), len(n2.entities)) + self.assertEqual(len(n1.notations), len(n2.notations)) + for i in range(len(n1.notations)): + # XXX this loop body doesn't seem to be executed? + no1 = n1.notations.item(i) + no2 = n1.notations.item(i) + self.assertEqual(no1.name, no2.name) + self.assertEqual(no1.publicId, no2.publicId) + self.assertEqual(no1.systemId, no2.systemId) + stack.append((no1, no2)) + for i in range(len(n1.entities)): + e1 = n1.entities.item(i) + e2 = n2.entities.item(i) + self.assertEqual(e1.notationName, e2.notationName) + self.assertEqual(e1.publicId, e2.publicId) + self.assertEqual(e1.systemId, e2.systemId) + stack.append((e1, e2)) + if n1.nodeType != Node.DOCUMENT_NODE: + self.assertTrue(n1.ownerDocument.isSameNode(doc)) + self.assertTrue(n2.ownerDocument.isSameNode(doc2)) + for i in range(len(n1.childNodes)): + stack.append((n1.childNodes[i], n2.childNodes[i])) + def testPickledDocument(self): - doc = parseString("<?xml version='1.0' encoding='us-ascii'?>\n" - "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" - " 'http://xml.python.org/system' [\n" - " <!ELEMENT e EMPTY>\n" - " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" - "]><doc attr='value'> text\n" - "<?pi sample?>  <e/> </doc>") + doc = parseString(sample) for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): s = pickle.dumps(doc, proto) doc2 = pickle.loads(s) - stack = [(doc, doc2)] - while stack: - n1, n2 = stack.pop() - self.confirm(n1.nodeType == n2.nodeType - and len(n1.childNodes) == len(n2.childNodes) - and n1.nodeName == n2.nodeName - and not n1.isSameNode(n2) - and not n2.isSameNode(n1)) - if n1.nodeType == Node.DOCUMENT_TYPE_NODE: - len(n1.entities) - len(n2.entities) - len(n1.notations) - len(n2.notations) - self.confirm(len(n1.entities) == len(n2.entities) - and len(n1.notations) == len(n2.notations)) - for i in range(len(n1.notations)): - # XXX this loop body doesn't seem to be executed? - no1 = n1.notations.item(i) - no2 = n1.notations.item(i) - self.confirm(no1.name == no2.name - and no1.publicId == no2.publicId - and no1.systemId == no2.systemId) - stack.append((no1, no2)) - for i in range(len(n1.entities)): - e1 = n1.entities.item(i) - e2 = n2.entities.item(i) - self.confirm(e1.notationName == e2.notationName - and e1.publicId == e2.publicId - and e1.systemId == e2.systemId) - stack.append((e1, e2)) - if n1.nodeType != Node.DOCUMENT_NODE: - self.confirm(n1.ownerDocument.isSameNode(doc) - and n2.ownerDocument.isSameNode(doc2)) - for i in range(len(n1.childNodes)): - stack.append((n1.childNodes[i], n2.childNodes[i])) + self.assert_recursive_equal(doc, doc2) + + def testDeepcopiedDocument(self): + doc = parseString(sample) + doc2 = copy.deepcopy(doc) + self.assert_recursive_equal(doc, doc2) def testSerializeCommentNodeWithDoubleHyphen(self): doc = create_doc_without_doctype() diff --git a/Lib/test/test_xml_dom_minicompat.py b/Lib/test/test_xml_dom_minicompat.py --- a/Lib/test/test_xml_dom_minicompat.py +++ b/Lib/test/test_xml_dom_minicompat.py @@ -1,5 +1,6 @@ # Tests for xml.dom.minicompat +import copy import pickle import unittest @@ -89,6 +90,7 @@ node_list = NodeList() pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) # Non-empty NodeList. @@ -96,7 +98,41 @@ node_list.append(2) pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) + def test_nodelist_copy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIs(x, y) + + def test_nodelist_deepcopy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIsNot(x, y) + self.assertEqual(x, y) + if __name__ == '__main__': unittest.main() diff --git a/Lib/xml/dom/minicompat.py b/Lib/xml/dom/minicompat.py --- a/Lib/xml/dom/minicompat.py +++ b/Lib/xml/dom/minicompat.py @@ -64,10 +64,10 @@ length = property(_get_length, _set_length, doc="The number of nodes in the NodeList.") - def __getstate__(self): - return list(self) - + # For backward compatibility def __setstate__(self, state): + if state is None: + state = [] self[:] = state diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #10131: Fixed deep copying of minidom documents. Based on patch + by Marian Ganisin. + - Issue #25725: Fixed a reference leak in pickle.loads() when unpickling invalid data including tuple instructions. -- Repository URL: https://hg.python.org/cpython

1 0

cpython (2.7): Issue #10131: Fixed deep copying of minidom documents. Based on patch
by serhiy.storchaka Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/cpython/rev/0eea57ddb75f changeset: 99374:0eea57ddb75f branch: 2.7 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Thu Nov 26 23:48:30 2015 +0200 summary: Issue #10131: Fixed deep copying of minidom documents. Based on patch by Marian Ganisin. files: Lib/test/test_minidom.py | 97 ++++++++++++++------------ Lib/xml/dom/minicompat.py | 6 +- Misc/NEWS | 3 + 3 files changed, 59 insertions(+), 47 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -1,5 +1,6 @@ # test for xml.dom.minidom +import copy import pickle from StringIO import StringIO from test.test_support import verbose, run_unittest, findfile @@ -14,7 +15,13 @@ tstfile = findfile("test.xml", subdir="xmltestdata") - +sample = ("<?xml version='1.0' encoding='us-ascii'?>\n" + "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" + " 'http://xml.python.org/system' [\n" + " <!ELEMENT e EMPTY>\n" + " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" + "]><doc attr='value'> text\n" + "<?pi sample?>  <e/> </doc>") # The tests of DocumentType importing use these helpers to construct # the documents to work with, since not all DOM builders actually @@ -1377,52 +1384,54 @@ self.confirm(e.isSameNode(doc.getElementById("w")) and a2.isId) + def assert_recursive_equal(self, doc, doc2): + stack = [(doc, doc2)] + while stack: + n1, n2 = stack.pop() + self.assertEqual(n1.nodeType, n2.nodeType) + self.assertEqual(len(n1.childNodes), len(n2.childNodes)) + self.assertEqual(n1.nodeName, n2.nodeName) + self.assertFalse(n1.isSameNode(n2)) + self.assertFalse(n2.isSameNode(n1)) + if n1.nodeType == Node.DOCUMENT_TYPE_NODE: + len(n1.entities) + len(n2.entities) + len(n1.notations) + len(n2.notations) + self.assertEqual(len(n1.entities), len(n2.entities)) + self.assertEqual(len(n1.notations), len(n2.notations)) + for i in range(len(n1.notations)): + # XXX this loop body doesn't seem to be executed? + no1 = n1.notations.item(i) + no2 = n1.notations.item(i) + self.assertEqual(no1.name, no2.name) + self.assertEqual(no1.publicId, no2.publicId) + self.assertEqual(no1.systemId, no2.systemId) + stack.append((no1, no2)) + for i in range(len(n1.entities)): + e1 = n1.entities.item(i) + e2 = n2.entities.item(i) + self.assertEqual(e1.notationName, e2.notationName) + self.assertEqual(e1.publicId, e2.publicId) + self.assertEqual(e1.systemId, e2.systemId) + stack.append((e1, e2)) + if n1.nodeType != Node.DOCUMENT_NODE: + self.assertTrue(n1.ownerDocument.isSameNode(doc)) + self.assertTrue(n2.ownerDocument.isSameNode(doc2)) + for i in range(len(n1.childNodes)): + stack.append((n1.childNodes[i], n2.childNodes[i])) + def testPickledDocument(self): - doc = parseString("<?xml version='1.0' encoding='us-ascii'?>\n" - "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" - " 'http://xml.python.org/system' [\n" - " <!ELEMENT e EMPTY>\n" - " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" - "]><doc attr='value'> text\n" - "<?pi sample?>  <e/> </doc>") - for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): + doc = parseString(sample) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): s = pickle.dumps(doc, proto) doc2 = pickle.loads(s) - stack = [(doc, doc2)] - while stack: - n1, n2 = stack.pop() - self.confirm(n1.nodeType == n2.nodeType - and len(n1.childNodes) == len(n2.childNodes) - and n1.nodeName == n2.nodeName - and not n1.isSameNode(n2) - and not n2.isSameNode(n1)) - if n1.nodeType == Node.DOCUMENT_TYPE_NODE: - len(n1.entities) - len(n2.entities) - len(n1.notations) - len(n2.notations) - self.confirm(len(n1.entities) == len(n2.entities) - and len(n1.notations) == len(n2.notations)) - for i in range(len(n1.notations)): - # XXX this loop body doesn't seem to be executed? - no1 = n1.notations.item(i) - no2 = n1.notations.item(i) - self.confirm(no1.name == no2.name - and no1.publicId == no2.publicId - and no1.systemId == no2.systemId) - stack.append((no1, no2)) - for i in range(len(n1.entities)): - e1 = n1.entities.item(i) - e2 = n2.entities.item(i) - self.confirm(e1.notationName == e2.notationName - and e1.publicId == e2.publicId - and e1.systemId == e2.systemId) - stack.append((e1, e2)) - if n1.nodeType != Node.DOCUMENT_NODE: - self.confirm(n1.ownerDocument.isSameNode(doc) - and n2.ownerDocument.isSameNode(doc2)) - for i in range(len(n1.childNodes)): - stack.append((n1.childNodes[i], n2.childNodes[i])) + self.assert_recursive_equal(doc, doc2) + + def testDeepcopiedDocument(self): + doc = parseString(sample) + doc2 = copy.deepcopy(doc) + self.assert_recursive_equal(doc, doc2) def testSerializeCommentNodeWithDoubleHyphen(self): doc = create_doc_without_doctype() diff --git a/Lib/xml/dom/minicompat.py b/Lib/xml/dom/minicompat.py --- a/Lib/xml/dom/minicompat.py +++ b/Lib/xml/dom/minicompat.py @@ -65,10 +65,10 @@ length = property(_get_length, _set_length, doc="The number of nodes in the NodeList.") - def __getstate__(self): - return list(self) - + # For backward compatibility def __setstate__(self, state): + if state is None: + state = [] self[:] = state diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #10131: Fixed deep copying of minidom documents. Based on patch + by Marian Ganisin. + - Issue #25725: Fixed a reference leak in cPickle.loads() when unpickling invalid data including tuple instructions. -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.4): Issue #10131: Fixed deep copying of minidom documents. Based on patch
by serhiy.storchaka Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/cpython/rev/aa304ad32292 changeset: 99375:aa304ad32292 branch: 3.4 parent: 99370:fa3c9faabfb0 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Thu Nov 26 23:49:42 2015 +0200 summary: Issue #10131: Fixed deep copying of minidom documents. Based on patch by Marian Ganisin. files: Lib/test/test_minidom.py | 94 +++++++++------- Lib/test/test_xml_dom_minicompat.py | 36 ++++++ Lib/xml/dom/minicompat.py | 6 +- Misc/NEWS | 3 + 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -1,5 +1,6 @@ # test for xml.dom.minidom +import copy import pickle from test.support import run_unittest, findfile import unittest @@ -11,6 +12,13 @@ tstfile = findfile("test.xml", subdir="xmltestdata") +sample = ("<?xml version='1.0' encoding='us-ascii'?>\n" + "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" + " 'http://xml.python.org/system' [\n" + " <!ELEMENT e EMPTY>\n" + " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" + "]><doc attr='value'> text\n" + "<?pi sample?>  <e/> </doc>") # The tests of DocumentType importing use these helpers to construct # the documents to work with, since not all DOM builders actually @@ -1466,52 +1474,54 @@ self.confirm(e.isSameNode(doc.getElementById("w")) and a2.isId) + def assert_recursive_equal(self, doc, doc2): + stack = [(doc, doc2)] + while stack: + n1, n2 = stack.pop() + self.assertEqual(n1.nodeType, n2.nodeType) + self.assertEqual(len(n1.childNodes), len(n2.childNodes)) + self.assertEqual(n1.nodeName, n2.nodeName) + self.assertFalse(n1.isSameNode(n2)) + self.assertFalse(n2.isSameNode(n1)) + if n1.nodeType == Node.DOCUMENT_TYPE_NODE: + len(n1.entities) + len(n2.entities) + len(n1.notations) + len(n2.notations) + self.assertEqual(len(n1.entities), len(n2.entities)) + self.assertEqual(len(n1.notations), len(n2.notations)) + for i in range(len(n1.notations)): + # XXX this loop body doesn't seem to be executed? + no1 = n1.notations.item(i) + no2 = n1.notations.item(i) + self.assertEqual(no1.name, no2.name) + self.assertEqual(no1.publicId, no2.publicId) + self.assertEqual(no1.systemId, no2.systemId) + stack.append((no1, no2)) + for i in range(len(n1.entities)): + e1 = n1.entities.item(i) + e2 = n2.entities.item(i) + self.assertEqual(e1.notationName, e2.notationName) + self.assertEqual(e1.publicId, e2.publicId) + self.assertEqual(e1.systemId, e2.systemId) + stack.append((e1, e2)) + if n1.nodeType != Node.DOCUMENT_NODE: + self.assertTrue(n1.ownerDocument.isSameNode(doc)) + self.assertTrue(n2.ownerDocument.isSameNode(doc2)) + for i in range(len(n1.childNodes)): + stack.append((n1.childNodes[i], n2.childNodes[i])) + def testPickledDocument(self): - doc = parseString("<?xml version='1.0' encoding='us-ascii'?>\n" - "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" - " 'http://xml.python.org/system' [\n" - " <!ELEMENT e EMPTY>\n" - " <!ENTITY ent SYSTEM 'http://xml.python.org/entity'>\n" - "]><doc attr='value'> text\n" - "<?pi sample?>  <e/> </doc>") + doc = parseString(sample) for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): s = pickle.dumps(doc, proto) doc2 = pickle.loads(s) - stack = [(doc, doc2)] - while stack: - n1, n2 = stack.pop() - self.confirm(n1.nodeType == n2.nodeType - and len(n1.childNodes) == len(n2.childNodes) - and n1.nodeName == n2.nodeName - and not n1.isSameNode(n2) - and not n2.isSameNode(n1)) - if n1.nodeType == Node.DOCUMENT_TYPE_NODE: - len(n1.entities) - len(n2.entities) - len(n1.notations) - len(n2.notations) - self.confirm(len(n1.entities) == len(n2.entities) - and len(n1.notations) == len(n2.notations)) - for i in range(len(n1.notations)): - # XXX this loop body doesn't seem to be executed? - no1 = n1.notations.item(i) - no2 = n1.notations.item(i) - self.confirm(no1.name == no2.name - and no1.publicId == no2.publicId - and no1.systemId == no2.systemId) - stack.append((no1, no2)) - for i in range(len(n1.entities)): - e1 = n1.entities.item(i) - e2 = n2.entities.item(i) - self.confirm(e1.notationName == e2.notationName - and e1.publicId == e2.publicId - and e1.systemId == e2.systemId) - stack.append((e1, e2)) - if n1.nodeType != Node.DOCUMENT_NODE: - self.confirm(n1.ownerDocument.isSameNode(doc) - and n2.ownerDocument.isSameNode(doc2)) - for i in range(len(n1.childNodes)): - stack.append((n1.childNodes[i], n2.childNodes[i])) + self.assert_recursive_equal(doc, doc2) + + def testDeepcopiedDocument(self): + doc = parseString(sample) + doc2 = copy.deepcopy(doc) + self.assert_recursive_equal(doc, doc2) def testSerializeCommentNodeWithDoubleHyphen(self): doc = create_doc_without_doctype() diff --git a/Lib/test/test_xml_dom_minicompat.py b/Lib/test/test_xml_dom_minicompat.py --- a/Lib/test/test_xml_dom_minicompat.py +++ b/Lib/test/test_xml_dom_minicompat.py @@ -1,5 +1,6 @@ # Tests for xml.dom.minicompat +import copy import pickle import unittest @@ -89,6 +90,7 @@ node_list = NodeList() pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) # Non-empty NodeList. @@ -96,7 +98,41 @@ node_list.append(2) pickled = pickle.dumps(node_list, proto) unpickled = pickle.loads(pickled) + self.assertIsNot(unpickled, node_list) self.assertEqual(unpickled, node_list) + def test_nodelist_copy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.copy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIs(x, y) + + def test_nodelist_deepcopy(self): + # Empty NodeList. + node_list = NodeList() + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + + # Non-empty NodeList. + node_list.append([1]) + node_list.append([2]) + copied = copy.deepcopy(node_list) + self.assertIsNot(copied, node_list) + self.assertEqual(copied, node_list) + for x, y in zip(copied, node_list): + self.assertIsNot(x, y) + self.assertEqual(x, y) + if __name__ == '__main__': unittest.main() diff --git a/Lib/xml/dom/minicompat.py b/Lib/xml/dom/minicompat.py --- a/Lib/xml/dom/minicompat.py +++ b/Lib/xml/dom/minicompat.py @@ -64,10 +64,10 @@ length = property(_get_length, _set_length, doc="The number of nodes in the NodeList.") - def __getstate__(self): - return list(self) - + # For backward compatibility def __setstate__(self, state): + if state is None: + state = [] self[:] = state diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -113,6 +113,9 @@ Library ------- +- Issue #10131: Fixed deep copying of minidom documents. Based on patch + by Marian Ganisin. + - Issue #25725: Fixed a reference leak in pickle.loads() when unpickling invalid data including tuple instructions. -- Repository URL: https://hg.python.org/cpython

1 0

devguide: #25194: Fix indentation in Nick's entry.
by r.david.murray Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/devguide/rev/2d4907e5b51c changeset: 775:2d4907e5b51c user: R David Murray <rdmurray(a)bitdance.com> date: Thu Nov 26 12:46:04 2015 -0500 summary: #25194: Fix indentation in Nick's entry. files: motivations.rst | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-) diff --git a/motivations.rst b/motivations.rst --- a/motivations.rst +++ b/motivations.rst @@ -166,22 +166,22 @@ .. topic:: Nick Coghlan (Australia) - * Personal site: `Curious Efficiency <http://www.curiousefficiency.org/>`_ - * `Extended bio <http://www.curiousefficiency.org/pages/about>`__ - * Red Hat (Software Engineer, Developer Experience) - * Python Software Foundation (Board of Directors) + * Personal site: `Curious Efficiency <http://www.curiousefficiency.org/>`_ + * `Extended bio <http://www.curiousefficiency.org/pages/about>`__ + * Red Hat (Software Engineer, Developer Experience) + * Python Software Foundation (Board of Directors) - Nick originally began participating in CPython core development as an - interesting and enlightening hobby activity while working for Boeing Defence - Australia. After commencing work for Red Hat, he also became involved in a - range of topics related directly to improving the experience of Python - developers on the Fedora Linux distribution and derived platforms, and now - works for Red Hat's Developer Experience team. + Nick originally began participating in CPython core development as an + interesting and enlightening hobby activity while working for Boeing Defence + Australia. After commencing work for Red Hat, he also became involved in a + range of topics related directly to improving the experience of Python + developers on the Fedora Linux distribution and derived platforms, and now + works for Red Hat's Developer Experience team. - In addition to his personal and professional interest in ensuring Python - remains an excellent choice for Linux-based network service and system - utility development, he is also interested in helping to ensure its - continued suitability for educational and data analysis use cases. + In addition to his personal and professional interest in ensuring Python + remains an excellent choice for Linux-based network service and system + utility development, he is also interested in helping to ensure its + continued suitability for educational and data analysis use cases. .. topic:: R. David Murray (United States) -- Repository URL: https://hg.python.org/devguide

1 0

devguide: #25194: add my entry to 'motivations'.
by r.david.murray Nov. 26, 2015

Nov. 26, 2015

https://hg.python.org/devguide/rev/b360bad24805 changeset: 774:b360bad24805 user: R David Murray <rdmurray(a)bitdance.com> date: Thu Nov 26 12:45:30 2015 -0500 summary: #25194: add my entry to 'motivations'. files: motivations.rst | 28 ++++++++++++++++++++++++++++ 1 files changed, 28 insertions(+), 0 deletions(-) diff --git a/motivations.rst b/motivations.rst --- a/motivations.rst +++ b/motivations.rst @@ -182,3 +182,31 @@ remains an excellent choice for Linux-based network service and system utility development, he is also interested in helping to ensure its continued suitability for educational and data analysis use cases. + +.. topic:: R. David Murray (United States) + + * Personal site: `bitdance.com <http://www.bitdance.com>`_ + * Available for `Python and Internet Services Consulting + and Python contract programming <http://www.murrayandwalker.com/>`_ + + David has been involved in the Internet since the days when the old IBM + BITNET and the ARPANet got cross connected, and in Python programming since + he first discovered it around the days of Python 1.4. After transitioning + from being Director of Operations for dialup Internet providers (when that + business started declining) to being a full time independent consultant, + David started contributing directly to CPython development. He became a + committer in 2009. He subsequently took over primary maintenance of the + email package from Barry Warsaw, and contributed the unicode oriented API. + David is also active in mentoring new contributors and, when time is + available, working on the infrastructure that supports CPython development, + specifically the Roundup-based bug tracker and the buildbot system. + + David currently does both proprietary and open source developement work, + primarily in Python, through the company in which he is a partner, `Murray & + Walker, Inc <http://www.murrayandwalker.com>`_. He has done contract work + focused specifically on CPython development both through the PSF (the + kickstart of the email unicode API development) and directly funded by + interested corporations (additional development work on email fundeded by + QNX, and work on CPython ICC support funded by Intel). He would like to + spend more of his (and his company's) time on open source work, and so is + actively seeking additional such contract opportunities. -- Repository URL: https://hg.python.org/devguide

1 0