Mailman 3 August 2013 - Python-checkins

cpython (3.3): Issue #18418: After fork(), reinit all threads states, not only active ones.
by charles-francois.natali Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/cpython/rev/29fce7f31539 changeset: 85463:29fce7f31539 branch: 3.3 parent: 85456:9810844126e1 user: Charles-François Natali <cf.natali(a)gmail.com> date: Fri Aug 30 23:32:53 2013 +0200 summary: Issue #18418: After fork(), reinit all threads states, not only active ones. Patch by A. Jesse Jiryu Davis. files: Lib/test/test_threading.py | 21 +++++++++++++++++++++ Lib/threading.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 26 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -444,6 +444,27 @@ self.assertEqual(out, b'') self.assertEqual(err, b'') + @unittest.skipUnless(hasattr(os, 'fork'), "needs os.fork()") + def test_is_alive_after_fork(self): + # Try hard to trigger #18418: is_alive() could sometimes be True on + # threads that vanished after a fork. + old_interval = sys.getswitchinterval() + self.addCleanup(sys.setswitchinterval, old_interval) + + # Make the bug more likely to manifest. + sys.setswitchinterval(1e-6) + + for i in range(20): + t = threading.Thread(target=lambda: None) + t.start() + self.addCleanup(t.join) + pid = os.fork() + if pid == 0: + os._exit(1 if t.is_alive() else 0) + else: + pid, status = os.waitpid(pid, 0) + self.assertEqual(0, status) + class ThreadJoinOnShutdown(BaseTestCase): diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -935,7 +935,7 @@ new_active = {} current = current_thread() with _active_limbo_lock: - for thread in _active.values(): + for thread in _enumerate(): # Any lock/condition variable may be currently locked or in an # invalid state, so we reinitialize them. thread._reset_internal_locks() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -279,6 +279,7 @@ Kushal Das Jonathan Dasteel Pierre-Yves David +A. Jesse Jiryu Davis John DeGood Ned Deily Vincent Delft diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -66,6 +66,9 @@ Library ------- +- Issue #18418: After fork(), reinit all threads states, not only active ones. + Patch by A. Jesse Jiryu Davis. + - Issue #16611: http.cookie now correctly parses the 'secure' and 'httponly' cookie flags. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #18418: After fork(), reinit all threads states, not only active ones.
by charles-francois.natali Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/cpython/rev/ba54011aa295 changeset: 85462:ba54011aa295 branch: 2.7 parent: 85452:7d530381fc0e user: Charles-François Natali <cf.natali(a)gmail.com> date: Fri Aug 30 23:30:50 2013 +0200 summary: Issue #18418: After fork(), reinit all threads states, not only active ones. Patch by A. Jesse Jiryu Davis. files: Lib/test/test_threading.py | 23 +++++++++++++++++++++++ Lib/threading.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 28 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -443,6 +443,29 @@ self.assertEqual(out, '') self.assertEqual(err, '') + @unittest.skipUnless(hasattr(os, 'fork'), "needs os.fork()") + def test_is_alive_after_fork(self): + # Try hard to trigger #18418: is_alive() could sometimes be True on + # threads that vanished after a fork. + old_interval = sys.getcheckinterval() + + # Make the bug more likely to manifest. + sys.setcheckinterval(10) + + try: + for i in range(20): + t = threading.Thread(target=lambda: None) + t.start() + pid = os.fork() + if pid == 0: + os._exit(1 if t.is_alive() else 0) + else: + t.join() + pid, status = os.waitpid(pid, 0) + self.assertEqual(0, status) + finally: + sys.setcheckinterval(old_interval) + class ThreadJoinOnShutdown(BaseTestCase): diff --git a/Lib/threading.py b/Lib/threading.py --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1220,7 +1220,7 @@ new_active = {} current = current_thread() with _active_limbo_lock: - for thread in _active.itervalues(): + for thread in _enumerate(): # Any lock/condition variable may be currently locked or in an # invalid state, so we reinitialize them. if hasattr(thread, '_reset_internal_locks'): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -230,6 +230,7 @@ Ben Darnell Kushal Das Jonathan Dasteel +A. Jesse Jiryu Davis John DeGood Ned Deily Vincent Delft diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -32,6 +32,9 @@ Library ------- +- Issue #18418: After fork(), reinit all threads states, not only active ones. + Patch by A. Jesse Jiryu Davis. + - Issue #11973: Fix a problem in kevent. The flags and fflags fields are now properly handled as unsigned. -- Repository URL: http://hg.python.org/cpython

1 0

peps: Update PEP453 for Grammar and add --without-pip to PyVenv
by donald.stufft Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/peps/rev/cff869e98a8c changeset: 5087:cff869e98a8c user: Donald Stufft <donald(a)stufft.io> date: Fri Aug 30 11:45:51 2013 -0400 summary: Update PEP453 for Grammar and add --without-pip to PyVenv files: pep-0453.txt | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/pep-0453.txt b/pep-0453.txt --- a/pep-0453.txt +++ b/pep-0453.txt @@ -146,10 +146,10 @@ ================ During the installation of Python from Python.org ``python -m getpip`` should -be executed. Leaving people using the Windows or OSX installers with a working +be executed, leaving people using the Windows or OSX installers with a working copy of pip once the installation has completed. The exact method of this is -left up to the maintainers of the installers however if the bootstrapping is -optional it should be opt out rather than opt in. +left up to the maintainers of the installers, however if the bootstrapping is +optional it should be opt-out rather than opt-in. The Windows and OSX installers distributed by Python.org will automatically attempt to run ``python -m getpip`` by default however the ``make install`` @@ -178,7 +178,9 @@ will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package making it a more -suitable replacement. +suitable replacement. In the case that a user does not wish to have pip +bootstrapped into their virtual environment a ``--without-pip`` option will be +added. Recommendations for Downstream Distributors -- Repository URL: http://hg.python.org/peps

1 0

peps: Grammar fixes for PEP453
by donald.stufft Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/peps/rev/62e0c20bf154 changeset: 5086:62e0c20bf154 user: Donald Stufft <donald(a)stufft.io> date: Fri Aug 30 09:38:12 2013 -0400 summary: Grammar fixes for PEP453 files: pep-0453.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0453.txt b/pep-0453.txt --- a/pep-0453.txt +++ b/pep-0453.txt @@ -21,7 +21,7 @@ redistributors of Python to also provide pip by default (in a way reasonable for their distributions). -This PEP does *not* proposes the inclusion of pip itslef in the standard +This PEP does *not* propose the inclusion of pip itself in the standard library. @@ -156,7 +156,7 @@ and ``make altinstall`` commands of the source distribution will not. Keeping the pip bootstrapping as a separate step for make based -installations should minimise the changes CPython redistributors need to +installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through make for the getpip invocation also ensures those installing from a custom source build can easily force an offline installation of pip, install it -- Repository URL: http://hg.python.org/peps

1 0

peps: Reject PEP 439
by nick.coghlan Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/peps/rev/c6b8eae33b69 changeset: 5085:c6b8eae33b69 user: Nick Coghlan <ncoghlan(a)gmail.com> date: Fri Aug 30 23:20:37 2013 +1000 summary: Reject PEP 439 files: pep-0439.txt | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diff --git a/pep-0439.txt b/pep-0439.txt --- a/pep-0439.txt +++ b/pep-0439.txt @@ -1,16 +1,17 @@ PEP: 439 -Title: Inclusion of pip bootstrap in Python installation +Title: Inclusion of implicit pip bootstrap in Python installation Version: $Revision$ Last-Modified: $Date$ Author: Richard Jones <richard(a)python.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> -Status: Draft +Status: Rejected Type: Standards Track Content-Type: text/x-rst Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013 +Resolution: http://mail.python.org/pipermail/distutils-sig/2013-August/022527.html Abstract @@ -26,6 +27,13 @@ 427 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP. +PEP Rejection +============= + +This PEP has been rejected in favour of a more explicit mechanism that +should achieve the same end result in a more reliable fashion. The more +explicit bootstrapping mechanism is described in PEP 453. + Rationale ========= -- Repository URL: http://hg.python.org/peps

1 0

peps: Add PEP 453: explicit pip bootstrapping
by nick.coghlan Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/peps/rev/e726ddffb7cf changeset: 5084:e726ddffb7cf user: Nick Coghlan <ncoghlan(a)gmail.com> date: Fri Aug 30 23:10:06 2013 +1000 summary: Add PEP 453: explicit pip bootstrapping files: pep-0453.txt | 323 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 323 insertions(+), 0 deletions(-) diff --git a/pep-0453.txt b/pep-0453.txt new file mode 100644 --- /dev/null +++ b/pep-0453.txt @@ -0,0 +1,323 @@ +PEP: 453 +Title: Explicit bootstrapping of pip in Python installations +Version: $Revision$ +Last-Modified: $Date$ +Author: Donald Stufft <donald(a)stufft.io>, + Nick Coghlan <ncoghlan(a)gmail.com> +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 10-Aug-2013 +Post-History: 30-Aug-2013 + + +Abstract +======== + +This PEP proposes the inclusion of a method for explicitly bootstrapping +`pip`_ as the default package manager for Python. It also proposes that the +distributions of Python available via Python.org will automatically run this +explicit bootstrapping method and a recommendation to third party +redistributors of Python to also provide pip by default (in a way reasonable +for their distributions). + +This PEP does *not* proposes the inclusion of pip itslef in the standard +library. + + +Proposal +======== + +This PEP proposes the inclusion of a ``getpip`` bootstrapping module in +Python 3.4, as well as in the upcoming maintenance releases of Python 2.7 +and Python 3.3. + + +Rationale +========= + +Installing a third party package into a freshly installed Python requires first +installing the package manager. This requires users ahead of time to know what +the package manager is, where to get them from, and how to install them. The +effect of this is that these external projects are required to either blindly +assume the user already has the package manager installed, needs to duplicate +the instructions and tell their users how to install the package manager, or +completely forgo the use of dependencies to ease installation concerns for +their users. + +All of the available options have their own drawbacks. + +If a project simply assumes a user already has the tooling then they get a +confusing error message when the installation command doesn't work. Some +operating may ease this pain by providing a global hook that looks for commands +that don't exist and suggest an OS package they can install to make the command +work. + +If a project chooses to duplicate the installation instructions and tell their +users how to install the package manager before telling them how to install +their own project then whenever these instructions need updates they need +updating by every project that has duplicated them. This will inevitably not +happen in every case leaving many different instructions on how to install it +many of them broken or less than optimal. These additional instructions might +also confuse users who try to install the package manager a second time +thinking that it's part of the instructions of installing the project. + +The problem of stale instructions can be alleviated by referencing `pip's +own bootstrapping instructions +<http://www.pip-installer.org/en/latest/installing.html>`__, but the user +experience involved still isn't good (especially on Windows, where +downloading and running a Python script with the default OS configuration is +significantly more painful than downloading and running a binary executable +or installer). + +The projects that have decided to forgo dependencies all together are forced +to either duplicate the efforts of other projects by inventing their own +solutions to problems or are required to simply include the other projects +in their own source trees. Both of these options present their own problems +either in duplicating maintenance work across the ecosystem or potentially +leaving users vulnerable to security issues because the included code or +duplicated efforts are not automatically updated when upstream releases a new +version. + +By providing the package manager by default it will be easier for users trying +to install these third party packages as well as easier for the people +distributing them as they no longer need to pick the lesser evil. This will +become more important in the future as the Wheel_ package format does not have +a built in "installer" in the form of ``setup.py`` so users wishing to install +a Wheel package will need an installer even in the simple case. + +Reducing the burden of actually installing a third party package should also +decrease the pressure to add every useful module to the standard library. This +will allow additions to the standard library to focus more on why Python should +have a particular tool out of the box instead of needing to use the difficulty +in installing a package as justification for inclusion. + + +Explicit Bootstrapping +====================== + +An additional module called ``getpip`` will be added to the standard library +whose purpose is to install pip and any of its dependencies into the +appropriate location (most commonly site-packages). It will expose a single +callable named ``bootstrap()`` as well as offer direct execution via +``python -m getpip``. Options for installing it such as index server, +installation location (``--user``, ``--root``, etc) will also be available +to enable different installation schemes. + +It is believed that users will want the most recent versions available to be +installed so that they can take advantage of the new advances in packaging. +Since any particular version of Python has a much longer staying power than +a version of pip in order to satisfy a user's desire to have the most recent +version the bootstrap will contact PyPI, find the latest version, download it, +and then install it. This process is security sensitive, difficult to get +right, and evolves along with the rest of packaging. + +Instead of attempting to maintain a "mini pip" for the sole purpose of +installing pip the ``getpip`` module will, as an implementation detail, include +a private copy of pip which will be used to discover and install pip from PyPI. +It is important to stress that this private copy of pip is *only* an +implementation detail and it should *not* be relied on or assumed to exist. + +Not all users will have network access to PyPI whenever they run the bootstrap. +In order to ensure that these users will still be able to bootstrap pip the +bootstrap will fallback to simply installing the included copy of pip. + +This presents a balance between giving users the latest version of pip, saving +them from needing to immediately upgrade pip after bootstrapping it, and +allowing the bootstrap to work offline in situations where users might already +have packages downloaded that they wish to install. + + +Updating the Bundled pip +------------------------ + +In order to keep up with evolutions in packaging as well as providing users +who are using the offline installation method with as recent version as +possible the ``getpip`` module should be updates to the latest versions of +everything it bootstraps. During the preparation for any release of Python, a +script, provided as part of this PEP, should be run to update the bundled +packages to the latest versions. + +This means that maintenance releases of the CPython installers will include +an updated version of the ``getpip`` bootstrap module. + + +Pre-installation +================ + +During the installation of Python from Python.org ``python -m getpip`` should +be executed. Leaving people using the Windows or OSX installers with a working +copy of pip once the installation has completed. The exact method of this is +left up to the maintainers of the installers however if the bootstrapping is +optional it should be opt out rather than opt in. + +The Windows and OSX installers distributed by Python.org will automatically +attempt to run ``python -m getpip`` by default however the ``make install`` +and ``make altinstall`` commands of the source distribution will not. + +Keeping the pip bootstrapping as a separate step for make based +installations should minimise the changes CPython redistributors need to +make to their build processes. Avoiding the layer of indirection through +make for the getpip invocation also ensures those installing from a custom +source build can easily force an offline installation of pip, install it +from a private index server, or skip installing pip entirely. + + +Python Virtual Environments +=========================== + +Python 3.3 included a standard library approach to virtual Python environments +through the ``venv`` module. Since it's release it has become clear that very +few users have been willing to use this feature in part due to the lack of +an installer present by default inside of the virtual environment. They have +instead opted to continue using the ``virtualenv`` package which *does* include +pip installed by default. + +To make the ``venv`` more useful to users it will be modified to issue the +pip bootstrap by default inside of the new environment while creating it. This +will allow people the same convenience inside of the virtual environment as +this PEP provides outside of it as well as bringing the ``venv`` module closer +to feature parity with the external ``virtualenv`` package making it a more +suitable replacement. + + +Recommendations for Downstream Distributors +=========================================== + +A common source of Python installations are through downstream distributors +such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX +package managers [#homebrew]_, or python specific tools [#conda]_. In order to +provide a consistent, user friendly experience to all users of Python +regardless of how they attained Python this PEP recommends and asks that +downstream distributors: + +* Ensure that whenever Python is installed pip is also installed. + + * This may take the form of separate with dependencies on each either so that + installing the python package installs the pip package and installing the + pip package installs the Python package. + +* Do not remove the bundled copy of pip. + + * This is required for offline installation of pip into a virtual environment. + * This is similar to the existing ``virtualenv`` package for which many + downstream distributors have already made exception to the common + "debundling" policy. + * This does mean that if ``pip`` needs to be updated due to a security + issue, so does the bundled version in the ``getpip`` bootstrap module + +* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly + using ``setup.py``. + + * This will ensure that downstream packages can utilize the new formats which + will not have a ``setup.py`` easier. + +* Ensure that all features of this PEP continue to work with any modifications + made. + + * Online installation of the latest version of pip into a global or virtual + python environment using ``python -m getpip``. + * Offline installation of the bundled version of pip into a global or virtual + python environment using ``python -m getpip``. + * ``pip install --upgrade pip`` in a global installation should not affect + any already created virtual environments. + * ``pip install --upgrade pip`` in a virtual environment should not affect + the global installation. + + +Policies & Governance +===================== + +The maintainers of the bundled software and the CPython core team will work +together in order to address the needs of both. The bundled software will still +remain external to CPython and this PEP does not include CPython subsuming the +responsibilities or decisions of the bundled software. This PEP aims to +decrease the burden on end users wanting to use third party packages and the +decisions inside it are pragmatic ones that represent the trust that the +Python community has placed in the authors and maintainers of the bundled +software. + + +Backwards Compatibility +----------------------- + +The public API of the ``getpip`` module itself will fall under the typical +backwards compatibility policy of Python for its standard library. The +externally developed software that this PEP bundles does not. + + +Security Releases +----------------- + +Any security update that affects the ``getpip`` module will be shared prior to +release with the PSRT. The PSRT will then decide if the issue inside warrants +a security release of Python. + + +Appendix: Rejected Proposals +============================ + + +Implicit Bootstrap +------------------ + +`PEP439`_, the predecessor for this PEP, proposes it's own solution. Its +solution involves shipping a fake ``pip`` command that when executed would +implicitly bootstrap and install pip if it does not already exist. This has +been rejected because it is too "magical". It hides from the end user when +exactly the pip command will be installed or that it is being installed at all. +It also does not provide any recommendations or considerations towards +downstream packagers who wish to manage the globally installed pip through the +mechanisms typical for their system. + + +Including pip In the Standard Library +------------------------------------- + +Similar to this PEP is the proposal of just including pip in the standard +library. This would ensure that Python always includes pip and fixes all of the +end user facing problems with not having pip present by default. This has been +rejected because we've learned through the inclusion and history of +``distutils`` in the standard library that losing the ability to update the +packaging tools independently can leave the tooling in a state of constant +limbo. Making it unable to ever reasonably evolve in a timeframe that actually +affects users as any new features will not be available to the general +population for *years*. + +Allowing the packaging tools to progress separately from the Python release +and adoption schedules allows the improvements to be used by *all* members +of the Python community and not just those able to live on the bleeding edge +of Python releases. + + +.. _Wheel: http://www.python.org/dev/peps/pep-0427/ +.. _pip: http://www.pip-installer.org +.. _setuptools: https://pypi.python.org/pypi/setuptools +.. _PEP439: http://www.python.org/dev/peps/pep-0439/ + + +References +========== + +.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` +.. [#debian] `Debian <http://www.debian.org>` +.. [#fedora] `Fedora <https://fedoraproject.org/>` +.. [#homebrew] `Homebrew <http://brew.sh/>` +.. [#conda] `Conda <http://www.continuum.io/blog/conda>` + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps

1 0

cpython: Issue #17741: Rename IncrementalParser and its methods.
by eli.bendersky Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/cpython/rev/8fd72b1bb262 changeset: 85461:8fd72b1bb262 user: Eli Bendersky <eliben(a)gmail.com> date: Fri Aug 30 05:51:20 2013 -0700 summary: Issue #17741: Rename IncrementalParser and its methods. The new names are hopefully more descriptive and consistent. If you feel you don't agree with this change, *please* read issue 17741 first - there's a lot of discussion in there. files: Doc/library/xml.etree.elementtree.rst | 191 +++++++------ Doc/whatsnew/3.4.rst | 5 +- Lib/test/test_xml_etree.py | 52 +- Lib/xml/etree/ElementTree.py | 36 +- 4 files changed, 151 insertions(+), 133 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -105,37 +105,42 @@ >>> root[0][1].text '2008' -Incremental parsing -^^^^^^^^^^^^^^^^^^^ +Pull API for asynchronous parsing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -It's possible to parse XML incrementally (i.e. not the whole document at once). -The most powerful tool for doing this is :class:`IncrementalParser`. It does -not require a blocking read to obtain the XML data, and is instead fed with -data incrementally with :meth:`IncrementalParser.data_received` calls. To get -the parsed XML elements, call :meth:`IncrementalParser.events`. Here's an -example:: +Most parsing functions provided by this module require to read the whole +document at once before returning any result. It is possible to use a +:class:`XMLParser` and feed data into it incrementally, but it's a push API that +calls methods on a callback target, which is too low-level and inconvenient for +most needs. Sometimes what the user really wants is to be able to parse XML +incrementally, without blocking operations, while enjoying the convenience of +fully constructed :class:`Element` objects. - >>> incparser = ET.IncrementalParser(['start', 'end']) - >>> incparser.data_received('<mytag>sometext') - >>> list(incparser.events()) - [('start', <Element 'mytag' at 0x7fba3f2a8688>)] - >>> incparser.data_received(' more text</mytag>') - >>> for event, elem in incparser.events(): - ... print(event) - ... print(elem.tag, 'text=', elem.text) - ... - end - mytag text= sometext more text +The most powerful tool for doing this is :class:`XMLPullParser`. It does not +require a blocking read to obtain the XML data, and is instead fed with data +incrementally with :meth:`XMLPullParser.feed` calls. To get the parsed XML +elements, call :meth:`XMLPullParser.read_events`. Here's an example:: + + >>> asyncparser = ET.XMLPullParser(['start', 'end']) + >>> asyncparser.feed('<mytag>sometext') + >>> list(asyncparser.read_events()) + [('start', <Element 'mytag' at 0x7fa66db2be58>)] + >>> asyncparser.feed(' more text</mytag>') + >>> for event, elem in asyncparser.read_events(): + ... print(event) + ... print(elem.tag, 'text=', elem.text) + ... + end The obvious use case is applications that operate in an asynchronous fashion where the XML data is being received from a socket or read incrementally from some storage device. In such cases, blocking reads are unacceptable. -Because it's so flexible, :class:`IncrementalParser` can be inconvenient -to use for simpler use-cases. If you don't mind your application blocking on -reading XML data but would still like to have incremental parsing capabilities, -take a look at :func:`iterparse`. It can be useful when you're reading a large -XML document and don't want to hold it wholly in memory. +Because it's so flexible, :class:`XMLPullParser` can be inconvenient to use for +simpler use-cases. If you don't mind your application blocking on reading XML +data but would still like to have incremental parsing capabilities, take a look +at :func:`iterparse`. It can be useful when you're reading a large XML document +and don't want to hold it wholly in memory. Finding interesting elements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -412,29 +417,32 @@ Parses an XML section into an element tree incrementally, and reports what's going on to the user. *source* is a filename or :term:`file object` containing XML data. *events* is a sequence of events to report back. The - supported events are the strings ``"start"``, ``"end"``, ``"start-ns"`` - and ``"end-ns"`` (the "ns" events are used to get detailed namespace + supported events are the strings ``"start"``, ``"end"``, ``"start-ns"`` and + ``"end-ns"`` (the "ns" events are used to get detailed namespace information). If *events* is omitted, only ``"end"`` events are reported. *parser* is an optional parser instance. If not given, the standard - :class:`XMLParser` parser is used. *parser* can only use the default - :class:`TreeBuilder` as a target. Returns an :term:`iterator` providing - ``(event, elem)`` pairs. + :class:`XMLParser` parser is used. *parser* must be a subclass of + :class:`XMLParser` and can only use the default :class:`TreeBuilder` as a + target. Returns an :term:`iterator` providing ``(event, elem)`` pairs. Note that while :func:`iterparse` builds the tree incrementally, it issues blocking reads on *source* (or the file it names). As such, it's unsuitable for asynchronous applications where blocking reads can't be made. For fully - asynchronous parsing, see :class:`IncrementalParser`. + asynchronous parsing, see :class:`XMLPullParser`. .. note:: - :func:`iterparse` only guarantees that it has seen the ">" - character of a starting tag when it emits a "start" event, so the - attributes are defined, but the contents of the text and tail attributes - are undefined at that point. The same applies to the element children; - they may or may not be present. + :func:`iterparse` only guarantees that it has seen the ">" character of a + starting tag when it emits a "start" event, so the attributes are defined, + but the contents of the text and tail attributes are undefined at that + point. The same applies to the element children; they may or may not be + present. If you need a fully populated element, look for "end" events instead. + .. deprecated:: 3.4 + The *parser* argument. + .. function:: parse(source, parser=None) Parses an XML section into an element tree. *source* is a filename or file @@ -871,48 +879,6 @@ :class:`QName` instances are opaque. -IncrementalParser Objects -^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. class:: IncrementalParser(events=None, parser=None) - - An incremental, event-driven parser suitable for non-blocking applications. - *events* is a sequence of events to report back. The supported events are - the strings ``"start"``, ``"end"``, ``"start-ns"`` and ``"end-ns"`` (the "ns" - events are used to get detailed namespace information). If *events* is - omitted, only ``"end"`` events are reported. *parser* is an optional - parser instance. If not given, the standard :class:`XMLParser` parser is - used. *parser* can only use the default :class:`TreeBuilder` as a target. - - .. method:: data_received(data) - - Feed the given bytes data to the incremental parser. - - .. method:: eof_received() - - Signal the incremental parser that the data stream is terminated. - - .. method:: events() - - Iterate over the events which have been encountered in the data fed - to the parser. This method yields ``(event, elem)`` pairs, where - *event* is a string representing the type of event (e.g. ``"end"``) - and *elem* is the encountered :class:`Element` object. Events - provided in a previous call to :meth:`events` will not be yielded - again. - - .. note:: - - :class:`IncrementalParser` only guarantees that it has seen the ">" - character of a starting tag when it emits a "start" event, so the - attributes are defined, but the contents of the text and tail attributes - are undefined at that point. The same applies to the element children; - they may or may not be present. - - If you need a fully populated element, look for "end" events instead. - - .. versionadded:: 3.4 - .. _elementtree-treebuilder-objects: @@ -973,13 +939,17 @@ .. class:: XMLParser(html=0, target=None, encoding=None) - :class:`Element` structure builder for XML source data, based on the expat - parser. *html* are predefined HTML entities. This flag is not supported by - the current implementation. *target* is the target object. If omitted, the - builder uses an instance of the standard :class:`TreeBuilder` class. - *encoding* [1]_ is optional. If given, the value overrides the encoding + This class is the low-level building block of the module. It uses + :mod:`xml.parsers.expat` for efficient, event-based parsing of XML. It can + be fed XML data incrementall with the :meth:`feed` method, and parsing events + are translated to a push API - by invoking callbacks on the *target* object. + If *target* is omitted, the standard :class:`TreeBuilder` is used. The + *html* argument was historically used for backwards compatibility and is now + deprecated. If *encoding* [1]_ is given, the value overrides the encoding specified in the XML file. + .. deprecated:: 3.4 + The *html* argument. .. method:: close() @@ -999,12 +969,12 @@ Feeds data to the parser. *data* is encoded data. - :meth:`XMLParser.feed` calls *target*\'s ``start()`` method - for each opening tag, its ``end()`` method for each closing tag, - and data is processed by method ``data()``. :meth:`XMLParser.close` - calls *target*\'s method ``close()``. - :class:`XMLParser` can be used not only for building a tree structure. - This is an example of counting the maximum depth of an XML file:: + :meth:`XMLParser.feed` calls *target*\'s ``start(tag, attrs_dict)`` method + for each opening tag, its ``end(tag)`` method for each closing tag, and data + is processed by method ``data(data)``. :meth:`XMLParser.close` calls + *target*\'s method ``close()``. :class:`XMLParser` can be used not only for + building a tree structure. This is an example of counting the maximum depth + of an XML file:: >>> from xml.etree.ElementTree import XMLParser >>> class MaxDepth: # The target object of the parser @@ -1038,6 +1008,51 @@ >>> parser.close() 4 + +.. _elementtree-xmlpullparser-objects: + +XMLPullParser Objects +^^^^^^^^^^^^^^^^^^^^^ + +.. class:: XMLPullParser(events=None) + + A pull parser suitable for nonblocking (asynchronous) applications. Its + input-side API is similar to that of :class:`XMLParser`, but instead of + pushing calls to a callback target, :class:`XMLPullParser` collects an + internal list of parsing events and lets the user read from it. *events* is a + sequence of events to report back. The supported events are the strings + ``"start"``, ``"end"``, ``"start-ns"`` and ``"end-ns"`` (the "ns" events are + used to get detailed namespace information). If *events* is omitted, only + ``"end"`` events are reported. + + .. method:: feed(data) + + Feed the given bytes data to the parser. + + .. method:: close() + + Signal the parser that the data stream is terminated. + + .. method:: read_events() + + Iterate over the events which have been encountered in the data fed to the + parser. This method yields ``(event, elem)`` pairs, where *event* is a + string representing the type of event (e.g. ``"end"``) and *elem* is the + encountered :class:`Element` object. Events provided in a previous call + to :meth:`read_events` will not be yielded again. + + .. note:: + + :class:`XMLPullParser` only guarantees that it has seen the ">" + character of a starting tag when it emits a "start" event, so the + attributes are defined, but the contents of the text and tail attributes + are undefined at that point. The same applies to the element children; + they may or may not be present. + + If you need a fully populated element, look for "end" events instead. + + .. versionadded:: 3.4 + Exceptions ^^^^^^^^^^ diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -402,10 +402,9 @@ --------- Add an event-driven parser for non-blocking applications, -:class:`~xml.etree.ElementTree.IncrementalParser`. +:class:`~xml.etree.ElementTree.XMLPullParser`. -(Contributed by Antoine Pitrou in :issue:`17782`.) - +(Contributed by Antoine Pitrou in :issue:`17741`.) Other improvements ================== diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -950,24 +950,24 @@ self.assertEqual(serialized, expected) -class IncrementalParserTest(unittest.TestCase): +class XMLPullParserTest(unittest.TestCase): def _feed(self, parser, data, chunk_size=None): if chunk_size is None: - parser.data_received(data) + parser.feed(data) else: for i in range(0, len(data), chunk_size): - parser.data_received(data[i:i+chunk_size]) + parser.feed(data[i:i+chunk_size]) def assert_event_tags(self, parser, expected): - events = parser.events() + events = parser.read_events() self.assertEqual([(action, elem.tag) for action, elem in events], expected) def test_simple_xml(self): for chunk_size in (None, 1, 5): with self.subTest(chunk_size=chunk_size): - parser = ET.IncrementalParser() + parser = ET.XMLPullParser() self.assert_event_tags(parser, []) self._feed(parser, "\n", chunk_size) self.assert_event_tags(parser, []) @@ -985,14 +985,14 @@ ]) self._feed(parser, "</root>\n", chunk_size) self.assert_event_tags(parser, [('end', 'root')]) - # Receiving EOF sets the `root` attribute + # Closing sets the `root` attribute self.assertIs(parser.root, None) - parser.eof_received() + parser.close() self.assertEqual(parser.root.tag, 'root') - def test_data_received_while_iterating(self): - parser = ET.IncrementalParser() - it = parser.events() + def test_feed_while_iterating(self): + parser = ET.XMLPullParser() + it = parser.read_events() self._feed(parser, "<root>\n <element key='value'>text</element>\n") action, elem = next(it) self.assertEqual((action, elem.tag), ('end', 'element')) @@ -1003,7 +1003,7 @@ next(it) def test_simple_xml_with_ns(self): - parser = ET.IncrementalParser() + parser = ET.XMLPullParser() self.assert_event_tags(parser, []) self._feed(parser, "\n") self.assert_event_tags(parser, []) @@ -1021,32 +1021,32 @@ ]) self._feed(parser, "</root>\n") self.assert_event_tags(parser, [('end', '{namespace}root')]) - # Receiving EOF sets the `root` attribute + # Closing sets the `root` attribute self.assertIs(parser.root, None) - parser.eof_received() + parser.close() self.assertEqual(parser.root.tag, '{namespace}root') def test_ns_events(self): - parser = ET.IncrementalParser(events=('start-ns', 'end-ns')) + parser = ET.XMLPullParser(events=('start-ns', 'end-ns')) self._feed(parser, "\n") self._feed(parser, "<root xmlns='namespace'>\n") self.assertEqual( - list(parser.events()), + list(parser.read_events()), [('start-ns', ('', 'namespace'))]) self._feed(parser, "<element key='value'>text</element") self._feed(parser, ">\n") self._feed(parser, "<element>text</element>tail\n") self._feed(parser, "<empty-element/>\n") self._feed(parser, "</root>\n") - self.assertEqual(list(parser.events()), [('end-ns', None)]) - parser.eof_received() + self.assertEqual(list(parser.read_events()), [('end-ns', None)]) + parser.close() def test_events(self): - parser = ET.IncrementalParser(events=()) + parser = ET.XMLPullParser(events=()) self._feed(parser, "<root/>\n") self.assert_event_tags(parser, []) - parser = ET.IncrementalParser(events=('start', 'end')) + parser = ET.XMLPullParser(events=('start', 'end')) self._feed(parser, "\n") self.assert_event_tags(parser, []) self._feed(parser, "<root>\n") @@ -1064,12 +1064,12 @@ ('end', '{foo}element'), ]) self._feed(parser, "</root>") - parser.eof_received() + parser.close() self.assertIs(parser.root, None) self.assert_event_tags(parser, [('end', 'root')]) self.assertEqual(parser.root.tag, 'root') - parser = ET.IncrementalParser(events=('start',)) + parser = ET.XMLPullParser(events=('start',)) self._feed(parser, "\n") self.assert_event_tags(parser, []) self._feed(parser, "<root>\n") @@ -1085,13 +1085,13 @@ ('start', '{foo}empty-element'), ]) self._feed(parser, "</root>") - parser.eof_received() + parser.close() self.assertEqual(parser.root.tag, 'root') def test_events_sequence(self): # Test that events can be some sequence that's not just a tuple or list eventset = {'end', 'start'} - parser = ET.IncrementalParser(events=eventset) + parser = ET.XMLPullParser(events=eventset) self._feed(parser, "<foo>bar</foo>") self.assert_event_tags(parser, [('start', 'foo'), ('end', 'foo')]) @@ -1103,14 +1103,14 @@ def __next__(self): return next(self.events) - parser = ET.IncrementalParser(events=DummyIter()) + parser = ET.XMLPullParser(events=DummyIter()) self._feed(parser, "<foo>bar</foo>") self.assert_event_tags(parser, [('start', 'foo'), ('end', 'foo')]) def test_unknown_event(self): with self.assertRaises(ValueError): - ET.IncrementalParser(events=('start', 'end', 'bogus')) + ET.XMLPullParser(events=('start', 'end', 'bogus')) # @@ -2546,7 +2546,6 @@ ElementSlicingTest, BasicElementTest, ElementTreeTest, - IncrementalParserTest, IOTest, ParseErrorTest, XIncludeTest, @@ -2555,6 +2554,7 @@ ElementIterTest, TreeBuilderTest, XMLParserTest, + XMLPullParserTest, BugsTest, ] diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -1210,37 +1210,39 @@ return _IterParseIterator(source, events, parser, close_source) -class IncrementalParser: +class XMLPullParser: - def __init__(self, events=None, parser=None): + def __init__(self, events=None, *, _parser=None): + # The _parser argument is for internal use only and must not be relied + # upon in user code. It will be removed in a future release. + # See http://bugs.python.org/issue17741 for more details. + # _elementtree.c expects a list, not a deque self._events_queue = [] self._index = 0 self.root = self._root = None - if not parser: - parser = XMLParser(target=TreeBuilder()) - self._parser = parser + self._parser = _parser or XMLParser(target=TreeBuilder()) # wire up the parser for event reporting if events is None: events = ("end",) self._parser._setevents(self._events_queue, events) - def data_received(self, data): + def feed(self, data): if self._parser is None: - raise ValueError("data_received() called after end of stream") + raise ValueError("feed() called after end of stream") if data: try: self._parser.feed(data) except SyntaxError as exc: self._events_queue.append(exc) - def eof_received(self): + def close(self): self._root = self._parser.close() self._parser = None if self._index >= len(self._events_queue): self.root = self._root - def events(self): + def read_events(self): events = self._events_queue while True: index = self._index @@ -1268,14 +1270,16 @@ class _IterParseIterator: def __init__(self, source, events, parser, close_source=False): - self._parser = IncrementalParser(events, parser) + # Use the internal, undocumented _parser argument for now; When the + # parser argument of iterparse is removed, this can be killed. + self._parser = XMLPullParser(events=events, _parser=parser) self._file = source self._close_file = close_source self.root = None def __next__(self): while 1: - for event in self._parser.events(): + for event in self._parser.read_events(): return event if self._parser._parser is None: self.root = self._parser.root @@ -1283,11 +1287,11 @@ self._file.close() raise StopIteration # load event buffer - data = self._file.read(16384) + data = self._file.read(16 * 1024) if data: - self._parser.data_received(data) + self._parser.feed(data) else: - self._parser.eof_received() + self._parser.close() def __iter__(self): return self @@ -1481,9 +1485,9 @@ pass # unknown def _setevents(self, events_queue, events_to_report): - # Internal API for IncrementalParser + # Internal API for XMLPullParser # events_to_report: a list of events to report during parsing (same as - # the *events* of IncrementalParser's constructor. + # the *events* of XMLPullParser's constructor. # events_queue: a list of actual parsing events that will be populated # by the underlying parser. # -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (6ec5d188c4db): sum=0
by solipsis＠pitrou.net Aug. 30, 2013

Aug. 30, 2013

results for 6ec5d188c4db on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogmwrk2k', '-x']

1 0

cpython: utilize int.from_bytes
by philip.jenvey Aug. 30, 2013

Aug. 30, 2013

http://hg.python.org/cpython/rev/c347b9063a9e changeset: 85460:c347b9063a9e user: Philip Jenvey <pjenvey(a)underboss.org> date: Thu Aug 29 18:33:50 2013 -0700 summary: utilize int.from_bytes files: Lib/uuid.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/uuid.py b/Lib/uuid.py --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -147,7 +147,7 @@ if len(bytes) != 16: raise ValueError('bytes is not a 16-char string') assert isinstance(bytes, bytes_), repr(bytes) - int = int_(('%02x'*16) % tuple(bytes), 16) + int = int_.from_bytes(bytes, byteorder='big') if fields is not None: if len(fields) != 6: raise ValueError('fields is not a 6-tuple') -- Repository URL: http://hg.python.org/cpython

1 0

cpython: improve comment
by benjamin.peterson Aug. 29, 2013

Aug. 29, 2013

http://hg.python.org/cpython/rev/6ec5d188c4db changeset: 85459:6ec5d188c4db user: Benjamin Peterson <benjamin(a)python.org> date: Thu Aug 29 17:29:30 2013 -0400 summary: improve comment files: Lib/datetime.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/datetime.py b/Lib/datetime.py --- a/Lib/datetime.py +++ b/Lib/datetime.py @@ -23,10 +23,10 @@ # for all computations. See the book for algorithms for converting between # proleptic Gregorian ordinals and many other calendar systems. -# -1 is a placeholder for 0. +# -1 is a placeholder for indexing purposes. _DAYS_IN_MONTH = [-1, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] -_DAYS_BEFORE_MONTH = [-1] # -1 is a placeholder for 0. +_DAYS_BEFORE_MONTH = [-1] # -1 is a placeholder for indexing purposes. dbm = 0 for dim in _DAYS_IN_MONTH[1:]: _DAYS_BEFORE_MONTH.append(dbm) -- Repository URL: http://hg.python.org/cpython

1 0